System

Abstract

A system is provided.SOLUTION: A system comprising: means for inputting a medical image of a patient; means for inputting basic information and image findings of the patient; database means for storing the medical image, the basic information, and the image findings; means for searching for a similar image from the database using a generative artificial intelligence model and a diffusion model; means for generating a suspected disease name based on the similar image, the basic information, and the image findings; and means for presenting the similar image and the suspected disease name to a user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] When doctors make diagnoses using medical images, there are challenges with the accuracy and speed of their diagnoses, especially when they lack confidence or encounter rare diseases. This problem affects the shortage of doctors in the medical field and the improvement of diagnostic quality. Currently, doctors have no easy way to search for similar images from the past or related diagnostic information, and each diagnosis requires a huge amount of time and effort. For this reason, there is a need for a system that allows doctors to have confidence in their diagnoses and efficiently utilize a large amount of information. [Means for solving the problem]

[0005] This invention provides a system that allows doctors to input patient medical images, annotate them with basic information and imaging findings, and store them in a database. The information stored in the database is analyzed using a generative artificial intelligence model and a diffusion model to search for similar past images. This system presents users with similar images along with the suspected disease name derived through the analysis, allowing doctors to quickly and accurately obtain the information necessary for diagnosis. It also has a function that accumulates post-diagnosis feedback in the database to improve search accuracy in future searches. This significantly improves the quality and efficiency of diagnoses in the medical field.

[0006] "Medical images of a patient" means images taken to visually display the internal or external condition of a patient, including, but not limited to, X-rays, CT scans, MRIs, and ultrasound images.

[0007] "Basic information" refers to basic personal information such as the patient's age, gender, medical history, family history, and lifestyle, which is used for diagnosis and treatment.

[0008] "Imaging findings" are results or observations obtained by a doctor when analyzing medical images, and include details such as the location, shape, size, and texture of a lesion.

[0009] "Database" refers to a collection of information that stores patient medical images, basic information, and imaging findings in an organized manner and allows for efficient search and retrieval.

[0010] A "generative artificial intelligence model" is an artificial intelligence algorithm that has the ability to analyze input data and generate new information and patterns, and is primarily used for searching for similar images and providing diagnostic support.

[0011] A "diffusion model" is a machine learning algorithm that has the ability to predict and generate new information from existing data, and is particularly used to evaluate the similarity of medical images.

[0012] A "similar image" is a past image that has been evaluated as having a similar structure or pattern to the input medical image, and serves as a reference for diagnosis.

[0013] The "suspected disease name" is a possible disease name estimated by the generative artificial intelligence model and the diffusion model based on the input data and similar images.

[0014] "Means for presenting to the user" refers to a display device or interface for visually displaying to the user (doctor) the similar images and suspected disease names obtained as search results. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11]FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The present invention relates to a diagnostic support system that uses medical images, basic information, and imaging findings. This system generates similar images and suspected disease names based on input of patient medical images and information by a doctor, and supports diagnosis. Specific embodiments of each element of this system are described below.

[0037] Data entry and saving

[0038] User (doctor): Enters the patient's medical images (e.g., CT scan), basic information such as age, gender, and medical history, as well as imaging findings (such as the location and shape of the lesion) into the terminal.

[0039] Terminal: Sends the entered information to the server.

[0040] Server: The received data is stored in a database. The stored data also includes metadata (date and time, information about the person who entered the data, etc.) at the time of input.

[0041] Search for similar images and suspected diseases

[0042] User (doctor): Enters medical images and basic information about a new patient who wishes to be diagnosed into the terminal, along with any imaging findings.

[0043] Terminal: Sends the entered data to the server.

[0044] Server: Receives the data and performs formatting processing, checking for missing data and converting it to a standard format.

[0045] Server: Using a generative AI model and a diffusion model, the server searches for similar images in the database. From the search results, it extracts images with high similarity and predicts the suspected disease names associated with them.

[0046] Results display and feedback

[0047] Server: Sends a list of similar images and suspected disease names to the terminal.

[0048] Terminal: Similar images and suspected disease names are displayed on the user interface.

[0049] User (doctor): Makes a diagnosis based on the information provided and enters the final diagnosis and its feedback into the terminal.

[0050] Terminal: Sends the input feedback information to the server.

[0051] Server: Updates the database based on the feedback information to improve search accuracy from the next time onwards.

[0052] Specific examples

[0053] 1. Examples of data entry and saving

[0054] User (doctor): Enters the chest X-ray image, age (45 years old), gender (male), and medical history (smoking history) of patient ID 12345 into the terminal form.

[0055] Terminal: Sends data to the server.

[0056] Server: Save the information of patient ID 12345 in the database.

[0057] 2. Example of searching for similar images and suspected diseases

[0058] User (doctor): Enters the chest X-ray image, age (65 years old), and gender (female) of new patient ID 67890 into the terminal.

[0059] Terminal: Sends data to the server.

[0060] Server: Uses a generative artificial intelligence model to search the database for similar past X-ray images. Extracts the most similar set of images from the previously stored images.

[0061] Server: Identifies the suspected disease name (e.g., pneumonia) related to the extracted image and sends it to the terminal.

[0062] 3. Examples of results and feedback

[0063] Terminal: The doctor's terminal displays a similar image of a chest X-ray and the suspected disease name "pneumonia."

[0064] User (doctor): Checks similar images and suspected disease names and confirms "pneumonia" as the definitive diagnosis for patient ID 67890.

[0065] User (doctor): Enters the definitive diagnosis and additional findings into the terminal and sends them to the server.

[0066] Server: Updates the database and accumulates feedback data to improve the accuracy of the next search.

[0067] In this way, the present invention allows doctors to easily search for similar images and obtain the name of a suspected disease when making a diagnosis, thereby improving the accuracy and efficiency of diagnosis.

[0068] The processing flow will be explained below.

[0069] Program processing flow

[0070] Data entry and saving

[0071] Step 1:

[0072] User (doctor): Enters basic information such as the patient's medical images (e.g., CT scan), age, gender, medical history, and imaging findings (such as the location and shape of the lesion) into a dedicated form on the terminal.

[0073] Step 2:

[0074] Terminal: Validate that the data entered is accurate and that all required fields are completed.

[0075] Step 3:

[0076] Terminal: Sends the verified data to the server.

[0077] Step 4:

[0078] Server: Double-checks the integrity of the data received and converts it into the appropriate format.

[0079] Step 5:

[0080] Server: Stores patient information, image data, and findings data in a database with unique identifiers.

[0081] Step 6:

[0082] Server: Adds metadata (such as the date and time of storage and information about the attending physician) to the stored data to improve future search performance.

[0083] Search for similar images and suspected diseases

[0084] Step 7:

[0085] User (doctor): Enters image data and basic information of a new patient for whom a diagnosis is desired into the terminal. If necessary, additional findings are also entered.

[0086] Step 8:

[0087] Terminal: Sends the entered data to the server.

[0088] Step 9:

[0089] Server: Validates the received data and checks for missing data.

[0090] Step 10:

[0091] Server: Converts image data into a standard format and formats it so that it can be easily processed by the AI ​​model.

[0092] Step 11:

[0093] Server: Using generative AI and diffusion models, the server searches for similar images in the database. The algorithm also takes into account the patient's basic information and findings.

[0094] Step 12:

[0095] Server: Extracts the most similar images from the search results and identifies the suspected disease associated with them.

[0096] Step 13:

[0097] Server: Sends similar images and the names of suspected diseases associated with them to the terminal.

[0098] Results display and feedback

[0099] Step 14:

[0100] Terminal: Displays the similar images and suspected disease names sent from the server on the user interface.

[0101] Step 15:

[0102] User (doctor): Check the displayed information and use it as a reference for diagnosis.

[0103] Step 16:

[0104] User (doctor): Enters the final diagnosis (e.g., confirmed disease name and additional findings) into the terminal and sends it to the server.

[0105] Step 17:

[0106] Server: Updates the database based on the feedback received, specifically by linking definitive diagnoses and additional findings to existing data to help improve the accuracy of future search results.

[0107] Step 18:

[0108] Server: The feedback information is also used as training data for the AI ​​model to improve its accuracy.

[0109] This detailed process flow will enable doctors to be more confident in their diagnoses, leading to more accurate and faster diagnoses. The system aims to improve the quality of healthcare by leveraging the power of automation and AI.

[0110] Example 1

[0111] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0112] In current medical settings, doctors often lack sufficient support to make accurate and prompt diagnoses. In particular, similar image searches and suspected disease diagnoses from medical images are often performed manually, resulting in time-consuming diagnosis and inconsistent diagnostic accuracy. A system that can solve these problems and support doctors' diagnoses is needed.

[0113] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0114] In this invention, the server includes: a means for inputting patient medical images; a means for inputting basic patient information and imaging findings; an information storage means for storing the medical images, basic information, and imaging findings; a transmission means for transmitting the input information to the server; a data formatting means for formatting the received data, checking for missing data, and converting it to a standard format; a means for searching for similar images from the information storage means using a generative artificial intelligence model and a diffusion model; a means for generating a suspected disease name based on the similar images, the basic information, and imaging findings; a means for displaying the similar images and the suspected disease name; a means for inputting a diagnosis result and feedback; a transmission means for transmitting the diagnosis result and feedback information to the server; and a means for updating the information storage means based on the feedback information to improve search accuracy from the next time onward. This allows doctors to easily search for similar images and obtain a suspected disease name when making a diagnosis.

[0115] "Patient medical images" refers to image data taken of a patient using medical equipment, including, specifically, CT scans, X-ray images, and MRI images.

[0116] "Basic information" refers to basic information about the patient, such as the patient's age, gender, and medical history.

[0117] "Imaging findings" refers to findings observed by a physician, such as the location or shape of a specific lesion or abnormality seen in a medical image.

[0118] "Information storage means" refers to a database or storage system for appropriately storing the aforementioned medical images, basic information, and image findings.

[0119] "Transmission means" refers to a communication protocol or interface for transmitting input information to a server.

[0120] "Data formatting means" refers to a function for formatting received data, checking for missing data, and converting it into a standard format.

[0121] A "generative artificial intelligence model" refers to a machine learning model that learns patterns and correlations from data and makes inferences and predictions based on the results.

[0122] A "diffusion model" is a type of generative artificial intelligence model that performs probabilistic sampling within a multidimensional data space to generate new data.

[0123] "Search means" refers to the function of searching for necessary information from information storage means, specifically an interface for performing database searches and searches for similar images.

[0124] "Means for generating" refers to the process or algorithm for calculating and generating a suspected disease name based on basic information and imaging findings.

[0125] "Displaying means" refers to a user interface for visually presenting similar images and suspected disease names to the user.

[0126] "Feedback information" refers to information including a doctor's diagnosis and additional findings, and includes data used to improve the system's search accuracy.

[0127] The present invention relates to a diagnostic support system that uses medical images, basic information, and image findings. This system generates similar images and suspected disease names and supports diagnosis by having a doctor input a patient's medical images and basic information. Specific embodiments of the present invention are described below.

[0128] System configuration

[0129] This system includes a terminal operated by a user (doctor) and a server connected to it. The terminal includes an input interface and a display interface, and the server includes a data storage means, a data formatting means, and a search means using a generative artificial intelligence model and a diffusion model.

[0130] Data entry and saving

[0131] The user (doctor) inputs the patient's medical images (e.g., CT scans), basic information such as age, gender, and medical history, as well as the imaging findings, into a terminal. The terminal then sends the input information to a server. The server then stores the received data in a database. At this time, metadata (date and time, information about the person who inputted, etc.) at the time of input is also stored.

[0132] Specific examples

[0133] For example, if a user inputs "Patient ID 12345, age 45, male, smoking history, chest X-ray image," the terminal sends this data to the server, which then stores this information in a database.

[0134] Search for similar images and suspected diseases

[0135] When a user (doctor) wishes to diagnose a new patient, they again input the medical images, basic information, and imaging findings into the terminal. The terminal then sends the input data to the server. The server then performs formatting on the received data, checks for missing data, and converts it into a standard format. A generative artificial intelligence model (using, for example, TensorFlow or PyTorch) and a diffusion model are used to search for similar images in the database. Images with high similarity are extracted from the search results, and suspected diseases related to them are estimated.

[0136] Specific examples

[0137] When a user inputs "Patient ID 67890, age 65, female, chest X-ray image," the device sends this data to the server. The server uses a generative artificial intelligence model to search a database for similar past X-ray images, extracts the most similar image set, and predicts the suspected disease name (e.g., "pneumonia").

[0138] Results display and feedback

[0139] The server sends a list of similar images and suspected disease names to the terminal. The terminal displays the similar images and suspected disease names on the user interface. The user (doctor) makes a diagnosis based on the provided information and enters a definitive diagnosis and its feedback into the terminal. The terminal sends the entered feedback information to the server. The server updates the database based on the feedback information, improving the accuracy of the next search.

[0140] Specific examples

[0141] For example, the terminal display shows "Similar Image 1," "Similar Image 2," and "Suspected Disease: Pneumonia." The user confirms this and enters "Patient ID 67890, Definitive Diagnosis: Pneumonia, Additional Findings: Shadow Spread." The terminal sends this feedback information to the server. The server updates the database and accumulates diagnosis history, improving search accuracy.

[0142] This system allows doctors to easily search for similar images and obtain the name of a suspected disease when making a diagnosis, improving the accuracy and efficiency of diagnosis.

[0143] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0144] Step 1:

[0145] The user (doctor) inputs the patient's medical images (e.g., CT scan), basic information such as age, gender, medical history, and imaging findings into the terminal. The input information is used as initial data for the medical diagnosis system. Specifically, the doctor inputs "Patient ID 12345, age 45, male, smoking history, chest X-ray image" into the input form on the terminal. Input is mainly done in the form of text fields and file uploads.

[0146] Step 2:

[0147] The terminal sends the input information to the server. A communication protocol such as an HTTP POST request is used for transmission. Specifically, when the send button is pressed, the terminal puts the input data into the body of an HTTP request and sends it to the specified endpoint on the server. This request includes the input information (patient ID, age, gender, medical history, and image data).

[0148] Step 3:

[0149] The server performs a formatting process on the received data. Data formatting includes checking for missing data and converting it to a standard format. For example, it checks whether required fields such as age and gender are missing, and converts it to JSON format. Specifically, the server analyzes the data and returns an error message if any information is missing. Once the formatting process is complete, the data is unified into a standard format. The input is the initial raw data, and the output is the formatted data.

[0150] Step 4:

[0151] The server stores the received and formatted data in a database. When storing the data, the metadata at the time of input (date and time, information about the person who entered, etc.) is also stored. Specifically, the server creates a database entry and stores the patient information appropriately. This process is performed using a specific database management system (DBMS). The input is formatted data, and the output is a record stored in the database.

[0152] Step 5:

[0153] The user (doctor) inputs medical images, basic information, and imaging findings of a new patient for whom they wish to diagnose into the terminal. Specifically, the doctor inputs, for example, "Patient ID 67890, age 65, female, chest X-ray image" into the terminal. The data input format is the same as a text field or file upload.

[0154] Step 6:

[0155] The terminal again sends the entered data to the server. The sending process is the same as in step 2, using an HTTP POST request. The input is the newly entered information about the patient seeking diagnosis, and the output is the request sent to the server.

[0156] Step 7:

[0157] The server uses a generative artificial intelligence model (using, for example, TensorFlow or PyTorch) and a diffusion model to search for similar images in the database. After data shaping, the input data is provided to the generative AI model, which calculates a similarity score. Based on this score, the most similar image set is extracted from the database. Specifically, the server runs the AI ​​model and compares previously saved images with newly input images. The input is the shaped new data, and the output is a set of highly similar images and a suspected disease name.

[0158] Step 8:

[0159] The server sends a list of estimated similar images and suspected disease names to the terminal. This is done again using an HTTP response. Specifically, the server returns the generated search results to the terminal in JSON format. The input is the output result of the AI ​​model, and the output is the data sent to the terminal.

[0160] Step 9:

[0161] The terminal displays similar images and the suspected disease name on the user interface. Specifically, the terminal analyzes the received data and displays information such as "Similar Image 1," "Similar Image 2," and "Suspected Disease Name: Pneumonia" on the screen. The input is the data sent from the server, and the output is the displayed user interface.

[0162] Step 10:

[0163] The user (doctor) makes a diagnosis based on the provided information and enters the definitive diagnosis and its feedback into the terminal. Specifically, the doctor enters "Patient ID 67890, definitive diagnosis: pneumonia, additional findings: shadow spreading" into the diagnosis result input form. The input is the diagnosis result and additional findings.

[0164] Step 11:

[0165] The terminal sends the entered feedback information to the server. The sending process again uses an HTTP POST request. The input is the doctor's final diagnosis and feedback information, and the output is the request sent to the server.

[0166] Step 12:

[0167] The server updates the database based on the feedback information and takes measures to improve search accuracy from the next time onwards. Specifically, the server analyzes the feedback information and updates the corresponding entries in the database. This enables diagnostic support to be performed with higher accuracy the next time the search is performed. The input is the feedback information and the output is the updated database.

[0168] (Application example 1)

[0169] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0170] Conventional diagnostic support systems have been used primarily in the medical field, making it difficult to apply them to other fields. Furthermore, abnormal behavior detection systems that use surveillance camera footage and sensor data lack effective means for detecting abnormal behavior in real time and responding quickly. Therefore, there is a need to apply the technology of diagnostic support systems used in the medical field to abnormal behavior detection systems that use surveillance camera footage and sensor data, thereby improving the accuracy of abnormal behavior detection.

[0171] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0172] In this invention, the server includes means for inputting patient medical images, means for inputting basic information and imaging findings of the patient, database means for storing the medical images, basic information, and imaging findings, means for searching for similar images from the database using a generative artificial intelligence model and a diffusion model, means for generating a suspected disease name based on the similar images and the basic information and imaging findings, means for presenting the similar images and suspected disease name to a user, means for inputting surveillance camera footage and sensor data, database means for storing the surveillance camera footage and sensor data, means for searching for similar behavior from the database using a generative artificial intelligence model, means for detecting abnormal behavior based on the similar behavior, the surveillance camera footage, and the sensor data, and means for presenting the abnormal behavior and the reason for the abnormal behavior to a user. This enables advanced abnormal behavior detection and diagnostic support in a variety of fields.

[0173] "Patient" refers to a person receiving medical services.

[0174] "Medical imaging" refers to image data, such as CT scans, X-rays, and MRI images, acquired to visualize the internal structures of a patient's body.

[0175] "Basic information" refers to basic information necessary for diagnosis and treatment, such as the patient's age, gender, and medical history.

[0176] "Imaging findings" refers to information and findings necessary for diagnosis that a doctor intuitively obtains based on medical images.

[0177] "Database means" refers to means for systematically storing, managing, and searching input data.

[0178] A "generative artificial intelligence model" refers to an artificial intelligence model that is trained to analyze data and generate new information.

[0179] A "diffusion model" refers to a model that spreads data to obtain accurate analysis results over a wide range.

[0180] "Similar images" refer to past images that have similar characteristics and features to the input medical image.

[0181] "Suspected disease name" refers to the name of a disease that is inferred based on similar images, basic information, and imaging findings.

[0182] "Surveillance camera footage" refers to video data captured by a surveillance camera.

[0183] "Sensor data" refers to data obtained from various sensors installed in a monitoring system.

[0184] "Similar behavior" refers to behavior that has similar characteristics to past behavioral data based on surveillance camera footage and sensor data.

[0185] "Abnormal behavior" refers to behavior that deviates from normal patterns of behavior and may pose a security problem.

[0186] "User" refers to the personnel and doctors who operate the system and use the information.

[0187] The present invention provides a system for assisting diagnosis using medical images and basic information of patients, and a system for detecting abnormal behavior using surveillance camera footage and sensor data. Specific embodiments of these systems are described below.

[0188] Diagnostic support system using medical images

[0189] First, a diagnosis support system using medical images and basic information will be described.

[0190] Data entry and saving

[0191] The user (doctor) inputs the patient's medical images (e.g., CT scans or X-ray images), basic information such as age, gender, and medical history, as well as imaging findings (such as the location and shape of the lesion) into a terminal. The terminal then sends this data to a server, which then stores the received data in a database. The stored data also includes metadata from the time of input (such as date and time and information about the person who entered the data).

[0192] Search for similar images and suspected diseases

[0193] The user (doctor) inputs medical images and basic information about a new patient they wish to diagnose into the terminal. The terminal then sends the input data to the server, which receives it and performs formatting processing. The data is checked for any missing parts and converted into a standard format. Next, a generative artificial intelligence model and a diffusion model are used to search for similar images in the database. Images with high similarity are extracted from the search results, and the suspected disease names associated with them are estimated.

[0194] Results display and feedback

[0195] The server sends a list of similar images and suspected diseases to the terminal, which displays them on the user interface. The user (doctor) makes a diagnosis based on the provided information and enters a definitive diagnosis and feedback into the terminal. The terminal then sends the entered feedback information to the server, which updates the database based on it.

[0196] As a specific example, a user (doctor) enters the chest X-ray image, age (45 years old), gender (male), and medical history (smoking history) of patient ID 12345 into a form on the terminal, and the terminal sends the data to the server and stores it.The user (doctor) then enters the chest X-ray image, age (65 years old), and gender (female) of a new patient ID 67890, and the server uses a generative artificial intelligence model to search for similar images from past data, identifies the suspected disease name, and sends it to the terminal.

[0197] Abnormal behavior detection system using surveillance camera footage and sensor data

[0198] Data entry and saving

[0199] Users (security personnel) input data into the system from surveillance camera footage, motion sensors, sound sensors, etc. The terminals send this data to the server, which then stores the received data in a database.

[0200] Abnormal behavior detection

[0201] The server uses a generative AI model to analyze past abnormal behavior data stored in a database and real-time surveillance camera footage and sensor data to detect abnormal behavior. The generative AI model uses YOLO (You Only Look Once) to analyze the video data and sensor data.

[0202] Results display and alerts

[0203] The server extracts video of suspected abnormal behavior and the reason for it, and issues an alert to the terminal. The video of suspected abnormal behavior and the reason for it are displayed on a user interface (for example, a web application built with React). The user (security officer) responds based on the information provided, and inputs feedback information (confirmation of abnormal behavior and how to deal with it) into the terminal and sends it to the server.

[0204] For example, if a surveillance camera detects an intrusion or abnormal activity, it uses a generative AI model to analyze the behavior and, if it determines that the behavior is abnormal, issues an alarm. Security personnel receive the alarm and can respond promptly.

[0205] Example prompt for a generative AI model:

[0206] "Detect abnormal behavior in surveillance camera footage in real time, compare it with past abnormal behavior data, and issue an alert. Abnormal behavior includes intrusions into specific areas or unusual movements."

[0207] This system will enable advanced abnormal behavior detection and diagnostic support not only in the medical field but also in the security field.

[0208] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0209] Diagnostic support system using medical images

[0210] Step 1:

[0211] Description of the process: The user (doctor) inputs the patient's medical images, basic information, and imaging findings into the terminal.

[0212] Input: Medical images (e.g., CT scan), patient age, gender, medical history, imaging findings

[0213] Data processing: Enter data into the form on the device

[0214] Output: Formatted data

[0215] Step 2:

[0216] Processing description: The terminal sends the input medical images, basic information, and image findings to the server.

[0217] Input: Formatted data (medical images, age, gender, medical history, imaging findings)

[0218] Data calculation: Packetize the data and send it to the server via the network

[0219] Output: Data received by the server

[0220] Step 3:

[0221] Process description: The server stores the received data in a database.

[0222] Input: The data received by the server

[0223] Data processing: Save the data to the corresponding record in the database

[0224] Output: Data stored in the database

[0225] Step 4:

[0226] Process description: The user (doctor) inputs medical images and basic information of a new patient into the terminal.

[0227] Input: New medical image, age, gender, medical history

[0228] Data processing: Enter data into the form on the device

[0229] Output: The new formatted data

[0230] Step 5:

[0231] Process description: The terminal sends the new data entered to the server.

[0232] Input: Formatted new data (medical images, age, gender, basic information, imaging findings)

[0233] Data calculation: Packetize the data and send it to the server via the network

[0234] Output: New data received by the server

[0235] Step 6:

[0236] Process Description: The server analyzes the new data received and searches for similar images in its database.

[0237] Input: New medical images and basic information

[0238] Data Computing: Searching for Similar Images Using Generative Artificial Intelligence Models and Diffusion Models

[0239] Output: A list of similar images

[0240] Step 7:

[0241] Process description: The server sends similar images and the suspected disease name to the terminal.

[0242] Input: A list of similar images

[0243] Data calculation: Estimation of suspected disease and data packetization

[0244] Output: List of similar images received by the device and suspected disease name

[0245] Step 8:

[0246] Process description: The terminal displays similar images and suspected disease names on the user interface.

[0247] Input: List of similar images and suspected disease name

[0248] Data processing: Display on the user interface

[0249] Output: Information that the user (doctor) can check

[0250] Step 9:

[0251] Process description: The user (doctor) inputs the definitive diagnosis and feedback information into the terminal.

[0252] Input: definitive diagnosis, additional findings

[0253] Data processing: Enter data into the form on the device

[0254] Output: Formatted feedback data

[0255] Step 10:

[0256] Process description: The terminal sends feedback data to the server.

[0257] Input: Formatted feedback data

[0258] Data calculation: Feedback data is packetized and sent to the server via the network

[0259] Output: Feedback data received by the server

[0260] Step 11:

[0261] Process description: The server stores the feedback data in a database to improve search accuracy for future searches.

[0262] Input: Received feedback data

[0263] Data processing: Update the database

[0264] Output: Updated database

[0265] Abnormal behavior detection system using surveillance camera footage and sensor data

[0266] Step 1:

[0267] Process description: The user (security officer) inputs surveillance camera footage and sensor data into the system.

[0268] Input: Surveillance camera footage, motion sensor data, sound sensor data

[0269] Data processing: Input into the system within the terminal

[0270] Output: Formatted monitoring data

[0271] Step 2:

[0272] Process description: The terminal sends the input monitoring data to the server.

[0273] Input: Formatted monitoring data

[0274] Data calculation: Packetize the data and send it to the server via the network

[0275] Output: Monitoring data received by the server

[0276] Step 3:

[0277] Process description: The server stores the received data in a database.

[0278] Input: Received monitoring data

[0279] Data processing: Save the data to the corresponding record in the database

[0280] Output: Monitoring data stored in a database

[0281] Step 4:

[0282] Process description: The server analyzes historical abnormal behavior data and real-time monitoring data in the database using a generative artificial intelligence model (e.g., YOLO).

[0283] Input: Real-time monitoring data, historical abnormal behavior data

[0284] Data Computation: A Comparative Analysis Using Generative Artificial Intelligence Models

[0285] Output: Abnormal behavior detection results

[0286] Step 5:

[0287] Processing description: When the server detects abnormal behavior, it sends alarm data including the reason to the terminal.

[0288] Input: Abnormal behavior detection result, reason

[0289] Data Computation: Alarm data generation and packetization

[0290] Output: Alarm data received by the terminal

[0291] Step 6:

[0292] Process Description: The terminal displays the alarm data on the user interface.

[0293] Input: Alarm data

[0294] Data processing: Display on the user interface

[0295] Output: Alert information that can be checked by the user (security officer)

[0296] Step 7:

[0297] Process description: The user (security officer) inputs the confirmed abnormal behavior and the response into the terminal.

[0298] Input: Confirmed abnormal behavior, response details

[0299] Data processing: Enter data into the form on the device

[0300] Output: Formatted feedback data

[0301] Step 8:

[0302] Process description: The terminal sends feedback data to the server.

[0303] Input: Formatted feedback data

[0304] Data calculation: Feedback data is packetized and sent to the server via the network

[0305] Output: Feedback data received by the server

[0306] Step 9:

[0307] Processing description: The server stores the received feedback data in a database to improve detection accuracy in the future.

[0308] Input: Received feedback data

[0309] Data processing: Update the database

[0310] Output: Updated database

[0311] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0312] The present invention relates to a diagnostic support system that uses medical images, basic information, and imaging findings. This system generates similar images and suspected disease names when a doctor inputs a patient's medical images and information, and also recognizes the user's emotions to support diagnosis. Specific embodiments of each element of this system are described below.

[0313] Data entry and saving

[0314] User (doctor): Enters the patient's medical images (e.g., CT scan), basic information such as age, gender, and medical history, as well as imaging findings (such as the location and shape of the lesion) into the terminal.

[0315] Terminal: Sends the entered information to the server.

[0316] Server: The received data is stored in a database, and the stored data also includes metadata at the time of input (such as the date and time of storage and information about the attending physician).

[0317] Search for similar images and suspected diseases

[0318] User (doctor): Enters medical images and basic information about a new patient who wishes to be diagnosed into the terminal, along with any imaging findings.

[0319] Terminal: Sends the entered data to the server.

[0320] Server: Validates the received data, checks for missing data, and converts the image data into a standard format that is easy for the AI ​​model to process.

[0321] Server: Using a generative AI model and a diffusion model, the server searches for similar images in the database. From the search results, it extracts images with high similarity and predicts the suspected disease names associated with them.

[0322] Displaying results and using the emotion engine

[0323] Server: Sends a list of similar images and suspected disease names to the terminal.

[0324] Terminal: Similar images and suspected disease names are displayed on the user interface.

[0325] User (doctor): A diagnosis is made based on the information provided. The device's built-in emotion engine recognizes the user's emotions from facial expressions and tone of voice, and evaluates their reliability and stress level.

[0326] Feedback and Database Updates

[0327] User (doctor): Enters the final diagnosis (e.g., confirmed disease name and additional findings) into the terminal. The emotion evaluation by the emotion engine is also recorded at the same time.

[0328] Terminal: Sends the input feedback information to the server.

[0329] Server: Updates the database based on feedback information to improve search accuracy from the next time onwards. Also, adjusts the display method and presentation order based on the evaluation results of the emotion engine, allowing users to use information more efficiently.

[0330] Specific examples

[0331] 1. Examples of data entry and saving

[0332] User (doctor): Enters the chest X-ray image, age (45 years old), gender (male), and medical history (smoking history) of patient ID 12345 into the terminal.

[0333] Terminal: Sends data to the server.

[0334] Server: Save the information of patient ID 12345 in the database.

[0335] 2. Example of searching for similar images and suspected diseases

[0336] User (doctor): Enters the chest X-ray image, age (65 years old), and gender (female) of new patient ID 67890 into the terminal.

[0337] Terminal: Sends data to the server.

[0338] Server: Uses a generative artificial intelligence model to search the database for similar past X-ray images. Extracts the most similar set of images from the previously stored images.

[0339] Server: Identifies the suspected disease name (e.g., pneumonia) related to the extracted image and sends it to the terminal.

[0340] 3. Displaying the results and using the emotion engine

[0341] Terminal: The doctor's terminal displays a similar image of a chest X-ray and the suspected disease name "pneumonia."

[0342] Emotion engine: Analyzes the user's facial expressions and tone of voice to assess their confidence in the diagnosis and stress level.

[0343] User (doctor): Checks similar images and suspected disease names, and in a stable emotional state, confirms that "pneumonia" is the definitive diagnosis for patient ID 67890.

[0344] User (doctor): Enters the definitive diagnosis, additional findings, and emotional assessment during the diagnosis into the terminal and sends them to the server.

[0345] 4. Examples of feedback and database updates

[0346] Server: The received feedback information is stored in a database to improve search and emotion evaluation accuracy in the future.

[0347] Server: Based on the feedback information, the AI ​​model is trained and diagnostic support functions are enhanced.

[0348] In this way, the present invention allows doctors to easily search for similar images and identify suspected diseases when making diagnoses. Furthermore, by using an emotion engine, it becomes possible to evaluate the confidence in a diagnosis and the stress level, enabling more appropriate information to be provided. This not only improves the accuracy and efficiency of diagnoses, but also contributes to reducing the burden on doctors.

[0349] The processing flow will be explained below.

[0350] Program processing flow: Data entry and saving

[0351] Step 1:

[0352] User (doctor): Enters basic information such as the patient's medical images (e.g., CT scan), age, gender, medical history, and imaging findings (such as the location and shape of the lesion) into a dedicated form on the terminal.

[0353] Step 2:

[0354] Terminal: Validate that the data entered is accurate and that all required fields are completed.

[0355] Step 3:

[0356] Terminal: Sends the verified data to the server.

[0357] Step 4:

[0358] Server: Double-checks the integrity of the data received and converts it into the appropriate format.

[0359] Step 5:

[0360] Server: Stores patient information, image data, and findings data in a database with unique identifiers.

[0361] Step 6:

[0362] Server: Adds metadata (such as the date and time of storage and information about the attending physician) to the stored data to improve future search performance.

[0363] Program processing flow: Search for similar images and suspected disease names

[0364] Step 7:

[0365] User (doctor): Enters image data and basic information of a new patient for whom a diagnosis is desired into the terminal. If necessary, additional findings are also entered.

[0366] Step 8:

[0367] Terminal: Sends the entered data to the server.

[0368] Step 9:

[0369] Server: Validates the received data and checks for missing data.

[0370] Step 10:

[0371] Server: Converts image data into a standard format and formats it so that it can be easily processed by the AI ​​model.

[0372] Step 11:

[0373] Server: Using generative AI and diffusion models, the server searches for similar images in the database. The algorithm also takes into account the patient's basic information and findings.

[0374] Step 12:

[0375] Server: Extracts the most similar images from the search results and identifies the suspected disease associated with them.

[0376] Step 13:

[0377] Server: Sends similar images and the names of suspected diseases associated with them to the terminal.

[0378] Program processing flow: "Using the emotion engine"

[0379] Step 14:

[0380] Terminal: Displays the similar images and suspected disease names sent from the server on the user interface.

[0381] Step 15:

[0382] Emotion engine: Analyzes the user's (doctor's) facial expressions and tone of voice in real time to assess their confidence in the diagnosis and stress level.

[0383] Step 16:

[0384] User (doctor): Checks the displayed information and makes a diagnostic decision while taking into account feedback from the emotion engine. If the user feels stressed during the process, the emotion engine automatically displays a diagnostic support message.

[0385] Program flow: Feedback and database updates

[0386] Step 17:

[0387] User (doctor): Enters the final diagnosis (e.g., confirmed disease name and additional findings) into the terminal. The emotion evaluation by the emotion engine is also recorded at the same time.

[0388] Step 18:

[0389] Terminal: Sends the input feedback information to the server.

[0390] Step 19:

[0391] Server: Updates the database based on the feedback received, specifically by linking definitive diagnoses and additional findings to existing data to help improve the accuracy of future search results.

[0392] Step 20:

[0393] Server: Based on the feedback information and the evaluation results of the emotion engine, it is also used as training data for the AI ​​model to improve the accuracy of the model.

[0394] This detailed processing flow allows doctors to easily search for similar images and identify suspected diseases when making a diagnosis, and the emotional engine's support improves the reliability of the diagnosis, leading to reduced stress. This increases the accuracy and efficiency of diagnoses, and greatly contributes to reducing the burden on medical professionals.

[0395] Example 2

[0396] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0397] In modern medical settings, rapid and accurate diagnosis using medical images is required for patient diagnosis. However, it is not easy to efficiently handle large amounts of image data during the diagnostic process and accurately identify suspected diseases. Furthermore, the doctor's emotions and stress can affect the accuracy of the diagnosis, which is also a major challenge. Therefore, there is a need for a diagnostic support system that not only quickly searches for similar images and provides an appropriate suspected disease name, but also takes the doctor's emotional state into account.

[0398] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0399] In this invention, the server includes: means for inputting patient medical images; means for inputting basic patient information and imaging findings; database means for storing the medical images, basic information, and imaging findings; means for searching the database for similar images using a generative artificial intelligence model and a diffusion model; means for generating suspected disease names based on the similar images, basic information, and imaging findings; means for presenting the similar images and suspected disease names to the user; emotion engine means for analyzing the user's facial expression and tone of voice to evaluate reliability and stress level; means for inputting diagnostic feedback and storing it together with an emotion evaluation; and means for updating the database based on the stored feedback information to improve the accuracy of subsequent searches and emotion evaluations. This enables rapid and accurate diagnostic support and also enables the provision of appropriate information that takes into account the doctor's emotional state.

[0400] "Patient medical images" means image data of a patient used by a physician for diagnostic or treatment purposes, including formats such as X-rays, CT scans, and MRIs.

[0401] "Basic information" refers to basic data necessary for diagnosis and treatment, such as the patient's age, gender, and medical history.

[0402] "Imaging findings" are data showing detailed observation results, such as the location and shape of lesions identified by doctors based on diagnostic imaging.

[0403] "Database Means" means a computer system for storing and managing patient medical images, baseline information, and imaging findings.

[0404] A "generative artificial intelligence model" is a collection of algorithms that generate new data based on trained data and perform classification and prediction.

[0405] The "diffusion model" is a mathematical technique for identifying similar images by diffusing image data in a high-dimensional space and performing similarity calculations.

[0406] The "emotion engine means" is a software module for analyzing a user's facial expressions and tone of voice to assess their emotional state, such as their trustworthiness or stress level.

[0407] "Diagnostic feedback" refers to data entered by doctors regarding diagnostic results and additional findings, which is used to improve the system's learning accuracy and to support future diagnoses.

[0408] "Feedback information" refers to data such as diagnostic feedback and evaluation results from the emotion engine that the system uses to improve search accuracy and the user interface in future searches.

[0409] "Search and emotion assessment accuracy" refers to the accuracy with which the system identifies similar images and related disease names, and the accuracy with which the emotion engine assesses the user's emotional state.

[0410] MODE FOR CARRYING OUT THE INVENTION

[0411] The present invention relates to a diagnostic support system that uses a patient's medical images, basic information, and imaging findings. This system generates similar images and suspected disease names when a doctor inputs the patient's medical images and information, and further supports diagnosis by recognizing the user's emotions. Specific embodiments of each element of the present invention are described below.

[0412] System configuration

[0413] This system mainly consists of the following elements:

[0414] User (doctor): Enters the patient's medical images, basic information, and imaging findings.

[0415] Terminal: Accepts user input, sends data to the server, and displays the results.

[0416] Server: Stores data in a database, searches for similar images using a generative artificial intelligence model and a diffusion model, and generates a suspected disease name.

[0417] Database means: Stores medical images, basic information, and imaging findings.

[0418] System operation procedures

[0419] Data entry and saving

[0420] 1. The user (doctor) inputs the patient's medical images (e.g., CT scans or X-ray images), basic information such as age, gender, and medical history, as well as imaging findings (such as the location and shape of the lesion) into the terminal.

[0421] Example: Chest X-ray image of patient ID 12345, age 45, gender male, and smoking history as medical history are entered into the terminal.

[0422] 2. The device sends the entered information to the server using a secure protocol such as HTTPS.

[0423] 3. The server stores the received data in a database. When storing the data, metadata (such as the date and time of storage and the ID of the attending physician) is also recorded. The database engine used is MySQL, for example.

[0424] Search for similar images and suspected diseases

[0425] 4. The user (doctor) inputs the medical images and basic information of the new patient, as well as imaging findings if necessary, into the terminal.

[0426] Example: Enter the chest X-ray image of patient ID 67890, age 65, and gender female into the terminal.

[0427] 5. The terminal sends the entered data back to the server.

[0428] 6. The server validates the received data and checks for missing data. It converts the image data into a standard format and formats it for the AI ​​model. For example, OpenCV is used as the image processing library.

[0429] 7. The server uses a generative artificial intelligence model (e.g., GAN, Diffusion Models) to search for similar images in the database and estimate the suspected disease name.

[0430] Example prompt: "Search the database of historical X-ray images for the image most similar to the chest X-ray image of patient ID 67890 and identify any suspected related diseases."

[0431] Displaying results and using the emotion engine

[0432] 8. The server sends the similar images and the suspected disease names to the terminal.

[0433] 9. The terminal displays the received similar images and suspected disease names on the user interface.

[0434] Example: Similar images and the suspected disease name "pneumonia" are displayed on the device.

[0435] 10. The user (doctor) makes a diagnosis based on the provided information. During this time, the emotion engine installed on the device analyzes the doctor's facial expressions and tone of voice to evaluate his / her trustworthiness and stress level.

[0436] Feedback and Database Updates

[0437] 11. The user (doctor) enters the final diagnosis into the terminal and simultaneously records the emotional evaluation.

[0438] Example: Enter a definitive diagnosis of "pneumonia," additional findings, and an emotional assessment of "high confidence."

[0439] 12. The terminal transmits the input feedback information to the server.

[0440] 13. The server updates the database based on the feedback information to improve the accuracy of future searches and the evaluation of the emotion engine.

[0441] Example: A new feedback record is added to the database, and retraining of the AI ​​model is automatically triggered in the backend.

[0442] This system allows doctors to easily search for similar images and identify the appropriate suspected disease when making a diagnosis. Furthermore, by utilizing an emotion engine, it is possible to assess confidence in the diagnosis and stress levels, and provide optimal information. This improves the accuracy and efficiency of diagnoses and also contributes to reducing the burden on doctors.

[0443] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0444] Step 1:

[0445] The user (doctor) uses the terminal to input the patient's medical images (e.g., CT scans, X-ray images), basic information such as age, gender, and medical history, as well as the image findings.

[0446] Input: Patient's medical images and basic information, imaging findings

[0447] Output: Data entered into the terminal

[0448] Specific operation: The doctor enters the necessary information into the input form on the terminal and uploads the image file. For example, the doctor enters the chest X-ray image of patient ID 12345, age 45, gender male, and smoking history as medical history, and uploads the image.

[0449] Step 2:

[0450] The device collects the data entered by the user and sends it to a server using a secure protocol such as HTTPS.

[0451] Input: Data entered into the terminal

[0452] Output: Data sent to the server

[0453] Specific operation: When the send button is pressed, the device encodes the data and sends it to the server via HTTPS. The data includes the patient's medical images, basic information, and imaging findings.

[0454] Step 3:

[0455] The server stores the received data in a database, along with metadata such as the date and time of storage and information about the attending physician.

[0456] Input: Data sent to the server

[0457] Output: Data stored in the database

[0458] Specific operation: The server connects to a database engine (e.g., MySQL), creates a new patient record, and saves the data, including the patient's medical images, basic information, imaging findings, save date and time, and attending physician.

[0459] Step 4:

[0460] The user (doctor) inputs medical images and basic information of a new patient, as well as imaging findings if necessary, into the terminal.

[0461] Input: Medical images and basic information of new patients, imaging findings

[0462] Output: Data entered into the terminal

[0463] Specific operation: The doctor enters the chest X-ray image of new patient ID 67890, age 65, and gender female into the input form on the terminal and uploads the image.

[0464] Step 5:

[0465] The terminal transmits the newly input data to the server again.

[0466] Input: Data entered into the terminal

[0467] Output: Data sent to the server

[0468] Specific operation: When you press the send button, the device encodes the data and sends it back to the server via HTTPS.

[0469] Step 6:

[0470] The server validates the received data, checking for missing data, then converts the image data into a standard format and prepares it for the AI ​​model.

[0471] Input: Data sent to the server

[0472] Output: Validated data and formatted image data

[0473] What happens: A validation script on the server checks the integrity of the data and converts the images to a standard format using an image processing library (e.g. OpenCV).

[0474] Step 7:

[0475] The server uses a generative artificial intelligence model and a diffusion model to search for similar images in a database and estimate the suspected disease name.

[0476] Input: Validated data and formatted image data

[0477] Output: Similar images and suspected disease names

[0478] Specific operation: A generative AI model (e.g., GAN, Diffusion Models) analyzes image data and performs similarity calculations. The associated disease name is inferred from the metadata of the most similar image. An example prompt is, "Search the historical X-ray image database for the image most similar to the chest X-ray image of patient ID 67890 and identify the suspected associated disease name."

[0479] Step 8:

[0480] The server transmits the similar image and the suspected disease name to the terminal.

[0481] Input: Similar images and suspected disease name

[0482] Output: Similar images and suspected disease names sent to the terminal

[0483] Specific operation: The server encodes the search results and the estimated disease name in JSON format and sends them to the terminal.

[0484] Step 9:

[0485] The terminal displays the received similar images and suspected disease names on the user interface.

[0486] Input: Similar images and suspected disease names sent to the terminal

[0487] Output: Displayed similar images and suspected disease names

[0488] Specific operation: Highly relevant similar images and the estimated disease name "pneumonia" are displayed on the device interface.

[0489] Step 10:

[0490] The user (doctor) makes a diagnosis based on the provided information, while the device's built-in emotion engine analyzes facial expressions and tone of voice to assess trustworthiness and stress level.

[0491] Input: Displayed similar images, suspected disease names, facial expressions and tone of voice during the user's diagnostic activity

[0492] Output: Emotion engine estimates confidence and stress level

[0493] How it works: During the diagnosis, the emotion engine collects and analyzes data from the webcam and microphone in real time, assessing the confidence and stress level of the user as they enter their diagnosis information.

[0494] Step 11:

[0495] The user (doctor) inputs the final diagnosis and emotional evaluation into the terminal.

[0496] Input: Final diagnosis and emotional evaluation

[0497] Output: Feedback and emotion rating data entered into the device

[0498] Specific operation: Enter the definitive diagnosis, additional findings, and emotional assessment into the input form on the device. Example: In addition to the definitive diagnosis of "pneumonia," enter "high confidence."

[0499] Step 12:

[0500] The terminal transmits the input feedback information to the server.

[0501] Input: Feedback and emotional rating data entered into the device

[0502] Output: Feedback and sentiment rating data sent to the server

[0503] Specific operation: When you press the send button, the device encodes the data and sends it to the server.

[0504] Step 13:

[0505] The server stores the received feedback information in a database to improve the accuracy of future searches and emotion evaluations.

[0506] Input: Feedback and sentiment rating data sent to the server

[0507] Output: An updated database and a trained AI model

[0508] Specific operation: The server adds new feedback records to the database and automatically initiates retraining of the AI ​​model to improve search accuracy and sentiment assessment accuracy.

[0509] (Application example 2)

[0510] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0511] Current machine maintenance and defective product inspection in factories places a heavy burden on operators, and anomaly detection and diagnosis require time and effort. Furthermore, there is a lack of appropriate information provision and diagnostic feedback based on the operator's emotions and stress level, which reduces work efficiency and diagnostic reliability. Therefore, there is a need for a system that reduces the burden on operators and improves diagnostic accuracy and work efficiency.

[0512] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0513] In this invention, the server includes means for inputting video data and various sensor data of machines and products, database means for storing the video data and various sensor data, and means for generating suspected fault locations and abnormalities of machines and products using the video data and various sensor data. This reduces the burden on operators and improves diagnostic accuracy and work efficiency during machine maintenance and defective product inspection in factories.

[0514] "Machinery and products" refers to the equipment used in the factory and the goods manufactured.

[0515] "Video data" refers to image information captured by a photographing device such as a camera.

[0516] "Sensor data" refers to information obtained from various sensors that detect vibration, temperature, sound, etc.

[0517] A "database" is a system that stores and manages input video data, sensor data, and associated metadata.

[0518] A "generative artificial intelligence model" is an artificial intelligence that uses machine learning technology to generate new information from data.

[0519] A "diffusion model" is an algorithm that discovers patterns and trends from large amounts of data and analyzes relationships in a diffuse manner.

[0520] A "similar image" is an image among the video data stored in the database that has characteristics similar to the newly input video data.

[0521] A "suspected disease name" is a disease name that may be inferred based on medical images and basic information.

[0522] A "suspected failure point" is a part or component of a machine or product where an abnormality is suspected.

[0523] An "emotion engine" is a system that analyzes a user's emotional responses, such as facial expressions and tone of voice, to assess their trustworthiness and stress level.

[0524] A "user" is someone who operates the system and checks the results.

[0525] This invention relates to a system that supports machine maintenance and defective product inspection in factories. This system inputs video data and various sensor data to detect failures and abnormalities early, improving diagnostic accuracy and work efficiency. It also includes an emotion engine that recognizes the emotions of operators and evaluates the reliability of diagnoses and stress levels.

[0526] System Configuration and Operation

[0527] Data entry and saving

[0528] The user (operator) inputs video data of machines and products in the factory and multiple sensor data (e.g., vibration, temperature, sound) into a dedicated terminal. The terminal sends this input information to a server, which then stores the received data in a database. This data also includes metadata at the time of input (e.g., the date and time of saving and information about the operator in charge).

[0529] Search for similar data and suspected fault locations

[0530] The user inputs newly acquired video data and sensor data into the device. This data is then sent back to the server, where it is first converted into a standard format and formatted so that the AI ​​model can easily process it. The server then uses a generative AI model and a diffusion model to search the database for similar past data. As a result of the search, data with high similarity is extracted, and related suspected fault locations and anomalies are inferred.

[0531] Displaying results and using the emotion engine

[0532] The server sends the extracted similar data and suspected fault locations to the device, which then displays them on the user interface. When the user performs a diagnosis based on this, the device's built-in emotion engine analyzes the user's facial expressions and tone of voice to evaluate the reliability and stress level.

[0533] Feedback and Database Updates

[0534] The user inputs the diagnosis results (complete failure, partial failure, normal, etc.) into the device and sends them to the server. The emotion evaluation by the emotion engine is also recorded at the same time. The server stores the received feedback information in a database and uses it as training data to improve search accuracy in future searches.

[0535] The specific hardware and software used

[0536] Hardware

[0537] Camera for capturing images

[0538] Various sensors (vibration sensors, temperature sensors, sound sensors, etc.)

[0539] GPU (e.g., NVIDIA CUDA-enabled GPU) for accelerating image processing and AI model calculations

[0540] software

[0541] Scripting languages ​​for image processing and data analysis (e.g., Python)

[0542] Machine learning models (e.g., TensorFlow, scikit-learn)

[0543] Libraries for emotion recognition (e.g., Some Emotion Recognition Library)

[0544] Specific examples

[0545] Examples of data entry and saving

[0546] 1. The user (operator) inputs the image data and vibration sensor data of machine A in the factory into the terminal.

[0547] 2. The device sends this data to the server, which stores it in a database.

[0548] Example of searching for similar data and suspected fault locations

[0549] 3. Enter the newly acquired video data and temperature sensor data from Machine B into the terminal.

[0550] 4. The server receives the data sent from the device and performs a similarity search.

[0551] 5. Similar data is extracted and suspected fault locations are estimated.

[0552] Displaying the results and using the emotion engine

[0553] 6. The suspected faulty part and past similar cases will be displayed on the device screen.

[0554] 7. The emotion engine analyzes the user's facial expressions and tone of voice to assess their confidence and stress level.

[0555] Examples of feedback and database updates

[0556] 8. The user enters the definitive diagnosis and comments into the terminal and sends them to the server.

[0557] 9. Feedback information is stored in the database to improve search accuracy in future searches.

[0558] Example of input prompt for generative AI model

[0559] "We have taken maintenance footage of machine A in the factory. Abnormal vibrations have been detected by the vibration sensor. Please search for similar past abnormal data and identify the suspected fault location."

[0560] This is expected to not only improve the efficiency of machine maintenance and defective product inspection within factories and reduce the burden on operators, but also improve the reliability of diagnoses.

[0561] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0562] Step 1:

[0563] Users input video data of machines and products in the factory and various sensor data into the terminal. The terminal acquires this data and sends it to the server. The input data includes video of the machines taken by a camera and numerical data from vibration and temperature sensors.

[0564] Step 2:

[0565] The server stores the received video and sensor data in a database. When the data is stored, metadata (such as the date and time of storage and information about the operator in charge) is also recorded. The server then converts this data into a standard format so that it can be searched later.

[0566] Step 3:

[0567] The user inputs new machine video and sensor data into the device. The device then sends this data to the server. The new data includes video of the suspected malfunctioning machine and abnormal vibration and temperature values.

[0568] Step 4:

[0569] The server converts the new data it receives into a standard format and formats it so that it can be easily processed by the AI ​​model. Specifically, image data is resized and noise is removed, and sensor data is normalized and smoothed.

[0570] Step 5:

[0571] The server uses a generative AI model and a diffusion model to search the database for similar past video data and sensor data. The AI ​​model compares the features of the input data with those in the database and extracts data with high similarity. This extracted data includes data on past instances of machines exhibiting similar vibration or temperature abnormalities.

[0572] Step 6:

[0573] The server sends the extracted similar data and a list of related suspected fault locations to the terminal. The data on similar cases and fault locations is displayed on the terminal. The user performs a diagnosis based on this information.

[0574] Step 7:

[0575] The device's built-in emotion engine analyzes the user's facial expressions and tone of voice to assess the reliability of the diagnosis and stress level. The emotion engine uses the camera and microphone to analyze the user's emotional state in real time.

[0576] Step 8:

[0577] The user inputs the final diagnosis result into the terminal, which includes information on whether the machine is normal, partially broken, or completely broken. The terminal then sends this information to the server.

[0578] Step 9:

[0579] The server records the received feedback information in a database. The recorded feedback information is used as training data to improve search accuracy in future searches. Specifically, it is used to retrain the AI ​​model, enabling more accurate failure prediction.

[0580] The specific operations at each step ensure efficient operation of the entire system, improving the efficiency of machine maintenance and defective product inspection within the factory. Through this series of processes, the burden on operators is reduced and the reliability and accuracy of fault diagnosis is improved.

[0581] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0582] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0583] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0584] [Second embodiment]

[0585] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0586] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0587] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0588] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0589] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0590] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0591] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0592] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0593] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0594] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0595] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0596] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0597] The present invention relates to a diagnostic support system that uses medical images, basic information, and imaging findings. This system generates similar images and suspected disease names based on input of patient medical images and information by a doctor, and supports diagnosis. Specific embodiments of each element of this system are described below.

[0598] Data entry and saving

[0599] User (doctor): Enters the patient's medical images (e.g., CT scan), basic information such as age, gender, and medical history, as well as imaging findings (such as the location and shape of the lesion) into the terminal.

[0600] Terminal: Sends the entered information to the server.

[0601] Server: The received data is stored in a database. The stored data also includes metadata (date and time, information about the person who entered the data, etc.) at the time of input.

[0602] Search for similar images and suspected diseases

[0603] User (doctor): Enters medical images and basic information about a new patient who wishes to be diagnosed into the terminal, along with any imaging findings.

[0604] Terminal: Sends the entered data to the server.

[0605] Server: Receives the data and performs formatting processing, checking for missing data and converting it to a standard format.

[0606] Server: Using a generative AI model and a diffusion model, the server searches for similar images in the database. From the search results, it extracts images with high similarity and predicts the suspected disease names associated with them.

[0607] Results display and feedback

[0608] Server: Sends a list of similar images and suspected disease names to the terminal.

[0609] Terminal: Similar images and suspected disease names are displayed on the user interface.

[0610] User (doctor): Makes a diagnosis based on the information provided and enters the final diagnosis and its feedback into the terminal.

[0611] Terminal: Sends the input feedback information to the server.

[0612] Server: Updates the database based on the feedback information to improve search accuracy from the next time onwards.

[0613] Specific examples

[0614] 1. Examples of data entry and saving

[0615] User (doctor): Enters the chest X-ray image, age (45 years old), gender (male), and medical history (smoking history) of patient ID 12345 into the terminal form.

[0616] Terminal: Sends data to the server.

[0617] Server: Save the information of patient ID 12345 in the database.

[0618] 2. Example of searching for similar images and suspected diseases

[0619] User (doctor): Enters the chest X-ray image, age (65 years old), and gender (female) of new patient ID 67890 into the terminal.

[0620] Terminal: Sends data to the server.

[0621] Server: Uses a generative artificial intelligence model to search the database for similar past X-ray images. Extracts the most similar set of images from the previously stored images.

[0622] Server: Identifies the suspected disease name (e.g., pneumonia) related to the extracted image and sends it to the terminal.

[0623] 3. Examples of results and feedback

[0624] Terminal: The doctor's terminal displays a similar image of a chest X-ray and the suspected disease name "pneumonia."

[0625] User (doctor): Checks similar images and suspected disease names and confirms "pneumonia" as the definitive diagnosis for patient ID 67890.

[0626] User (doctor): Enters the definitive diagnosis and additional findings into the terminal and sends them to the server.

[0627] Server: Updates the database and accumulates feedback data to improve the accuracy of the next search.

[0628] In this way, the present invention allows doctors to easily search for similar images and obtain the name of a suspected disease when making a diagnosis, thereby improving the accuracy and efficiency of diagnosis.

[0629] The processing flow will be explained below.

[0630] Program processing flow

[0631] Data entry and saving

[0632] Step 1:

[0633] User (doctor): Enters basic information such as the patient's medical images (e.g., CT scan), age, gender, medical history, and imaging findings (such as the location and shape of the lesion) into a dedicated form on the terminal.

[0634] Step 2:

[0635] Terminal: Validate that the data entered is accurate and that all required fields are completed.

[0636] Step 3:

[0637] Terminal: Sends the verified data to the server.

[0638] Step 4:

[0639] Server: Double-checks the integrity of the data received and converts it into the appropriate format.

[0640] Step 5:

[0641] Server: Stores patient information, image data, and findings data in a database with unique identifiers.

[0642] Step 6:

[0643] Server: Adds metadata (such as the date and time of storage and information about the attending physician) to the stored data to improve future search performance.

[0644] Search for similar images and suspected diseases

[0645] Step 7:

[0646] User (doctor): Enters image data and basic information of a new patient for whom a diagnosis is desired into the terminal. If necessary, additional findings are also entered.

[0647] Step 8:

[0648] Terminal: Sends the entered data to the server.

[0649] Step 9:

[0650] Server: Validates the received data and checks for missing data.

[0651] Step 10:

[0652] Server: Converts image data into a standard format and formats it so that it can be easily processed by the AI ​​model.

[0653] Step 11:

[0654] Server: Using generative AI and diffusion models, the server searches for similar images in the database. The algorithm also takes into account the patient's basic information and findings.

[0655] Step 12:

[0656] Server: Extracts the most similar images from the search results and identifies the suspected disease associated with them.

[0657] Step 13:

[0658] Server: Sends similar images and the names of suspected diseases associated with them to the terminal.

[0659] Results display and feedback

[0660] Step 14:

[0661] Terminal: Displays the similar images and suspected disease names sent from the server on the user interface.

[0662] Step 15:

[0663] User (doctor): Check the displayed information and use it as a reference for diagnosis.

[0664] Step 16:

[0665] User (doctor): Enters the final diagnosis (e.g., confirmed disease name and additional findings) into the terminal and sends it to the server.

[0666] Step 17:

[0667] Server: Updates the database based on the feedback received, specifically by linking definitive diagnoses and additional findings to existing data to help improve the accuracy of future search results.

[0668] Step 18:

[0669] Server: The feedback information is also used as training data for the AI ​​model to improve its accuracy.

[0670] This detailed process flow will enable doctors to be more confident in their diagnoses, leading to more accurate and faster diagnoses. The system aims to improve the quality of healthcare by leveraging the power of automation and AI.

[0671] Example 1

[0672] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0673] In current medical settings, doctors often lack sufficient support to make accurate and prompt diagnoses. In particular, similar image searches and suspected disease diagnoses from medical images are often performed manually, resulting in time-consuming diagnosis and inconsistent diagnostic accuracy. A system that can solve these problems and support doctors' diagnoses is needed.

[0674] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0675] In this invention, the server includes: a means for inputting patient medical images; a means for inputting basic patient information and imaging findings; an information storage means for storing the medical images, basic information, and imaging findings; a transmission means for transmitting the input information to the server; a data formatting means for formatting the received data, checking for missing data, and converting it to a standard format; a means for searching for similar images from the information storage means using a generative artificial intelligence model and a diffusion model; a means for generating a suspected disease name based on the similar images, the basic information, and imaging findings; a means for displaying the similar images and the suspected disease name; a means for inputting a diagnosis result and feedback; a transmission means for transmitting the diagnosis result and feedback information to the server; and a means for updating the information storage means based on the feedback information to improve search accuracy from the next time onward. This allows doctors to easily search for similar images and obtain a suspected disease name when making a diagnosis.

[0676] "Patient medical images" refers to image data taken of a patient using medical equipment, including, specifically, CT scans, X-ray images, and MRI images.

[0677] "Basic information" refers to basic information about the patient, such as the patient's age, gender, and medical history.

[0678] "Imaging findings" refers to findings observed by a physician, such as the location or shape of a specific lesion or abnormality seen in a medical image.

[0679] "Information storage means" refers to a database or storage system for appropriately storing the aforementioned medical images, basic information, and image findings.

[0680] "Transmission means" refers to a communication protocol or interface for transmitting input information to a server.

[0681] "Data formatting means" refers to a function for formatting received data, checking for missing data, and converting it into a standard format.

[0682] A "generative artificial intelligence model" refers to a machine learning model that learns patterns and correlations from data and makes inferences and predictions based on the results.

[0683] A "diffusion model" is a type of generative artificial intelligence model that performs probabilistic sampling within a multidimensional data space to generate new data.

[0684] "Search means" refers to the function of searching for necessary information from information storage means, specifically an interface for performing database searches and searches for similar images.

[0685] "Means for generating" refers to the process or algorithm for calculating and generating a suspected disease name based on basic information and imaging findings.

[0686] "Displaying means" refers to a user interface for visually presenting similar images and suspected disease names to the user.

[0687] "Feedback information" refers to information including a doctor's diagnosis and additional findings, and includes data used to improve the system's search accuracy.

[0688] The present invention relates to a diagnostic support system that uses medical images, basic information, and image findings. This system generates similar images and suspected disease names and supports diagnosis by having a doctor input a patient's medical images and basic information. Specific embodiments of the present invention are described below.

[0689] System configuration

[0690] This system includes a terminal operated by a user (doctor) and a server connected to it. The terminal includes an input interface and a display interface, and the server includes a data storage means, a data formatting means, and a search means using a generative artificial intelligence model and a diffusion model.

[0691] Data entry and saving

[0692] The user (doctor) inputs the patient's medical images (e.g., CT scans), basic information such as age, gender, and medical history, as well as the imaging findings, into a terminal. The terminal then sends the input information to a server. The server then stores the received data in a database. At this time, metadata (date and time, information about the person who inputted, etc.) at the time of input is also stored.

[0693] Specific examples

[0694] For example, if a user inputs "Patient ID 12345, age 45, male, smoking history, chest X-ray image," the terminal sends this data to the server, which then stores this information in a database.

[0695] Search for similar images and suspected diseases

[0696] When a user (doctor) wishes to diagnose a new patient, they again input the medical images, basic information, and imaging findings into the terminal. The terminal then sends the input data to the server. The server then performs formatting on the received data, checks for missing data, and converts it into a standard format. A generative artificial intelligence model (using, for example, TensorFlow or PyTorch) and a diffusion model are used to search for similar images in the database. Images with high similarity are extracted from the search results, and suspected diseases related to them are estimated.

[0697] Specific examples

[0698] When a user inputs "Patient ID 67890, age 65, female, chest X-ray image," the device sends this data to the server. The server uses a generative artificial intelligence model to search a database for similar past X-ray images, extracts the most similar image set, and predicts the suspected disease name (e.g., "pneumonia").

[0699] Results display and feedback

[0700] The server sends a list of similar images and suspected disease names to the terminal. The terminal displays the similar images and suspected disease names on the user interface. The user (doctor) makes a diagnosis based on the provided information and enters a definitive diagnosis and its feedback into the terminal. The terminal sends the entered feedback information to the server. The server updates the database based on the feedback information, improving the accuracy of the next search.

[0701] Specific examples

[0702] For example, the terminal display shows "Similar Image 1," "Similar Image 2," and "Suspected Disease: Pneumonia." The user confirms this and enters "Patient ID 67890, Definitive Diagnosis: Pneumonia, Additional Findings: Shadow Spread." The terminal sends this feedback information to the server. The server updates the database and accumulates diagnosis history, improving search accuracy.

[0703] This system allows doctors to easily search for similar images and obtain the name of a suspected disease when making a diagnosis, improving the accuracy and efficiency of diagnosis.

[0704] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0705] Step 1:

[0706] The user (doctor) inputs the patient's medical images (e.g., CT scan), basic information such as age, gender, medical history, and imaging findings into the terminal. The input information is used as initial data for the medical diagnosis system. Specifically, the doctor inputs "Patient ID 12345, age 45, male, smoking history, chest X-ray image" into the input form on the terminal. Input is mainly done in the form of text fields and file uploads.

[0707] Step 2:

[0708] The terminal sends the input information to the server. A communication protocol such as an HTTP POST request is used for transmission. Specifically, when the send button is pressed, the terminal puts the input data into the body of an HTTP request and sends it to the specified endpoint on the server. This request includes the input information (patient ID, age, gender, medical history, and image data).

[0709] Step 3:

[0710] The server performs a formatting process on the received data. Data formatting includes checking for missing data and converting it to a standard format. For example, it checks whether required fields such as age and gender are missing, and converts it to JSON format. Specifically, the server analyzes the data and returns an error message if any information is missing. Once the formatting process is complete, the data is unified into a standard format. The input is the initial raw data, and the output is the formatted data.

[0711] Step 4:

[0712] The server stores the received and formatted data in a database. When storing the data, the metadata at the time of input (date and time, information about the person who entered, etc.) is also stored. Specifically, the server creates a database entry and stores the patient information appropriately. This process is performed using a specific database management system (DBMS). The input is formatted data, and the output is a record stored in the database.

[0713] Step 5:

[0714] The user (doctor) inputs medical images, basic information, and imaging findings of a new patient for whom they wish to diagnose into the terminal. Specifically, the doctor inputs, for example, "Patient ID 67890, age 65, female, chest X-ray image" into the terminal. The data input format is the same as a text field or file upload.

[0715] Step 6:

[0716] The terminal again sends the entered data to the server. The sending process is the same as in step 2, using an HTTP POST request. The input is the newly entered information about the patient seeking diagnosis, and the output is the request sent to the server.

[0717] Step 7:

[0718] The server uses a generative artificial intelligence model (using, for example, TensorFlow or PyTorch) and a diffusion model to search for similar images in the database. After data shaping, the input data is provided to the generative AI model, which calculates a similarity score. Based on this score, the most similar image set is extracted from the database. Specifically, the server runs the AI ​​model and compares previously saved images with newly input images. The input is the shaped new data, and the output is a set of highly similar images and a suspected disease name.

[0719] Step 8:

[0720] The server sends a list of estimated similar images and suspected disease names to the terminal. This is done again using an HTTP response. Specifically, the server returns the generated search results to the terminal in JSON format. The input is the output result of the AI ​​model, and the output is the data sent to the terminal.

[0721] Step 9:

[0722] The terminal displays similar images and the suspected disease name on the user interface. Specifically, the terminal analyzes the received data and displays information such as "Similar Image 1," "Similar Image 2," and "Suspected Disease Name: Pneumonia" on the screen. The input is the data sent from the server, and the output is the displayed user interface.

[0723] Step 10:

[0724] The user (doctor) makes a diagnosis based on the provided information and enters the definitive diagnosis and its feedback into the terminal. Specifically, the doctor enters "Patient ID 67890, definitive diagnosis: pneumonia, additional findings: shadow spreading" into the diagnosis result input form. The input is the diagnosis result and additional findings.

[0725] Step 11:

[0726] The terminal sends the entered feedback information to the server. The sending process again uses an HTTP POST request. The input is the doctor's final diagnosis and feedback information, and the output is the request sent to the server.

[0727] Step 12:

[0728] The server updates the database based on the feedback information and takes measures to improve search accuracy from the next time onwards. Specifically, the server analyzes the feedback information and updates the corresponding entries in the database. This enables diagnostic support to be performed with higher accuracy the next time the search is performed. The input is the feedback information and the output is the updated database.

[0729] (Application example 1)

[0730] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0731] Conventional diagnostic support systems have been used primarily in the medical field, making it difficult to apply them to other fields. Furthermore, abnormal behavior detection systems that use surveillance camera footage and sensor data lack effective means for detecting abnormal behavior in real time and responding quickly. Therefore, there is a need to apply the technology of diagnostic support systems used in the medical field to abnormal behavior detection systems that use surveillance camera footage and sensor data, thereby improving the accuracy of abnormal behavior detection.

[0732] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0733] In this invention, the server includes means for inputting patient medical images, means for inputting basic information and imaging findings of the patient, database means for storing the medical images, basic information, and imaging findings, means for searching for similar images from the database using a generative artificial intelligence model and a diffusion model, means for generating a suspected disease name based on the similar images and the basic information and imaging findings, means for presenting the similar images and suspected disease name to a user, means for inputting surveillance camera footage and sensor data, database means for storing the surveillance camera footage and sensor data, means for searching for similar behavior from the database using a generative artificial intelligence model, means for detecting abnormal behavior based on the similar behavior, the surveillance camera footage, and the sensor data, and means for presenting the abnormal behavior and the reason for the abnormal behavior to a user. This enables advanced abnormal behavior detection and diagnostic support in a variety of fields.

[0734] "Patient" refers to a person receiving medical services.

[0735] "Medical imaging" refers to image data, such as CT scans, X-rays, and MRI images, acquired to visualize the internal structures of a patient's body.

[0736] "Basic information" refers to basic information necessary for diagnosis and treatment, such as the patient's age, gender, and medical history.

[0737] "Imaging findings" refers to information and findings necessary for diagnosis that a doctor intuitively obtains based on medical images.

[0738] "Database means" refers to means for systematically storing, managing, and searching input data.

[0739] A "generative artificial intelligence model" refers to an artificial intelligence model that is trained to analyze data and generate new information.

[0740] A "diffusion model" refers to a model that spreads data to obtain accurate analysis results over a wide range.

[0741] "Similar images" refer to past images that have similar characteristics and features to the input medical image.

[0742] "Suspected disease name" refers to the name of a disease that is inferred based on similar images, basic information, and imaging findings.

[0743] "Surveillance camera footage" refers to video data captured by a surveillance camera.

[0744] "Sensor data" refers to data obtained from various sensors installed in a monitoring system.

[0745] "Similar behavior" refers to behavior that has similar characteristics to past behavioral data based on surveillance camera footage and sensor data.

[0746] "Abnormal behavior" refers to behavior that deviates from normal patterns of behavior and may pose a security problem.

[0747] "User" refers to the personnel and doctors who operate the system and use the information.

[0748] The present invention provides a system for assisting diagnosis using medical images and basic information of patients, and a system for detecting abnormal behavior using surveillance camera footage and sensor data. Specific embodiments of these systems are described below.

[0749] Diagnostic support system using medical images

[0750] First, a diagnosis support system using medical images and basic information will be described.

[0751] Data entry and saving

[0752] The user (doctor) inputs the patient's medical images (e.g., CT scans or X-ray images), basic information such as age, gender, and medical history, as well as imaging findings (such as the location and shape of the lesion) into a terminal. The terminal then sends this data to a server, which then stores the received data in a database. The stored data also includes metadata from the time of input (such as date and time and information about the person who entered the data).

[0753] Search for similar images and suspected diseases

[0754] The user (doctor) inputs medical images and basic information about a new patient they wish to diagnose into the terminal. The terminal then sends the input data to the server, which receives it and performs formatting processing. The data is checked for any missing parts and converted into a standard format. Next, a generative artificial intelligence model and a diffusion model are used to search for similar images in the database. Images with high similarity are extracted from the search results, and the suspected disease names associated with them are estimated.

[0755] Results display and feedback

[0756] The server sends a list of similar images and suspected diseases to the terminal, which displays them on the user interface. The user (doctor) makes a diagnosis based on the provided information and enters a definitive diagnosis and feedback into the terminal. The terminal then sends the entered feedback information to the server, which updates the database based on it.

[0757] As a specific example, a user (doctor) enters the chest X-ray image, age (45 years old), gender (male), and medical history (smoking history) of patient ID 12345 into a form on the terminal, and the terminal sends the data to the server and stores it.The user (doctor) then enters the chest X-ray image, age (65 years old), and gender (female) of a new patient ID 67890, and the server uses a generative artificial intelligence model to search for similar images from past data, identifies the suspected disease name, and sends it to the terminal.

[0758] Abnormal behavior detection system using surveillance camera footage and sensor data

[0759] Data entry and saving

[0760] Users (security personnel) input data into the system from surveillance camera footage, motion sensors, sound sensors, etc. The terminals send this data to the server, which then stores the received data in a database.

[0761] Abnormal behavior detection

[0762] The server uses a generative AI model to analyze past abnormal behavior data stored in a database and real-time surveillance camera footage and sensor data to detect abnormal behavior. The generative AI model uses YOLO (You Only Look Once) to analyze the video data and sensor data.

[0763] Results display and alerts

[0764] The server extracts video of suspected abnormal behavior and the reason for it, and issues an alert to the terminal. The video of suspected abnormal behavior and the reason for it are displayed on a user interface (for example, a web application built with React). The user (security officer) responds based on the information provided, and inputs feedback information (confirmation of abnormal behavior and how to deal with it) into the terminal and sends it to the server.

[0765] For example, if a surveillance camera detects an intrusion or abnormal activity, it uses a generative AI model to analyze the behavior and, if it determines that the behavior is abnormal, issues an alarm. Security personnel receive the alarm and can respond promptly.

[0766] Example prompt for a generative AI model:

[0767] "Detect abnormal behavior in surveillance camera footage in real time, compare it with past abnormal behavior data, and issue an alert. Abnormal behavior includes intrusions into specific areas or unusual movements."

[0768] This system will enable advanced abnormal behavior detection and diagnostic support not only in the medical field but also in the security field.

[0769] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0770] Diagnostic support system using medical images

[0771] Step 1:

[0772] Description of the process: The user (doctor) inputs the patient's medical images, basic information, and imaging findings into the terminal.

[0773] Input: Medical images (e.g., CT scan), patient age, gender, medical history, imaging findings

[0774] Data processing: Enter data into the form on the device

[0775] Output: Formatted data

[0776] Step 2:

[0777] Processing description: The terminal sends the input medical images, basic information, and image findings to the server.

[0778] Input: Formatted data (medical images, age, gender, medical history, imaging findings)

[0779] Data calculation: Packetize the data and send it to the server via the network

[0780] Output: Data received by the server

[0781] Step 3:

[0782] Process description: The server stores the received data in a database.

[0783] Input: The data received by the server

[0784] Data processing: Save the data to the corresponding record in the database

[0785] Output: Data stored in the database

[0786] Step 4:

[0787] Process description: The user (doctor) inputs medical images and basic information of a new patient into the terminal.

[0788] Input: New medical image, age, gender, medical history

[0789] Data processing: Enter data into the form on the device

[0790] Output: The new formatted data

[0791] Step 5:

[0792] Process description: The terminal sends the new data entered to the server.

[0793] Input: Formatted new data (medical images, age, gender, basic information, imaging findings)

[0794] Data calculation: Packetize the data and send it to the server via the network

[0795] Output: New data received by the server

[0796] Step 6:

[0797] Process Description: The server analyzes the new data received and searches for similar images in its database.

[0798] Input: New medical images and basic information

[0799] Data Computing: Searching for Similar Images Using Generative Artificial Intelligence Models and Diffusion Models

[0800] Output: A list of similar images

[0801] Step 7:

[0802] Process description: The server sends similar images and the suspected disease name to the terminal.

[0803] Input: A list of similar images

[0804] Data calculation: Estimation of suspected disease and data packetization

[0805] Output: List of similar images received by the device and suspected disease name

[0806] Step 8:

[0807] Process description: The terminal displays similar images and suspected disease names on the user interface.

[0808] Input: List of similar images and suspected disease name

[0809] Data processing: Display on the user interface

[0810] Output: Information that the user (doctor) can check

[0811] Step 9:

[0812] Process description: The user (doctor) inputs the definitive diagnosis and feedback information into the terminal.

[0813] Input: definitive diagnosis, additional findings

[0814] Data processing: Enter data into the form on the device

[0815] Output: Formatted feedback data

[0816] Step 10:

[0817] Process description: The terminal sends feedback data to the server.

[0818] Input: Formatted feedback data

[0819] Data calculation: Feedback data is packetized and sent to the server via the network

[0820] Output: Feedback data received by the server

[0821] Step 11:

[0822] Process description: The server stores the feedback data in a database to improve search accuracy for future searches.

[0823] Input: Received feedback data

[0824] Data processing: Update the database

[0825] Output: Updated database

[0826] Abnormal behavior detection system using surveillance camera footage and sensor data

[0827] Step 1:

[0828] Process description: The user (security officer) inputs surveillance camera footage and sensor data into the system.

[0829] Input: Surveillance camera footage, motion sensor data, sound sensor data

[0830] Data processing: Input into the system within the terminal

[0831] Output: Formatted monitoring data

[0832] Step 2:

[0833] Process description: The terminal sends the input monitoring data to the server.

[0834] Input: Formatted monitoring data

[0835] Data calculation: Packetize the data and send it to the server via the network

[0836] Output: Monitoring data received by the server

[0837] Step 3:

[0838] Process description: The server stores the received data in a database.

[0839] Input: Received monitoring data

[0840] Data processing: Save the data to the corresponding record in the database

[0841] Output: Monitoring data stored in a database

[0842] Step 4:

[0843] Process description: The server analyzes historical abnormal behavior data and real-time monitoring data in the database using a generative artificial intelligence model (e.g., YOLO).

[0844] Input: Real-time monitoring data, historical abnormal behavior data

[0845] Data Computation: A Comparative Analysis Using Generative Artificial Intelligence Models

[0846] Output: Abnormal behavior detection results

[0847] Step 5:

[0848] Processing description: When the server detects abnormal behavior, it sends alarm data including the reason to the terminal.

[0849] Input: Abnormal behavior detection result, reason

[0850] Data Computation: Alarm data generation and packetization

[0851] Output: Alarm data received by the terminal

[0852] Step 6:

[0853] Process Description: The terminal displays the alarm data on the user interface.

[0854] Input: Alarm data

[0855] Data processing: Display on the user interface

[0856] Output: Alert information that can be checked by the user (security officer)

[0857] Step 7:

[0858] Process description: The user (security officer) inputs the confirmed abnormal behavior and the response into the terminal.

[0859] Input: Confirmed abnormal behavior, response details

[0860] Data processing: Enter data into the form on the device

[0861] Output: Formatted feedback data

[0862] Step 8:

[0863] Process description: The terminal sends feedback data to the server.

[0864] Input: Formatted feedback data

[0865] Data calculation: Feedback data is packetized and sent to the server via the network

[0866] Output: Feedback data received by the server

[0867] Step 9:

[0868] Processing description: The server stores the received feedback data in a database to improve detection accuracy in the future.

[0869] Input: Received feedback data

[0870] Data processing: Update the database

[0871] Output: Updated database

[0872] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0873] The present invention relates to a diagnostic support system that uses medical images, basic information, and imaging findings. This system generates similar images and suspected disease names when a doctor inputs a patient's medical images and information, and also recognizes the user's emotions to support diagnosis. Specific embodiments of each element of this system are described below.

[0874] Data entry and saving

[0875] User (doctor): Enters the patient's medical images (e.g., CT scan), basic information such as age, gender, and medical history, as well as imaging findings (such as the location and shape of the lesion) into the terminal.

[0876] Terminal: Sends the entered information to the server.

[0877] Server: The received data is stored in a database, and the stored data also includes metadata at the time of input (such as the date and time of storage and information about the attending physician).

[0878] Search for similar images and suspected diseases

[0879] User (doctor): Enters medical images and basic information about a new patient who wishes to be diagnosed into the terminal, along with any imaging findings.

[0880] Terminal: Sends the entered data to the server.

[0881] Server: Validates the received data, checks for missing data, and converts the image data into a standard format that is easy for the AI ​​model to process.

[0882] Server: Using a generative AI model and a diffusion model, the server searches for similar images in the database. From the search results, it extracts images with high similarity and predicts the suspected disease names associated with them.

[0883] Displaying results and using the emotion engine

[0884] Server: Sends a list of similar images and suspected disease names to the terminal.

[0885] Terminal: Similar images and suspected disease names are displayed on the user interface.

[0886] User (doctor): A diagnosis is made based on the information provided. The device's built-in emotion engine recognizes the user's emotions from facial expressions and tone of voice, and evaluates their reliability and stress level.

[0887] Feedback and Database Updates

[0888] User (doctor): Enters the final diagnosis (e.g., confirmed disease name and additional findings) into the terminal. The emotion evaluation by the emotion engine is also recorded at the same time.

[0889] Terminal: Sends the input feedback information to the server.

[0890] Server: Updates the database based on feedback information to improve search accuracy from the next time onwards. Also, adjusts the display method and presentation order based on the evaluation results of the emotion engine, allowing users to use information more efficiently.

[0891] Specific examples

[0892] 1. Examples of data entry and saving

[0893] User (doctor): Enters the chest X-ray image, age (45 years old), gender (male), and medical history (smoking history) of patient ID 12345 into the terminal.

[0894] Terminal: Sends data to the server.

[0895] Server: Save the information of patient ID 12345 in the database.

[0896] 2. Example of searching for similar images and suspected diseases

[0897] User (doctor): Enters the chest X-ray image, age (65 years old), and gender (female) of new patient ID 67890 into the terminal.

[0898] Terminal: Sends data to the server.

[0899] Server: Uses a generative artificial intelligence model to search the database for similar past X-ray images. Extracts the most similar set of images from the previously stored images.

[0900] Server: Identifies the suspected disease name (e.g., pneumonia) related to the extracted image and sends it to the terminal.

[0901] 3. Displaying the results and using the emotion engine

[0902] Terminal: The doctor's terminal displays a similar image of a chest X-ray and the suspected disease name "pneumonia."

[0903] Emotion engine: Analyzes the user's facial expressions and tone of voice to assess their confidence in the diagnosis and stress level.

[0904] User (doctor): Checks similar images and suspected disease names, and in a stable emotional state, confirms that "pneumonia" is the definitive diagnosis for patient ID 67890.

[0905] User (doctor): Enters the definitive diagnosis, additional findings, and emotional assessment during the diagnosis into the terminal and sends them to the server.

[0906] 4. Examples of feedback and database updates

[0907] Server: The received feedback information is stored in a database to improve search and emotion evaluation accuracy in the future.

[0908] Server: Based on the feedback information, the AI ​​model is trained and diagnostic support functions are enhanced.

[0909] In this way, the present invention allows doctors to easily search for similar images and identify suspected diseases when making diagnoses. Furthermore, by using an emotion engine, it becomes possible to evaluate the confidence in a diagnosis and the stress level, enabling more appropriate information to be provided. This not only improves the accuracy and efficiency of diagnoses, but also contributes to reducing the burden on doctors.

[0910] The processing flow will be explained below.

[0911] Program processing flow: Data entry and saving

[0912] Step 1:

[0913] User (doctor): Enters basic information such as the patient's medical images (e.g., CT scan), age, gender, medical history, and imaging findings (such as the location and shape of the lesion) into a dedicated form on the terminal.

[0914] Step 2:

[0915] Terminal: Validate that the data entered is accurate and that all required fields are completed.

[0916] Step 3:

[0917] Terminal: Sends the verified data to the server.

[0918] Step 4:

[0919] Server: Double-checks the integrity of the data received and converts it into the appropriate format.

[0920] Step 5:

[0921] Server: Stores patient information, image data, and findings data in a database with unique identifiers.

[0922] Step 6:

[0923] Server: Adds metadata (such as the date and time of storage and information about the attending physician) to the stored data to improve future search performance.

[0924] Program processing flow: Search for similar images and suspected disease names

[0925] Step 7:

[0926] User (doctor): Enters image data and basic information of a new patient for whom a diagnosis is desired into the terminal. If necessary, additional findings are also entered.

[0927] Step 8:

[0928] Terminal: Sends the entered data to the server.

[0929] Step 9:

[0930] Server: Validates the received data and checks for missing data.

[0931] Step 10:

[0932] Server: Converts image data into a standard format and formats it so that it can be easily processed by the AI ​​model.

[0933] Step 11:

[0934] Server: Using generative AI and diffusion models, the server searches for similar images in the database. The algorithm also takes into account the patient's basic information and findings.

[0935] Step 12:

[0936] Server: Extracts the most similar images from the search results and identifies the suspected disease associated with them.

[0937] Step 13:

[0938] Server: Sends similar images and the names of suspected diseases associated with them to the terminal.

[0939] Program processing flow: "Using the emotion engine"

[0940] Step 14:

[0941] Terminal: Displays the similar images and suspected disease names sent from the server on the user interface.

[0942] Step 15:

[0943] Emotion engine: Analyzes the user's (doctor's) facial expressions and tone of voice in real time to assess their confidence in the diagnosis and stress level.

[0944] Step 16:

[0945] User (doctor): Checks the displayed information and makes a diagnostic decision while taking into account feedback from the emotion engine. If the user feels stressed during the process, the emotion engine automatically displays a diagnostic support message.

[0946] Program flow: Feedback and database updates

[0947] Step 17:

[0948] User (doctor): Enters the final diagnosis (e.g., confirmed disease name and additional findings) into the terminal. The emotion evaluation by the emotion engine is also recorded at the same time.

[0949] Step 18:

[0950] Terminal: Sends the input feedback information to the server.

[0951] Step 19:

[0952] Server: Updates the database based on the feedback received, specifically by linking definitive diagnoses and additional findings to existing data to help improve the accuracy of future search results.

[0953] Step 20:

[0954] Server: Based on the feedback information and the evaluation results of the emotion engine, it is also used as training data for the AI ​​model to improve the accuracy of the model.

[0955] This detailed processing flow allows doctors to easily search for similar images and identify suspected diseases when making a diagnosis, and the emotional engine's support improves the reliability of the diagnosis, leading to reduced stress. This increases the accuracy and efficiency of diagnoses, and greatly contributes to reducing the burden on medical professionals.

[0956] Example 2

[0957] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0958] In modern medical settings, rapid and accurate diagnosis using medical images is required for patient diagnosis. However, it is not easy to efficiently handle large amounts of image data during the diagnostic process and accurately identify suspected diseases. Furthermore, the doctor's emotions and stress can affect the accuracy of the diagnosis, which is also a major challenge. Therefore, there is a need for a diagnostic support system that not only quickly searches for similar images and provides an appropriate suspected disease name, but also takes the doctor's emotional state into account.

[0959] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0960] In this invention, the server includes: means for inputting patient medical images; means for inputting basic patient information and imaging findings; database means for storing the medical images, basic information, and imaging findings; means for searching the database for similar images using a generative artificial intelligence model and a diffusion model; means for generating suspected disease names based on the similar images, basic information, and imaging findings; means for presenting the similar images and suspected disease names to the user; emotion engine means for analyzing the user's facial expression and tone of voice to evaluate reliability and stress level; means for inputting diagnostic feedback and storing it together with an emotion evaluation; and means for updating the database based on the stored feedback information to improve the accuracy of subsequent searches and emotion evaluations. This enables rapid and accurate diagnostic support and also enables the provision of appropriate information that takes into account the doctor's emotional state.

[0961] "Patient medical images" means image data of a patient used by a physician for diagnostic or treatment purposes, including formats such as X-rays, CT scans, and MRIs.

[0962] "Basic information" refers to basic data necessary for diagnosis and treatment, such as the patient's age, gender, and medical history.

[0963] "Imaging findings" are data showing detailed observation results, such as the location and shape of lesions identified by doctors based on diagnostic imaging.

[0964] "Database Means" means a computer system for storing and managing patient medical images, baseline information, and imaging findings.

[0965] A "generative artificial intelligence model" is a collection of algorithms that generate new data based on trained data and perform classification and prediction.

[0966] The "diffusion model" is a mathematical technique for identifying similar images by diffusing image data in a high-dimensional space and performing similarity calculations.

[0967] The "emotion engine means" is a software module for analyzing a user's facial expressions and tone of voice to assess their emotional state, such as their trustworthiness or stress level.

[0968] "Diagnostic feedback" refers to data entered by doctors regarding diagnostic results and additional findings, which is used to improve the system's learning accuracy and to support future diagnoses.

[0969] "Feedback information" refers to data such as diagnostic feedback and evaluation results from the emotion engine that the system uses to improve search accuracy and the user interface in future searches.

[0970] "Search and emotion assessment accuracy" refers to the accuracy with which the system identifies similar images and related disease names, and the accuracy with which the emotion engine assesses the user's emotional state.

[0971] MODE FOR CARRYING OUT THE INVENTION

[0972] The present invention relates to a diagnostic support system that uses a patient's medical images, basic information, and imaging findings. This system generates similar images and suspected disease names when a doctor inputs the patient's medical images and information, and further supports diagnosis by recognizing the user's emotions. Specific embodiments of each element of the present invention are described below.

[0973] System configuration

[0974] This system mainly consists of the following elements:

[0975] User (doctor): Enters the patient's medical images, basic information, and imaging findings.

[0976] Terminal: Accepts user input, sends data to the server, and displays the results.

[0977] Server: Stores data in a database, searches for similar images using a generative artificial intelligence model and a diffusion model, and generates a suspected disease name.

[0978] Database means: Stores medical images, basic information, and imaging findings.

[0979] System operation procedures

[0980] Data entry and saving

[0981] 1. The user (doctor) inputs the patient's medical images (e.g., CT scans or X-ray images), basic information such as age, gender, and medical history, as well as imaging findings (such as the location and shape of the lesion) into the terminal.

[0982] Example: Chest X-ray image of patient ID 12345, age 45, gender male, and smoking history as medical history are entered into the terminal.

[0983] 2. The device sends the entered information to the server using a secure protocol such as HTTPS.

[0984] 3. The server stores the received data in a database. When storing the data, metadata (such as the date and time of storage and the ID of the attending physician) is also recorded. The database engine used is MySQL, for example.

[0985] Search for similar images and suspected diseases

[0986] 4. The user (doctor) inputs the medical images and basic information of the new patient, as well as imaging findings if necessary, into the terminal.

[0987] Example: Enter the chest X-ray image of patient ID 67890, age 65, and gender female into the terminal.

[0988] 5. The terminal sends the entered data back to the server.

[0989] 6. The server validates the received data and checks for missing data. It converts the image data into a standard format and formats it for the AI ​​model. For example, OpenCV is used as the image processing library.

[0990] 7. The server uses a generative artificial intelligence model (e.g., GAN, Diffusion Models) to search for similar images in the database and estimate the suspected disease name.

[0991] Example prompt: "Search the database of historical X-ray images for the image most similar to the chest X-ray image of patient ID 67890 and identify any suspected related diseases."

[0992] Displaying results and using the emotion engine

[0993] 8. The server sends the similar images and the suspected disease names to the terminal.

[0994] 9. The terminal displays the received similar images and suspected disease names on the user interface.

[0995] Example: Similar images and the suspected disease name "pneumonia" are displayed on the device.

[0996] 10. The user (doctor) makes a diagnosis based on the provided information. During this time, the emotion engine installed on the device analyzes the doctor's facial expressions and tone of voice to evaluate his / her trustworthiness and stress level.

[0997] Feedback and Database Updates

[0998] 11. The user (doctor) enters the final diagnosis into the terminal and simultaneously records the emotional evaluation.

[0999] Example: Enter a definitive diagnosis of "pneumonia," additional findings, and an emotional assessment of "high confidence."

[1000] 12. The terminal transmits the input feedback information to the server.

[1001] 13. The server updates the database based on the feedback information to improve the accuracy of future searches and the evaluation of the emotion engine.

[1002] Example: A new feedback record is added to the database, and retraining of the AI ​​model is automatically triggered in the backend.

[1003] This system allows doctors to easily search for similar images and identify the appropriate suspected disease when making a diagnosis. Furthermore, by utilizing an emotion engine, it is possible to assess confidence in the diagnosis and stress levels, and provide optimal information. This improves the accuracy and efficiency of diagnoses and also contributes to reducing the burden on doctors.

[1004] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1005] Step 1:

[1006] The user (doctor) uses the terminal to input the patient's medical images (e.g., CT scans, X-ray images), basic information such as age, gender, and medical history, as well as the image findings.

[1007] Input: Patient's medical images and basic information, imaging findings

[1008] Output: Data entered into the terminal

[1009] Specific operation: The doctor enters the necessary information into the input form on the terminal and uploads the image file. For example, the doctor enters the chest X-ray image of patient ID 12345, age 45, gender male, and smoking history as medical history, and uploads the image.

[1010] Step 2:

[1011] The device collects the data entered by the user and sends it to a server using a secure protocol such as HTTPS.

[1012] Input: Data entered into the terminal

[1013] Output: Data sent to the server

[1014] Specific operation: When the send button is pressed, the device encodes the data and sends it to the server via HTTPS. The data includes the patient's medical images, basic information, and imaging findings.

[1015] Step 3:

[1016] The server stores the received data in a database, along with metadata such as the date and time of storage and information about the attending physician.

[1017] Input: Data sent to the server

[1018] Output: Data stored in the database

[1019] Specific operation: The server connects to a database engine (e.g., MySQL), creates a new patient record, and saves the data, including the patient's medical images, basic information, imaging findings, save date and time, and attending physician.

[1020] Step 4:

[1021] The user (doctor) inputs medical images and basic information of a new patient, as well as imaging findings if necessary, into the terminal.

[1022] Input: Medical images and basic information of new patients, imaging findings

[1023] Output: Data entered into the terminal

[1024] Specific operation: The doctor enters the chest X-ray image of new patient ID 67890, age 65, and gender female into the input form on the terminal and uploads the image.

[1025] Step 5:

[1026] The terminal transmits the newly input data to the server again.

[1027] Input: Data entered into the terminal

[1028] Output: Data sent to the server

[1029] Specific operation: When you press the send button, the device encodes the data and sends it back to the server via HTTPS.

[1030] Step 6:

[1031] The server validates the received data, checking for missing data, then converts the image data into a standard format and prepares it for the AI ​​model.

[1032] Input: Data sent to the server

[1033] Output: Validated data and formatted image data

[1034] What happens: A validation script on the server checks the integrity of the data and converts the images to a standard format using an image processing library (e.g. OpenCV).

[1035] Step 7:

[1036] The server uses a generative artificial intelligence model and a diffusion model to search for similar images in a database and estimate the suspected disease name.

[1037] Input: Validated data and formatted image data

[1038] Output: Similar images and suspected disease names

[1039] Specific operation: A generative AI model (e.g., GAN, Diffusion Models) analyzes image data and performs similarity calculations. The associated disease name is inferred from the metadata of the most similar image. An example prompt is, "Search the historical X-ray image database for the image most similar to the chest X-ray image of patient ID 67890 and identify the suspected associated disease name."

[1040] Step 8:

[1041] The server transmits the similar image and the suspected disease name to the terminal.

[1042] Input: Similar images and suspected disease name

[1043] Output: Similar images and suspected disease names sent to the terminal

[1044] Specific operation: The server encodes the search results and the estimated disease name in JSON format and sends them to the terminal.

[1045] Step 9:

[1046] The terminal displays the received similar images and suspected disease names on the user interface.

[1047] Input: Similar images and suspected disease names sent to the terminal

[1048] Output: Displayed similar images and suspected disease names

[1049] Specific operation: Highly relevant similar images and the estimated disease name "pneumonia" are displayed on the device interface.

[1050] Step 10:

[1051] The user (doctor) makes a diagnosis based on the provided information, while the device's built-in emotion engine analyzes facial expressions and tone of voice to assess trustworthiness and stress level.

[1052] Input: Displayed similar images, suspected disease names, facial expressions and tone of voice during the user's diagnostic activity

[1053] Output: Emotion engine estimates confidence and stress level

[1054] How it works: During the diagnosis, the emotion engine collects and analyzes data from the webcam and microphone in real time, assessing the confidence and stress level of the user as they enter their diagnosis information.

[1055] Step 11:

[1056] The user (doctor) inputs the final diagnosis and emotional evaluation into the terminal.

[1057] Input: Final diagnosis and emotional evaluation

[1058] Output: Feedback and emotion rating data entered into the device

[1059] Specific operation: Enter the definitive diagnosis, additional findings, and emotional assessment into the input form on the device. Example: In addition to the definitive diagnosis of "pneumonia," enter "high confidence."

[1060] Step 12:

[1061] The terminal transmits the input feedback information to the server.

[1062] Input: Feedback and emotional rating data entered into the device

[1063] Output: Feedback and sentiment rating data sent to the server

[1064] Specific operation: When you press the send button, the device encodes the data and sends it to the server.

[1065] Step 13:

[1066] The server stores the received feedback information in a database to improve the accuracy of future searches and emotion evaluations.

[1067] Input: Feedback and sentiment rating data sent to the server

[1068] Output: An updated database and a trained AI model

[1069] Specific operation: The server adds new feedback records to the database and automatically initiates retraining of the AI ​​model to improve search accuracy and sentiment assessment accuracy.

[1070] (Application example 2)

[1071] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1072] Current machine maintenance and defective product inspection in factories places a heavy burden on operators, and anomaly detection and diagnosis require time and effort. Furthermore, there is a lack of appropriate information provision and diagnostic feedback based on the operator's emotions and stress level, which reduces work efficiency and diagnostic reliability. Therefore, there is a need for a system that reduces the burden on operators and improves diagnostic accuracy and work efficiency.

[1073] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1074] In this invention, the server includes means for inputting video data and various sensor data of machines and products, database means for storing the video data and various sensor data, and means for generating suspected fault locations and abnormalities of machines and products using the video data and various sensor data. This reduces the burden on operators and improves diagnostic accuracy and work efficiency during machine maintenance and defective product inspection in factories.

[1075] "Machinery and products" refers to the equipment used in the factory and the goods manufactured.

[1076] "Video data" refers to image information captured by a photographing device such as a camera.

[1077] "Sensor data" refers to information obtained from various sensors that detect vibration, temperature, sound, etc.

[1078] A "database" is a system that stores and manages input video data, sensor data, and associated metadata.

[1079] A "generative artificial intelligence model" is an artificial intelligence that uses machine learning technology to generate new information from data.

[1080] A "diffusion model" is an algorithm that discovers patterns and trends from large amounts of data and analyzes relationships in a diffuse manner.

[1081] A "similar image" is an image among the video data stored in the database that has characteristics similar to the newly input video data.

[1082] A "suspected disease name" is a disease name that may be inferred based on medical images and basic information.

[1083] A "suspected failure point" is a part or component of a machine or product where an abnormality is suspected.

[1084] An "emotion engine" is a system that analyzes a user's emotional responses, such as facial expressions and tone of voice, to assess their trustworthiness and stress level.

[1085] A "user" is someone who operates the system and checks the results.

[1086] This invention relates to a system that supports machine maintenance and defective product inspection in factories. This system inputs video data and various sensor data to detect failures and abnormalities early, improving diagnostic accuracy and work efficiency. It also includes an emotion engine that recognizes the emotions of operators and evaluates the reliability of diagnoses and stress levels.

[1087] System Configuration and Operation

[1088] Data entry and saving

[1089] The user (operator) inputs video data of machines and products in the factory and multiple sensor data (e.g., vibration, temperature, sound) into a dedicated terminal. The terminal sends this input information to a server, which then stores the received data in a database. This data also includes metadata at the time of input (e.g., the date and time of saving and information about the operator in charge).

[1090] Search for similar data and suspected fault locations

[1091] The user inputs newly acquired video data and sensor data into the device. This data is then sent back to the server, where it is first converted into a standard format and formatted so that the AI ​​model can easily process it. The server then uses a generative AI model and a diffusion model to search the database for similar past data. As a result of the search, data with high similarity is extracted, and related suspected fault locations and anomalies are inferred.

[1092] Displaying results and using the emotion engine

[1093] The server sends the extracted similar data and suspected fault locations to the device, which then displays them on the user interface. When the user performs a diagnosis based on this, the device's built-in emotion engine analyzes the user's facial expressions and tone of voice to evaluate the reliability and stress level.

[1094] Feedback and Database Updates

[1095] The user inputs the diagnosis results (complete failure, partial failure, normal, etc.) into the device and sends them to the server. The emotion evaluation by the emotion engine is also recorded at the same time. The server stores the received feedback information in a database and uses it as training data to improve search accuracy in future searches.

[1096] The specific hardware and software used

[1097] Hardware

[1098] Camera for capturing images

[1099] Various sensors (vibration sensors, temperature sensors, sound sensors, etc.)

[1100] GPU (e.g., NVIDIA CUDA-enabled GPU) for accelerating image processing and AI model calculations

[1101] software

[1102] Scripting languages ​​for image processing and data analysis (e.g., Python)

[1103] Machine learning models (e.g., TensorFlow, scikit-learn)

[1104] Libraries for emotion recognition (e.g., Some Emotion Recognition Library)

[1105] Specific examples

[1106] Examples of data entry and saving

[1107] 1. The user (operator) inputs the image data and vibration sensor data of machine A in the factory into the terminal.

[1108] 2. The device sends this data to the server, which stores it in a database.

[1109] Example of searching for similar data and suspected fault locations

[1110] 3. Enter the newly acquired video data and temperature sensor data from Machine B into the terminal.

[1111] 4. The server receives the data sent from the device and performs a similarity search.

[1112] 5. Similar data is extracted and suspected fault locations are estimated.

[1113] Displaying the results and using the emotion engine

[1114] 6. The suspected faulty part and past similar cases will be displayed on the device screen.

[1115] 7. The emotion engine analyzes the user's facial expressions and tone of voice to assess their confidence and stress level.

[1116] Examples of feedback and database updates

[1117] 8. The user enters the definitive diagnosis and comments into the terminal and sends them to the server.

[1118] 9. Feedback information is stored in the database to improve search accuracy in future searches.

[1119] Example of input prompt for generative AI model

[1120] "We have taken maintenance footage of machine A in the factory. Abnormal vibrations have been detected by the vibration sensor. Please search for similar past abnormal data and identify the suspected fault location."

[1121] This is expected to not only improve the efficiency of machine maintenance and defective product inspection within factories and reduce the burden on operators, but also improve the reliability of diagnoses.

[1122] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1123] Step 1:

[1124] Users input video data of machines and products in the factory and various sensor data into the terminal. The terminal acquires this data and sends it to the server. The input data includes video of the machines taken by a camera and numerical data from vibration and temperature sensors.

[1125] Step 2:

[1126] The server stores the received video and sensor data in a database. When the data is stored, metadata (such as the date and time of storage and information about the operator in charge) is also recorded. The server then converts this data into a standard format so that it can be searched later.

[1127] Step 3:

[1128] The user inputs new machine video and sensor data into the device. The device then sends this data to the server. The new data includes video of the suspected malfunctioning machine and abnormal vibration and temperature values.

[1129] Step 4:

[1130] The server converts the new data it receives into a standard format and formats it so that it can be easily processed by the AI ​​model. Specifically, image data is resized and noise is removed, and sensor data is normalized and smoothed.

[1131] Step 5:

[1132] The server uses a generative AI model and a diffusion model to search the database for similar past video data and sensor data. The AI ​​model compares the features of the input data with those in the database and extracts data with high similarity. This extracted data includes data on past instances of machines exhibiting similar vibration or temperature abnormalities.

[1133] Step 6:

[1134] The server sends the extracted similar data and a list of related suspected fault locations to the terminal. The data on similar cases and fault locations is displayed on the terminal. The user performs a diagnosis based on this information.

[1135] Step 7:

[1136] The device's built-in emotion engine analyzes the user's facial expressions and tone of voice to assess the reliability of the diagnosis and stress level. The emotion engine uses the camera and microphone to analyze the user's emotional state in real time.

[1137] Step 8:

[1138] The user inputs the final diagnosis result into the terminal, which includes information on whether the machine is normal, partially broken, or completely broken. The terminal then sends this information to the server.

[1139] Step 9:

[1140] The server records the received feedback information in a database. The recorded feedback information is used as training data to improve search accuracy in future searches. Specifically, it is used to retrain the AI ​​model, enabling more accurate failure prediction.

[1141] The specific operations at each step ensure efficient operation of the entire system, improving the efficiency of machine maintenance and defective product inspection within the factory. Through this series of processes, the burden on operators is reduced and the reliability and accuracy of fault diagnosis is improved.

[1142] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1143] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1144] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1145] [Third embodiment]

[1146] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1147] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1148] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1149] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1150] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1151] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1152] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1153] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1154] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1155] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1156] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1157] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1158] The present invention relates to a diagnostic support system that uses medical images, basic information, and imaging findings. This system generates similar images and suspected disease names based on input of patient medical images and information by a doctor, and supports diagnosis. Specific embodiments of each element of this system are described below.

[1159] Data entry and saving

[1160] User (doctor): Enters the patient's medical images (e.g., CT scan), basic information such as age, gender, and medical history, as well as imaging findings (such as the location and shape of the lesion) into the terminal.

[1161] Terminal: Sends the entered information to the server.

[1162] Server: The received data is stored in a database. The stored data also includes metadata (date and time, information about the person who entered the data, etc.) at the time of input.

[1163] Search for similar images and suspected diseases

[1164] User (doctor): Enters medical images and basic information about a new patient who wishes to be diagnosed into the terminal, along with any imaging findings.

[1165] Terminal: Sends the entered data to the server.

[1166] Server: Receives the data and performs formatting processing, checking for missing data and converting it to a standard format.

[1167] Server: Using a generative AI model and a diffusion model, the server searches for similar images in the database. From the search results, it extracts images with high similarity and predicts the suspected disease names associated with them.

[1168] Results display and feedback

[1169] Server: Sends a list of similar images and suspected disease names to the terminal.

[1170] Terminal: Similar images and suspected disease names are displayed on the user interface.

[1171] User (doctor): Makes a diagnosis based on the information provided and enters the final diagnosis and its feedback into the terminal.

[1172] Terminal: Sends the input feedback information to the server.

[1173] Server: Updates the database based on the feedback information to improve search accuracy from the next time onwards.

[1174] Specific examples

[1175] 1. Examples of data entry and saving

[1176] User (doctor): Enters the chest X-ray image, age (45 years old), gender (male), and medical history (smoking history) of patient ID 12345 into the terminal form.

[1177] Terminal: Sends data to the server.

[1178] Server: Save the information of patient ID 12345 in the database.

[1179] 2. Example of searching for similar images and suspected diseases

[1180] User (doctor): Enters the chest X-ray image, age (65 years old), and gender (female) of new patient ID 67890 into the terminal.

[1181] Terminal: Sends data to the server.

[1182] Server: Uses a generative artificial intelligence model to search the database for similar past X-ray images. Extracts the most similar set of images from the previously stored images.

[1183] Server: Identifies the suspected disease name (e.g., pneumonia) related to the extracted image and sends it to the terminal.

[1184] 3. Examples of results and feedback

[1185] Terminal: The doctor's terminal displays a similar image of a chest X-ray and the suspected disease name "pneumonia."

[1186] User (doctor): Checks similar images and suspected disease names and confirms "pneumonia" as the definitive diagnosis for patient ID 67890.

[1187] User (doctor): Enters the definitive diagnosis and additional findings into the terminal and sends them to the server.

[1188] Server: Updates the database and accumulates feedback data to improve the accuracy of the next search.

[1189] In this way, the present invention allows doctors to easily search for similar images and obtain the name of a suspected disease when making a diagnosis, thereby improving the accuracy and efficiency of diagnosis.

[1190] The processing flow will be explained below.

[1191] Program processing flow

[1192] Data entry and saving

[1193] Step 1:

[1194] User (doctor): Enters basic information such as the patient's medical images (e.g., CT scan), age, gender, medical history, and imaging findings (such as the location and shape of the lesion) into a dedicated form on the terminal.

[1195] Step 2:

[1196] Terminal: Validate that the data entered is accurate and that all required fields are completed.

[1197] Step 3:

[1198] Terminal: Sends the verified data to the server.

[1199] Step 4:

[1200] Server: Double-checks the integrity of the data received and converts it into the appropriate format.

[1201] Step 5:

[1202] Server: Stores patient information, image data, and findings data in a database with unique identifiers.

[1203] Step 6:

[1204] Server: Adds metadata (such as the date and time of storage and information about the attending physician) to the stored data to improve future search performance.

[1205] Search for similar images and suspected diseases

[1206] Step 7:

[1207] User (doctor): Enters image data and basic information of a new patient for whom a diagnosis is desired into the terminal. If necessary, additional findings are also entered.

[1208] Step 8:

[1209] Terminal: Sends the entered data to the server.

[1210] Step 9:

[1211] Server: Validates the received data and checks for missing data.

[1212] Step 10:

[1213] Server: Converts image data into a standard format and formats it so that it can be easily processed by the AI ​​model.

[1214] Step 11:

[1215] Server: Using generative AI and diffusion models, the server searches for similar images in the database. The algorithm also takes into account the patient's basic information and findings.

[1216] Step 12:

[1217] Server: Extracts the most similar images from the search results and identifies the suspected disease associated with them.

[1218] Step 13:

[1219] Server: Sends similar images and the names of suspected diseases associated with them to the terminal.

[1220] Results display and feedback

[1221] Step 14:

[1222] Terminal: Displays the similar images and suspected disease names sent from the server on the user interface.

[1223] Step 15:

[1224] User (doctor): Check the displayed information and use it as a reference for diagnosis.

[1225] Step 16:

[1226] User (doctor): Enters the final diagnosis (e.g., confirmed disease name and additional findings) into the terminal and sends it to the server.

[1227] Step 17:

[1228] Server: Updates the database based on the feedback received, specifically by linking definitive diagnoses and additional findings to existing data to help improve the accuracy of future search results.

[1229] Step 18:

[1230] Server: The feedback information is also used as training data for the AI ​​model to improve its accuracy.

[1231] This detailed process flow will enable doctors to be more confident in their diagnoses, leading to more accurate and faster diagnoses. The system aims to improve the quality of healthcare by leveraging the power of automation and AI.

[1232] Example 1

[1233] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1234] In current medical settings, doctors often lack sufficient support to make accurate and prompt diagnoses. In particular, similar image searches and suspected disease diagnoses from medical images are often performed manually, resulting in time-consuming diagnosis and inconsistent diagnostic accuracy. A system that can solve these problems and support doctors' diagnoses is needed.

[1235] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1236] In this invention, the server includes: a means for inputting patient medical images; a means for inputting basic patient information and imaging findings; an information storage means for storing the medical images, basic information, and imaging findings; a transmission means for transmitting the input information to the server; a data formatting means for formatting the received data, checking for missing data, and converting it to a standard format; a means for searching for similar images from the information storage means using a generative artificial intelligence model and a diffusion model; a means for generating a suspected disease name based on the similar images, the basic information, and imaging findings; a means for displaying the similar images and the suspected disease name; a means for inputting a diagnosis result and feedback; a transmission means for transmitting the diagnosis result and feedback information to the server; and a means for updating the information storage means based on the feedback information to improve search accuracy from the next time onward. This allows doctors to easily search for similar images and obtain a suspected disease name when making a diagnosis.

[1237] "Patient medical images" refers to image data taken of a patient using medical equipment, including, specifically, CT scans, X-ray images, and MRI images.

[1238] "Basic information" refers to basic information about the patient, such as the patient's age, gender, and medical history.

[1239] "Imaging findings" refers to findings observed by a physician, such as the location or shape of a specific lesion or abnormality seen in a medical image.

[1240] "Information storage means" refers to a database or storage system for appropriately storing the aforementioned medical images, basic information, and image findings.

[1241] "Transmission means" refers to a communication protocol or interface for transmitting input information to a server.

[1242] "Data formatting means" refers to a function for formatting received data, checking for missing data, and converting it into a standard format.

[1243] A "generative artificial intelligence model" refers to a machine learning model that learns patterns and correlations from data and makes inferences and predictions based on the results.

[1244] A "diffusion model" is a type of generative artificial intelligence model that performs probabilistic sampling within a multidimensional data space to generate new data.

[1245] "Search means" refers to the function of searching for necessary information from information storage means, specifically an interface for performing database searches and searches for similar images.

[1246] "Means for generating" refers to the process or algorithm for calculating and generating a suspected disease name based on basic information and imaging findings.

[1247] "Displaying means" refers to a user interface for visually presenting similar images and suspected disease names to the user.

[1248] "Feedback information" refers to information including a doctor's diagnosis and additional findings, and includes data used to improve the system's search accuracy.

[1249] The present invention relates to a diagnostic support system that uses medical images, basic information, and image findings. This system generates similar images and suspected disease names and supports diagnosis by having a doctor input a patient's medical images and basic information. Specific embodiments of the present invention are described below.

[1250] System configuration

[1251] This system includes a terminal operated by a user (doctor) and a server connected to it. The terminal includes an input interface and a display interface, and the server includes a data storage means, a data formatting means, and a search means using a generative artificial intelligence model and a diffusion model.

[1252] Data entry and saving

[1253] The user (doctor) inputs the patient's medical images (e.g., CT scans), basic information such as age, gender, and medical history, as well as the imaging findings, into a terminal. The terminal then sends the input information to a server. The server then stores the received data in a database. At this time, metadata (date and time, information about the person who inputted, etc.) at the time of input is also stored.

[1254] Specific examples

[1255] For example, if a user inputs "Patient ID 12345, age 45, male, smoking history, chest X-ray image," the terminal sends this data to the server, which then stores this information in a database.

[1256] Search for similar images and suspected diseases

[1257] When a user (doctor) wishes to diagnose a new patient, they again input the medical images, basic information, and imaging findings into the terminal. The terminal then sends the input data to the server. The server then performs formatting on the received data, checks for missing data, and converts it into a standard format. A generative artificial intelligence model (using, for example, TensorFlow or PyTorch) and a diffusion model are used to search for similar images in the database. Images with high similarity are extracted from the search results, and suspected diseases related to them are estimated.

[1258] Specific examples

[1259] When a user inputs "Patient ID 67890, age 65, female, chest X-ray image," the device sends this data to the server. The server uses a generative artificial intelligence model to search a database for similar past X-ray images, extracts the most similar image set, and predicts the suspected disease name (e.g., "pneumonia").

[1260] Results display and feedback

[1261] The server sends a list of similar images and suspected disease names to the terminal. The terminal displays the similar images and suspected disease names on the user interface. The user (doctor) makes a diagnosis based on the provided information and enters a definitive diagnosis and its feedback into the terminal. The terminal sends the entered feedback information to the server. The server updates the database based on the feedback information, improving the accuracy of the next search.

[1262] Specific examples

[1263] For example, the terminal display shows "Similar Image 1," "Similar Image 2," and "Suspected Disease: Pneumonia." The user confirms this and enters "Patient ID 67890, Definitive Diagnosis: Pneumonia, Additional Findings: Shadow Spread." The terminal sends this feedback information to the server. The server updates the database and accumulates diagnosis history, improving search accuracy.

[1264] This system allows doctors to easily search for similar images and obtain the name of a suspected disease when making a diagnosis, improving the accuracy and efficiency of diagnosis.

[1265] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1266] Step 1:

[1267] The user (doctor) inputs the patient's medical images (e.g., CT scan), basic information such as age, gender, medical history, and imaging findings into the terminal. The input information is used as initial data for the medical diagnosis system. Specifically, the doctor inputs "Patient ID 12345, age 45, male, smoking history, chest X-ray image" into the input form on the terminal. Input is mainly done in the form of text fields and file uploads.

[1268] Step 2:

[1269] The terminal sends the input information to the server. A communication protocol such as an HTTP POST request is used for transmission. Specifically, when the send button is pressed, the terminal puts the input data into the body of an HTTP request and sends it to the specified endpoint on the server. This request includes the input information (patient ID, age, gender, medical history, and image data).

[1270] Step 3:

[1271] The server performs a formatting process on the received data. Data formatting includes checking for missing data and converting it to a standard format. For example, it checks whether required fields such as age and gender are missing, and converts it to JSON format. Specifically, the server analyzes the data and returns an error message if any information is missing. Once the formatting process is complete, the data is unified into a standard format. The input is the initial raw data, and the output is the formatted data.

[1272] Step 4:

[1273] The server stores the received and formatted data in a database. When storing the data, the metadata at the time of input (date and time, information about the person who entered, etc.) is also stored. Specifically, the server creates a database entry and stores the patient information appropriately. This process is performed using a specific database management system (DBMS). The input is formatted data, and the output is a record stored in the database.

[1274] Step 5:

[1275] The user (doctor) inputs medical images, basic information, and imaging findings of a new patient for whom they wish to diagnose into the terminal. Specifically, the doctor inputs, for example, "Patient ID 67890, age 65, female, chest X-ray image" into the terminal. The data input format is the same as a text field or file upload.

[1276] Step 6:

[1277] The terminal again sends the entered data to the server. The sending process is the same as in step 2, using an HTTP POST request. The input is the newly entered information about the patient seeking diagnosis, and the output is the request sent to the server.

[1278] Step 7:

[1279] The server uses a generative artificial intelligence model (using, for example, TensorFlow or PyTorch) and a diffusion model to search for similar images in the database. After data shaping, the input data is provided to the generative AI model, which calculates a similarity score. Based on this score, the most similar image set is extracted from the database. Specifically, the server runs the AI ​​model and compares previously saved images with newly input images. The input is the shaped new data, and the output is a set of highly similar images and a suspected disease name.

[1280] Step 8:

[1281] The server sends a list of estimated similar images and suspected disease names to the terminal. This is done again using an HTTP response. Specifically, the server returns the generated search results to the terminal in JSON format. The input is the output result of the AI ​​model, and the output is the data sent to the terminal.

[1282] Step 9:

[1283] The terminal displays similar images and the suspected disease name on the user interface. Specifically, the terminal analyzes the received data and displays information such as "Similar Image 1," "Similar Image 2," and "Suspected Disease Name: Pneumonia" on the screen. The input is the data sent from the server, and the output is the displayed user interface.

[1284] Step 10:

[1285] The user (doctor) makes a diagnosis based on the provided information and enters the definitive diagnosis and its feedback into the terminal. Specifically, the doctor enters "Patient ID 67890, definitive diagnosis: pneumonia, additional findings: shadow spreading" into the diagnosis result input form. The input is the diagnosis result and additional findings.

[1286] Step 11:

[1287] The terminal sends the entered feedback information to the server. The sending process again uses an HTTP POST request. The input is the doctor's final diagnosis and feedback information, and the output is the request sent to the server.

[1288] Step 12:

[1289] The server updates the database based on the feedback information and takes measures to improve search accuracy from the next time onwards. Specifically, the server analyzes the feedback information and updates the corresponding entries in the database. This enables diagnostic support to be performed with higher accuracy the next time the search is performed. The input is the feedback information and the output is the updated database.

[1290] (Application example 1)

[1291] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1292] Conventional diagnostic support systems have been used primarily in the medical field, making it difficult to apply them to other fields. Furthermore, abnormal behavior detection systems that use surveillance camera footage and sensor data lack effective means for detecting abnormal behavior in real time and responding quickly. Therefore, there is a need to apply the technology of diagnostic support systems used in the medical field to abnormal behavior detection systems that use surveillance camera footage and sensor data, thereby improving the accuracy of abnormal behavior detection.

[1293] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1294] In this invention, the server includes means for inputting patient medical images, means for inputting basic information and imaging findings of the patient, database means for storing the medical images, basic information, and imaging findings, means for searching for similar images from the database using a generative artificial intelligence model and a diffusion model, means for generating a suspected disease name based on the similar images and the basic information and imaging findings, means for presenting the similar images and suspected disease name to a user, means for inputting surveillance camera footage and sensor data, database means for storing the surveillance camera footage and sensor data, means for searching for similar behavior from the database using a generative artificial intelligence model, means for detecting abnormal behavior based on the similar behavior, the surveillance camera footage, and the sensor data, and means for presenting the abnormal behavior and the reason for the abnormal behavior to a user. This enables advanced abnormal behavior detection and diagnostic support in a variety of fields.

[1295] "Patient" refers to a person receiving medical services.

[1296] "Medical imaging" refers to image data, such as CT scans, X-rays, and MRI images, acquired to visualize the internal structures of a patient's body.

[1297] "Basic information" refers to basic information necessary for diagnosis and treatment, such as the patient's age, gender, and medical history.

[1298] "Imaging findings" refers to information and findings necessary for diagnosis that a doctor intuitively obtains based on medical images.

[1299] "Database means" refers to means for systematically storing, managing, and searching input data.

[1300] A "generative artificial intelligence model" refers to an artificial intelligence model that is trained to analyze data and generate new information.

[1301] A "diffusion model" refers to a model that spreads data to obtain accurate analysis results over a wide range.

[1302] "Similar images" refer to past images that have similar characteristics and features to the input medical image.

[1303] "Suspected disease name" refers to the name of a disease that is inferred based on similar images, basic information, and imaging findings.

[1304] "Surveillance camera footage" refers to video data captured by a surveillance camera.

[1305] "Sensor data" refers to data obtained from various sensors installed in a monitoring system.

[1306] "Similar behavior" refers to behavior that has similar characteristics to past behavioral data based on surveillance camera footage and sensor data.

[1307] "Abnormal behavior" refers to behavior that deviates from normal patterns of behavior and may pose a security problem.

[1308] "User" refers to the personnel and doctors who operate the system and use the information.

[1309] The present invention provides a system for assisting diagnosis using medical images and basic information of patients, and a system for detecting abnormal behavior using surveillance camera footage and sensor data. Specific embodiments of these systems are described below.

[1310] Diagnostic support system using medical images

[1311] First, a diagnosis support system using medical images and basic information will be described.

[1312] Data entry and saving

[1313] The user (doctor) inputs the patient's medical images (e.g., CT scans or X-ray images), basic information such as age, gender, and medical history, as well as imaging findings (such as the location and shape of the lesion) into a terminal. The terminal then sends this data to a server, which then stores the received data in a database. The stored data also includes metadata from the time of input (such as date and time and information about the person who entered the data).

[1314] Search for similar images and suspected diseases

[1315] The user (doctor) inputs medical images and basic information about a new patient they wish to diagnose into the terminal. The terminal then sends the input data to the server, which receives it and performs formatting processing. The data is checked for any missing parts and converted into a standard format. Next, a generative artificial intelligence model and a diffusion model are used to search for similar images in the database. Images with high similarity are extracted from the search results, and the suspected disease names associated with them are estimated.

[1316] Results display and feedback

[1317] The server sends a list of similar images and suspected diseases to the terminal, which displays them on the user interface. The user (doctor) makes a diagnosis based on the provided information and enters a definitive diagnosis and feedback into the terminal. The terminal then sends the entered feedback information to the server, which updates the database based on it.

[1318] As a specific example, a user (doctor) enters the chest X-ray image, age (45 years old), gender (male), and medical history (smoking history) of patient ID 12345 into a form on the terminal, and the terminal sends the data to the server and stores it.The user (doctor) then enters the chest X-ray image, age (65 years old), and gender (female) of a new patient ID 67890, and the server uses a generative artificial intelligence model to search for similar images from past data, identifies the suspected disease name, and sends it to the terminal.

[1319] Abnormal behavior detection system using surveillance camera footage and sensor data

[1320] Data entry and saving

[1321] Users (security personnel) input data into the system from surveillance camera footage, motion sensors, sound sensors, etc. The terminals send this data to the server, which then stores the received data in a database.

[1322] Abnormal behavior detection

[1323] The server uses a generative AI model to analyze past abnormal behavior data stored in a database and real-time surveillance camera footage and sensor data to detect abnormal behavior. The generative AI model uses YOLO (You Only Look Once) to analyze the video data and sensor data.

[1324] Results display and alerts

[1325] The server extracts video of suspected abnormal behavior and the reason for it, and issues an alert to the terminal. The video of suspected abnormal behavior and the reason for it are displayed on a user interface (for example, a web application built with React). The user (security officer) responds based on the information provided, and inputs feedback information (confirmation of abnormal behavior and how to deal with it) into the terminal and sends it to the server.

[1326] For example, if a surveillance camera detects an intrusion or abnormal activity, it uses a generative AI model to analyze the behavior and, if it determines that the behavior is abnormal, issues an alarm. Security personnel receive the alarm and can respond promptly.

[1327] Example prompt for a generative AI model:

[1328] "Detect abnormal behavior in surveillance camera footage in real time, compare it with past abnormal behavior data, and issue an alert. Abnormal behavior includes intrusions into specific areas or unusual movements."

[1329] This system will enable advanced abnormal behavior detection and diagnostic support not only in the medical field but also in the security field.

[1330] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1331] Diagnostic support system using medical images

[1332] Step 1:

[1333] Description of the process: The user (doctor) inputs the patient's medical images, basic information, and imaging findings into the terminal.

[1334] Input: Medical images (e.g., CT scan), patient age, gender, medical history, imaging findings

[1335] Data processing: Enter data into the form on the device

[1336] Output: Formatted data

[1337] Step 2:

[1338] Processing description: The terminal sends the input medical images, basic information, and image findings to the server.

[1339] Input: Formatted data (medical images, age, gender, medical history, imaging findings)

[1340] Data calculation: Packetize the data and send it to the server via the network

[1341] Output: Data received by the server

[1342] Step 3:

[1343] Process description: The server stores the received data in a database.

[1344] Input: The data received by the server

[1345] Data processing: Save the data to the corresponding record in the database

[1346] Output: Data stored in the database

[1347] Step 4:

[1348] Process description: The user (doctor) inputs medical images and basic information of a new patient into the terminal.

[1349] Input: New medical image, age, gender, medical history

[1350] Data processing: Enter data into the form on the device

[1351] Output: The new formatted data

[1352] Step 5:

[1353] Process description: The terminal sends the new data entered to the server.

[1354] Input: Formatted new data (medical images, age, gender, basic information, imaging findings)

[1355] Data calculation: Packetize the data and send it to the server via the network

[1356] Output: New data received by the server

[1357] Step 6:

[1358] Process Description: The server analyzes the new data received and searches for similar images in its database.

[1359] Input: New medical images and basic information

[1360] Data Computing: Searching for Similar Images Using Generative Artificial Intelligence Models and Diffusion Models

[1361] Output: A list of similar images

[1362] Step 7:

[1363] Process description: The server sends similar images and the suspected disease name to the terminal.

[1364] Input: A list of similar images

[1365] Data calculation: Estimation of suspected disease and data packetization

[1366] Output: List of similar images received by the device and suspected disease name

[1367] Step 8:

[1368] Process description: The terminal displays similar images and suspected disease names on the user interface.

[1369] Input: List of similar images and suspected disease name

[1370] Data processing: Display on the user interface

[1371] Output: Information that the user (doctor) can check

[1372] Step 9:

[1373] Process description: The user (doctor) inputs the definitive diagnosis and feedback information into the terminal.

[1374] Input: definitive diagnosis, additional findings

[1375] Data processing: Enter data into the form on the device

[1376] Output: Formatted feedback data

[1377] Step 10:

[1378] Process description: The terminal sends feedback data to the server.

[1379] Input: Formatted feedback data

[1380] Data calculation: Feedback data is packetized and sent to the server via the network

[1381] Output: Feedback data received by the server

[1382] Step 11:

[1383] Process description: The server stores the feedback data in a database to improve search accuracy for future searches.

[1384] Input: Received feedback data

[1385] Data processing: Update the database

[1386] Output: Updated database

[1387] Abnormal behavior detection system using surveillance camera footage and sensor data

[1388] Step 1:

[1389] Process description: The user (security officer) inputs surveillance camera footage and sensor data into the system.

[1390] Input: Surveillance camera footage, motion sensor data, sound sensor data

[1391] Data processing: Input into the system within the terminal

[1392] Output: Formatted monitoring data

[1393] Step 2:

[1394] Process description: The terminal sends the input monitoring data to the server.

[1395] Input: Formatted monitoring data

[1396] Data calculation: Packetize the data and send it to the server via the network

[1397] Output: Monitoring data received by the server

[1398] Step 3:

[1399] Process description: The server stores the received data in a database.

[1400] Input: Received monitoring data

[1401] Data processing: Save the data to the corresponding record in the database

[1402] Output: Monitoring data stored in a database

[1403] Step 4:

[1404] Process description: The server analyzes historical abnormal behavior data and real-time monitoring data in the database using a generative artificial intelligence model (e.g., YOLO).

[1405] Input: Real-time monitoring data, historical abnormal behavior data

[1406] Data Computation: A Comparative Analysis Using Generative Artificial Intelligence Models

[1407] Output: Abnormal behavior detection results

[1408] Step 5:

[1409] Processing description: When the server detects abnormal behavior, it sends alarm data including the reason to the terminal.

[1410] Input: Abnormal behavior detection result, reason

[1411] Data Computation: Alarm data generation and packetization

[1412] Output: Alarm data received by the terminal

[1413] Step 6:

[1414] Process Description: The terminal displays the alarm data on the user interface.

[1415] Input: Alarm data

[1416] Data processing: Display on the user interface

[1417] Output: Alert information that can be checked by the user (security officer)

[1418] Step 7:

[1419] Process description: The user (security officer) inputs the confirmed abnormal behavior and the response into the terminal.

[1420] Input: Confirmed abnormal behavior, response details

[1421] Data processing: Enter data into the form on the device

[1422] Output: Formatted feedback data

[1423] Step 8:

[1424] Process description: The terminal sends feedback data to the server.

[1425] Input: Formatted feedback data

[1426] Data calculation: Feedback data is packetized and sent to the server via the network

[1427] Output: Feedback data received by the server

[1428] Step 9:

[1429] Processing description: The server stores the received feedback data in a database to improve detection accuracy in the future.

[1430] Input: Received feedback data

[1431] Data processing: Update the database

[1432] Output: Updated database

[1433] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1434] The present invention relates to a diagnostic support system that uses medical images, basic information, and imaging findings. This system generates similar images and suspected disease names when a doctor inputs a patient's medical images and information, and also recognizes the user's emotions to support diagnosis. Specific embodiments of each element of this system are described below.

[1435] Data entry and saving

[1436] User (doctor): Enters the patient's medical images (e.g., CT scan), basic information such as age, gender, and medical history, as well as imaging findings (such as the location and shape of the lesion) into the terminal.

[1437] Terminal: Sends the entered information to the server.

[1438] Server: The received data is stored in a database, and the stored data also includes metadata at the time of input (such as the date and time of storage and information about the attending physician).

[1439] Search for similar images and suspected diseases

[1440] User (doctor): Enters medical images and basic information about a new patient who wishes to be diagnosed into the terminal, along with any imaging findings.

[1441] Terminal: Sends the entered data to the server.

[1442] Server: Validates the received data, checks for missing data, and converts the image data into a standard format that is easy for the AI ​​model to process.

[1443] Server: Using a generative AI model and a diffusion model, the server searches for similar images in the database. From the search results, it extracts images with high similarity and predicts the suspected disease names associated with them.

[1444] Displaying results and using the emotion engine

[1445] Server: Sends a list of similar images and suspected disease names to the terminal.

[1446] Terminal: Similar images and suspected disease names are displayed on the user interface.

[1447] User (doctor): A diagnosis is made based on the information provided. The device's built-in emotion engine recognizes the user's emotions from facial expressions and tone of voice, and evaluates their reliability and stress level.

[1448] Feedback and Database Updates

[1449] User (doctor): Enters the final diagnosis (e.g., confirmed disease name and additional findings) into the terminal. The emotion evaluation by the emotion engine is also recorded at the same time.

[1450] Terminal: Sends the input feedback information to the server.

[1451] Server: Updates the database based on feedback information to improve search accuracy from the next time onwards. Also, adjusts the display method and presentation order based on the evaluation results of the emotion engine, allowing users to use information more efficiently.

[1452] Specific examples

[1453] 1. Examples of data entry and saving

[1454] User (doctor): Enters the chest X-ray image, age (45 years old), gender (male), and medical history (smoking history) of patient ID 12345 into the terminal.

[1455] Terminal: Sends data to the server.

[1456] Server: Save the information of patient ID 12345 in the database.

[1457] 2. Example of searching for similar images and suspected diseases

[1458] User (doctor): Enters the chest X-ray image, age (65 years old), and gender (female) of new patient ID 67890 into the terminal.

[1459] Terminal: Sends data to the server.

[1460] Server: Uses a generative artificial intelligence model to search the database for similar past X-ray images. Extracts the most similar set of images from the previously stored images.

[1461] Server: Identifies the suspected disease name (e.g., pneumonia) related to the extracted image and sends it to the terminal.

[1462] 3. Displaying the results and using the emotion engine

[1463] Terminal: The doctor's terminal displays a similar image of a chest X-ray and the suspected disease name "pneumonia."

[1464] Emotion engine: Analyzes the user's facial expressions and tone of voice to assess their confidence in the diagnosis and stress level.

[1465] User (doctor): Checks similar images and suspected disease names, and in a stable emotional state, confirms that "pneumonia" is the definitive diagnosis for patient ID 67890.

[1466] User (doctor): Enters the definitive diagnosis, additional findings, and emotional assessment during the diagnosis into the terminal and sends them to the server.

[1467] 4. Examples of feedback and database updates

[1468] Server: The received feedback information is stored in a database to improve search and emotion evaluation accuracy in the future.

[1469] Server: Based on the feedback information, the AI ​​model is trained and diagnostic support functions are enhanced.

[1470] In this way, the present invention allows doctors to easily search for similar images and identify suspected diseases when making diagnoses. Furthermore, by using an emotion engine, it becomes possible to evaluate the confidence in a diagnosis and the stress level, enabling more appropriate information to be provided. This not only improves the accuracy and efficiency of diagnoses, but also contributes to reducing the burden on doctors.

[1471] The processing flow will be explained below.

[1472] Program processing flow: Data entry and saving

[1473] Step 1:

[1474] User (doctor): Enters basic information such as the patient's medical images (e.g., CT scan), age, gender, medical history, and imaging findings (such as the location and shape of the lesion) into a dedicated form on the terminal.

[1475] Step 2:

[1476] Terminal: Validate that the data entered is accurate and that all required fields are completed.

[1477] Step 3:

[1478] Terminal: Sends the verified data to the server.

[1479] Step 4:

[1480] Server: Double-checks the integrity of the data received and converts it into the appropriate format.

[1481] Step 5:

[1482] Server: Stores patient information, image data, and findings data in a database with unique identifiers.

[1483] Step 6:

[1484] Server: Adds metadata (such as the date and time of storage and information about the attending physician) to the stored data to improve future search performance.

[1485] Program processing flow: Search for similar images and suspected disease names

[1486] Step 7:

[1487] User (doctor): Enters image data and basic information of a new patient for whom a diagnosis is desired into the terminal. If necessary, additional findings are also entered.

[1488] Step 8:

[1489] Terminal: Sends the entered data to the server.

[1490] Step 9:

[1491] Server: Validates the received data and checks for missing data.

[1492] Step 10:

[1493] Server: Converts image data into a standard format and formats it so that it can be easily processed by the AI ​​model.

[1494] Step 11:

[1495] Server: Using generative AI and diffusion models, the server searches for similar images in the database. The algorithm also takes into account the patient's basic information and findings.

[1496] Step 12:

[1497] Server: Extracts the most similar images from the search results and identifies the suspected disease associated with them.

[1498] Step 13:

[1499] Server: Sends similar images and the names of suspected diseases associated with them to the terminal.

[1500] Program processing flow: "Using the emotion engine"

[1501] Step 14:

[1502] Terminal: Displays the similar images and suspected disease names sent from the server on the user interface.

[1503] Step 15:

[1504] Emotion engine: Analyzes the user's (doctor's) facial expressions and tone of voice in real time to assess their confidence in the diagnosis and stress level.

[1505] Step 16:

[1506] User (doctor): Checks the displayed information and makes a diagnostic decision while taking into account feedback from the emotion engine. If the user feels stressed during the process, the emotion engine automatically displays a diagnostic support message.

[1507] Program flow: Feedback and database updates

[1508] Step 17:

[1509] User (doctor): Enters the final diagnosis (e.g., confirmed disease name and additional findings) into the terminal. The emotion evaluation by the emotion engine is also recorded at the same time.

[1510] Step 18:

[1511] Terminal: Sends the input feedback information to the server.

[1512] Step 19:

[1513] Server: Updates the database based on the feedback received, specifically by linking definitive diagnoses and additional findings to existing data to help improve the accuracy of future search results.

[1514] Step 20:

[1515] Server: Based on the feedback information and the evaluation results of the emotion engine, it is also used as training data for the AI ​​model to improve the accuracy of the model.

[1516] This detailed processing flow allows doctors to easily search for similar images and identify suspected diseases when making a diagnosis, and the emotional engine's support improves the reliability of the diagnosis, leading to reduced stress. This increases the accuracy and efficiency of diagnoses, and greatly contributes to reducing the burden on medical professionals.

[1517] Example 2

[1518] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1519] In modern medical settings, rapid and accurate diagnosis using medical images is required for patient diagnosis. However, it is not easy to efficiently handle large amounts of image data during the diagnostic process and accurately identify suspected diseases. Furthermore, the doctor's emotions and stress can affect the accuracy of the diagnosis, which is also a major challenge. Therefore, there is a need for a diagnostic support system that not only quickly searches for similar images and provides an appropriate suspected disease name, but also takes the doctor's emotional state into account.

[1520] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1521] In this invention, the server includes: means for inputting patient medical images; means for inputting basic patient information and imaging findings; database means for storing the medical images, basic information, and imaging findings; means for searching the database for similar images using a generative artificial intelligence model and a diffusion model; means for generating suspected disease names based on the similar images, basic information, and imaging findings; means for presenting the similar images and suspected disease names to the user; emotion engine means for analyzing the user's facial expression and tone of voice to evaluate reliability and stress level; means for inputting diagnostic feedback and storing it together with an emotion evaluation; and means for updating the database based on the stored feedback information to improve the accuracy of subsequent searches and emotion evaluations. This enables rapid and accurate diagnostic support and also enables the provision of appropriate information that takes into account the doctor's emotional state.

[1522] "Patient medical images" means image data of a patient used by a physician for diagnostic or treatment purposes, including formats such as X-rays, CT scans, and MRIs.

[1523] "Basic information" refers to basic data necessary for diagnosis and treatment, such as the patient's age, gender, and medical history.

[1524] "Imaging findings" are data showing detailed observation results, such as the location and shape of lesions identified by doctors based on diagnostic imaging.

[1525] "Database Means" means a computer system for storing and managing patient medical images, baseline information, and imaging findings.

[1526] A "generative artificial intelligence model" is a collection of algorithms that generate new data based on trained data and perform classification and prediction.

[1527] The "diffusion model" is a mathematical technique for identifying similar images by diffusing image data in a high-dimensional space and performing similarity calculations.

[1528] The "emotion engine means" is a software module for analyzing a user's facial expressions and tone of voice to assess their emotional state, such as their trustworthiness or stress level.

[1529] "Diagnostic feedback" refers to data entered by doctors regarding diagnostic results and additional findings, which is used to improve the system's learning accuracy and to support future diagnoses.

[1530] "Feedback information" refers to data such as diagnostic feedback and evaluation results from the emotion engine that the system uses to improve search accuracy and the user interface in future searches.

[1531] "Search and emotion assessment accuracy" refers to the accuracy with which the system identifies similar images and related disease names, and the accuracy with which the emotion engine assesses the user's emotional state.

[1532] MODE FOR CARRYING OUT THE INVENTION

[1533] The present invention relates to a diagnostic support system that uses a patient's medical images, basic information, and imaging findings. This system generates similar images and suspected disease names when a doctor inputs the patient's medical images and information, and further supports diagnosis by recognizing the user's emotions. Specific embodiments of each element of the present invention are described below.

[1534] System configuration

[1535] This system mainly consists of the following elements:

[1536] User (doctor): Enters the patient's medical images, basic information, and imaging findings.

[1537] Terminal: Accepts user input, sends data to the server, and displays the results.

[1538] Server: Stores data in a database, searches for similar images using a generative artificial intelligence model and a diffusion model, and generates a suspected disease name.

[1539] Database means: Stores medical images, basic information, and imaging findings.

[1540] System operation procedures

[1541] Data entry and saving

[1542] 1. The user (doctor) inputs the patient's medical images (e.g., CT scans or X-ray images), basic information such as age, gender, and medical history, as well as imaging findings (such as the location and shape of the lesion) into the terminal.

[1543] Example: Chest X-ray image of patient ID 12345, age 45, gender male, and smoking history as medical history are entered into the terminal.

[1544] 2. The device sends the entered information to the server using a secure protocol such as HTTPS.

[1545] 3. The server stores the received data in a database. When storing the data, metadata (such as the date and time of storage and the ID of the attending physician) is also recorded. The database engine used is MySQL, for example.

[1546] Search for similar images and suspected diseases

[1547] 4. The user (doctor) inputs the medical images and basic information of the new patient, as well as imaging findings if necessary, into the terminal.

[1548] Example: Enter the chest X-ray image of patient ID 67890, age 65, and gender female into the terminal.

[1549] 5. The terminal sends the entered data back to the server.

[1550] 6. The server validates the received data and checks for missing data. It converts the image data into a standard format and formats it for the AI ​​model. For example, OpenCV is used as the image processing library.

[1551] 7. The server uses a generative artificial intelligence model (e.g., GAN, Diffusion Models) to search for similar images in the database and estimate the suspected disease name.

[1552] Example prompt: "Search the database of historical X-ray images for the image most similar to the chest X-ray image of patient ID 67890 and identify any suspected related diseases."

[1553] Displaying results and using the emotion engine

[1554] 8. The server sends the similar images and the suspected disease names to the terminal.

[1555] 9. The terminal displays the received similar images and suspected disease names on the user interface.

[1556] Example: Similar images and the suspected disease name "pneumonia" are displayed on the device.

[1557] 10. The user (doctor) makes a diagnosis based on the provided information. During this time, the emotion engine installed on the device analyzes the doctor's facial expressions and tone of voice to evaluate his / her trustworthiness and stress level.

[1558] Feedback and Database Updates

[1559] 11. The user (doctor) enters the final diagnosis into the terminal and simultaneously records the emotional evaluation.

[1560] Example: Enter a definitive diagnosis of "pneumonia," additional findings, and an emotional assessment of "high confidence."

[1561] 12. The terminal transmits the input feedback information to the server.

[1562] 13. The server updates the database based on the feedback information to improve the accuracy of future searches and the evaluation of the emotion engine.

[1563] Example: A new feedback record is added to the database, and retraining of the AI ​​model is automatically triggered in the backend.

[1564] This system allows doctors to easily search for similar images and identify the appropriate suspected disease when making a diagnosis. Furthermore, by utilizing an emotion engine, it is possible to assess confidence in the diagnosis and stress levels, and provide optimal information. This improves the accuracy and efficiency of diagnoses and also contributes to reducing the burden on doctors.

[1565] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1566] Step 1:

[1567] The user (doctor) uses the terminal to input the patient's medical images (e.g., CT scans, X-ray images), basic information such as age, gender, and medical history, as well as the image findings.

[1568] Input: Patient's medical images and basic information, imaging findings

[1569] Output: Data entered into the terminal

[1570] Specific operation: The doctor enters the necessary information into the input form on the terminal and uploads the image file. For example, the doctor enters the chest X-ray image of patient ID 12345, age 45, gender male, and smoking history as medical history, and uploads the image.

[1571] Step 2:

[1572] The device collects the data entered by the user and sends it to a server using a secure protocol such as HTTPS.

[1573] Input: Data entered into the terminal

[1574] Output: Data sent to the server

[1575] Specific operation: When the send button is pressed, the device encodes the data and sends it to the server via HTTPS. The data includes the patient's medical images, basic information, and imaging findings.

[1576] Step 3:

[1577] The server stores the received data in a database, along with metadata such as the date and time of storage and information about the attending physician.

[1578] Input: Data sent to the server

[1579] Output: Data stored in the database

[1580] Specific operation: The server connects to a database engine (e.g., MySQL), creates a new patient record, and saves the data, including the patient's medical images, basic information, imaging findings, save date and time, and attending physician.

[1581] Step 4:

[1582] The user (doctor) inputs medical images and basic information of a new patient, as well as imaging findings if necessary, into the terminal.

[1583] Input: Medical images and basic information of new patients, imaging findings

[1584] Output: Data entered into the terminal

[1585] Specific operation: The doctor enters the chest X-ray image of new patient ID 67890, age 65, and gender female into the input form on the terminal and uploads the image.

[1586] Step 5:

[1587] The terminal transmits the newly input data to the server again.

[1588] Input: Data entered into the terminal

[1589] Output: Data sent to the server

[1590] Specific operation: When you press the send button, the device encodes the data and sends it back to the server via HTTPS.

[1591] Step 6:

[1592] The server validates the received data, checking for missing data, then converts the image data into a standard format and prepares it for the AI ​​model.

[1593] Input: Data sent to the server

[1594] Output: Validated data and formatted image data

[1595] What happens: A validation script on the server checks the integrity of the data and converts the images to a standard format using an image processing library (e.g. OpenCV).

[1596] Step 7:

[1597] The server uses a generative artificial intelligence model and a diffusion model to search for similar images in a database and estimate the suspected disease name.

[1598] Input: Validated data and formatted image data

[1599] Output: Similar images and suspected disease names

[1600] Specific operation: A generative AI model (e.g., GAN, Diffusion Models) analyzes image data and performs similarity calculations. The associated disease name is inferred from the metadata of the most similar image. An example prompt is, "Search the historical X-ray image database for the image most similar to the chest X-ray image of patient ID 67890 and identify the suspected associated disease name."

[1601] Step 8:

[1602] The server transmits the similar image and the suspected disease name to the terminal.

[1603] Input: Similar images and suspected disease name

[1604] Output: Similar images and suspected disease names sent to the terminal

[1605] Specific operation: The server encodes the search results and the estimated disease name in JSON format and sends them to the terminal.

[1606] Step 9:

[1607] The terminal displays the received similar images and suspected disease names on the user interface.

[1608] Input: Similar images and suspected disease names sent to the terminal

[1609] Output: Displayed similar images and suspected disease names

[1610] Specific operation: Highly relevant similar images and the estimated disease name "pneumonia" are displayed on the device interface.

[1611] Step 10:

[1612] The user (doctor) makes a diagnosis based on the provided information, while the device's built-in emotion engine analyzes facial expressions and tone of voice to assess trustworthiness and stress level.

[1613] Input: Displayed similar images, suspected disease names, facial expressions and tone of voice during the user's diagnostic activity

[1614] Output: Emotion engine estimates confidence and stress level

[1615] How it works: During the diagnosis, the emotion engine collects and analyzes data from the webcam and microphone in real time, assessing the confidence and stress level of the user as they enter their diagnosis information.

[1616] Step 11:

[1617] The user (doctor) inputs the final diagnosis and emotional evaluation into the terminal.

[1618] Input: Final diagnosis and emotional evaluation

[1619] Output: Feedback and emotion rating data entered into the device

[1620] Specific operation: Enter the definitive diagnosis, additional findings, and emotional assessment into the input form on the device. Example: In addition to the definitive diagnosis of "pneumonia," enter "high confidence."

[1621] Step 12:

[1622] The terminal transmits the input feedback information to the server.

[1623] Input: Feedback and emotional rating data entered into the device

[1624] Output: Feedback and sentiment rating data sent to the server

[1625] Specific operation: When you press the send button, the device encodes the data and sends it to the server.

[1626] Step 13:

[1627] The server stores the received feedback information in a database to improve the accuracy of future searches and emotion evaluations.

[1628] Input: Feedback and sentiment rating data sent to the server

[1629] Output: An updated database and a trained AI model

[1630] Specific operation: The server adds new feedback records to the database and automatically initiates retraining of the AI ​​model to improve search accuracy and sentiment assessment accuracy.

[1631] (Application example 2)

[1632] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1633] Current machine maintenance and defective product inspection in factories places a heavy burden on operators, and anomaly detection and diagnosis require time and effort. Furthermore, there is a lack of appropriate information provision and diagnostic feedback based on the operator's emotions and stress level, which reduces work efficiency and diagnostic reliability. Therefore, there is a need for a system that reduces the burden on operators and improves diagnostic accuracy and work efficiency.

[1634] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1635] In this invention, the server includes means for inputting video data and various sensor data of machines and products, database means for storing the video data and various sensor data, and means for generating suspected fault locations and abnormalities of machines and products using the video data and various sensor data. This reduces the burden on operators and improves diagnostic accuracy and work efficiency during machine maintenance and defective product inspection in factories.

[1636] "Machinery and products" refers to the equipment used in the factory and the goods manufactured.

[1637] "Video data" refers to image information captured by a photographing device such as a camera.

[1638] "Sensor data" refers to information obtained from various sensors that detect vibration, temperature, sound, etc.

[1639] A "database" is a system that stores and manages input video data, sensor data, and associated metadata.

[1640] A "generative artificial intelligence model" is an artificial intelligence that uses machine learning technology to generate new information from data.

[1641] A "diffusion model" is an algorithm that discovers patterns and trends from large amounts of data and analyzes relationships in a diffuse manner.

[1642] A "similar image" is an image among the video data stored in the database that has characteristics similar to the newly input video data.

[1643] A "suspected disease name" is a disease name that may be inferred based on medical images and basic information.

[1644] A "suspected failure point" is a part or component of a machine or product where an abnormality is suspected.

[1645] An "emotion engine" is a system that analyzes a user's emotional responses, such as facial expressions and tone of voice, to assess their trustworthiness and stress level.

[1646] A "user" is someone who operates the system and checks the results.

[1647] This invention relates to a system that supports machine maintenance and defective product inspection in factories. This system inputs video data and various sensor data to detect failures and abnormalities early, improving diagnostic accuracy and work efficiency. It also includes an emotion engine that recognizes the emotions of operators and evaluates the reliability of diagnoses and stress levels.

[1648] System Configuration and Operation

[1649] Data entry and saving

[1650] The user (operator) inputs video data of machines and products in the factory and multiple sensor data (e.g., vibration, temperature, sound) into a dedicated terminal. The terminal sends this input information to a server, which then stores the received data in a database. This data also includes metadata at the time of input (e.g., the date and time of saving and information about the operator in charge).

[1651] Search for similar data and suspected fault locations

[1652] The user inputs newly acquired video data and sensor data into the device. This data is then sent back to the server, where it is first converted into a standard format and formatted so that the AI ​​model can easily process it. The server then uses a generative AI model and a diffusion model to search the database for similar past data. As a result of the search, data with high similarity is extracted, and related suspected fault locations and anomalies are inferred.

[1653] Displaying results and using the emotion engine

[1654] The server sends the extracted similar data and suspected fault locations to the device, which then displays them on the user interface. When the user performs a diagnosis based on this, the device's built-in emotion engine analyzes the user's facial expressions and tone of voice to evaluate the reliability and stress level.

[1655] Feedback and Database Updates

[1656] The user inputs the diagnosis results (complete failure, partial failure, normal, etc.) into the device and sends them to the server. The emotion evaluation by the emotion engine is also recorded at the same time. The server stores the received feedback information in a database and uses it as training data to improve search accuracy in future searches.

[1657] The specific hardware and software used

[1658] Hardware

[1659] Camera for capturing images

[1660] Various sensors (vibration sensors, temperature sensors, sound sensors, etc.)

[1661] GPU (e.g., NVIDIA CUDA-enabled GPU) for accelerating image processing and AI model calculations

[1662] software

[1663] Scripting languages ​​for image processing and data analysis (e.g., Python)

[1664] Machine learning models (e.g., TensorFlow, scikit-learn)

[1665] Libraries for emotion recognition (e.g., Some Emotion Recognition Library)

[1666] Specific examples

[1667] Examples of data entry and saving

[1668] 1. The user (operator) inputs the image data and vibration sensor data of machine A in the factory into the terminal.

[1669] 2. The device sends this data to the server, which stores it in a database.

[1670] Example of searching for similar data and suspected fault locations

[1671] 3. Enter the newly acquired video data and temperature sensor data from Machine B into the terminal.

[1672] 4. The server receives the data sent from the device and performs a similarity search.

[1673] 5. Similar data is extracted and suspected fault locations are estimated.

[1674] Displaying the results and using the emotion engine

[1675] 6. The suspected faulty part and past similar cases will be displayed on the device screen.

[1676] 7. The emotion engine analyzes the user's facial expressions and tone of voice to assess their confidence and stress level.

[1677] Examples of feedback and database updates

[1678] 8. The user enters the definitive diagnosis and comments into the terminal and sends them to the server.

[1679] 9. Feedback information is stored in the database to improve search accuracy in future searches.

[1680] Example of input prompt for generative AI model

[1681] "We have taken maintenance footage of machine A in the factory. Abnormal vibrations have been detected by the vibration sensor. Please search for similar past abnormal data and identify the suspected fault location."

[1682] This is expected to not only improve the efficiency of machine maintenance and defective product inspection within factories and reduce the burden on operators, but also improve the reliability of diagnoses.

[1683] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1684] Step 1:

[1685] Users input video data of machines and products in the factory and various sensor data into the terminal. The terminal acquires this data and sends it to the server. The input data includes video of the machines taken by a camera and numerical data from vibration and temperature sensors.

[1686] Step 2:

[1687] The server stores the received video and sensor data in a database. When the data is stored, metadata (such as the date and time of storage and information about the operator in charge) is also recorded. The server then converts this data into a standard format so that it can be searched later.

[1688] Step 3:

[1689] The user inputs new machine video and sensor data into the device. The device then sends this data to the server. The new data includes video of the suspected malfunctioning machine and abnormal vibration and temperature values.

[1690] Step 4:

[1691] The server converts the new data it receives into a standard format and formats it so that it can be easily processed by the AI ​​model. Specifically, image data is resized and noise is removed, and sensor data is normalized and smoothed.

[1692] Step 5:

[1693] The server uses a generative AI model and a diffusion model to search the database for similar past video data and sensor data. The AI ​​model compares the features of the input data with those in the database and extracts data with high similarity. This extracted data includes data on past instances of machines exhibiting similar vibration or temperature abnormalities.

[1694] Step 6:

[1695] The server sends the extracted similar data and a list of related suspected fault locations to the terminal. The data on similar cases and fault locations is displayed on the terminal. The user performs a diagnosis based on this information.

[1696] Step 7:

[1697] The device's built-in emotion engine analyzes the user's facial expressions and tone of voice to assess the reliability of the diagnosis and stress level. The emotion engine uses the camera and microphone to analyze the user's emotional state in real time.

[1698] Step 8:

[1699] The user inputs the final diagnosis result into the terminal, which includes information on whether the machine is normal, partially broken, or completely broken. The terminal then sends this information to the server.

[1700] Step 9:

[1701] The server records the received feedback information in a database. The recorded feedback information is used as training data to improve search accuracy in future searches. Specifically, it is used to retrain the AI ​​model, enabling more accurate failure prediction.

[1702] The specific operations at each step ensure efficient operation of the entire system, improving the efficiency of machine maintenance and defective product inspection within the factory. Through this series of processes, the burden on operators is reduced and the reliability and accuracy of fault diagnosis is improved.

[1703] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1704] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1705] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1706] [Fourth embodiment]

[1707] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1708] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1709] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1710] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1711] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1712] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1713] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1714] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1715] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1716] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1717] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1718] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1719] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1720] The present invention relates to a diagnostic support system that uses medical images, basic information, and imaging findings. This system generates similar images and suspected disease names based on input of patient medical images and information by a doctor, and supports diagnosis. Specific embodiments of each element of this system are described below.

[1721] Data entry and saving

[1722] User (doctor): Enters the patient's medical images (e.g., CT scan), basic information such as age, gender, and medical history, as well as imaging findings (such as the location and shape of the lesion) into the terminal.

[1723] Terminal: Sends the entered information to the server.

[1724] Server: The received data is stored in a database. The stored data also includes metadata (date and time, information about the person who entered the data, etc.) at the time of input.

[1725] Search for similar images and suspected diseases

[1726] User (doctor): Enters medical images and basic information about a new patient who wishes to be diagnosed into the terminal, along with any imaging findings.

[1727] Terminal: Sends the entered data to the server.

[1728] Server: Receives the data and performs formatting processing, checking for missing data and converting it to a standard format.

[1729] Server: Using a generative AI model and a diffusion model, the server searches for similar images in the database. From the search results, it extracts images with high similarity and predicts the suspected disease names associated with them.

[1730] Results display and feedback

[1731] Server: Sends a list of similar images and suspected disease names to the terminal.

[1732] Terminal: Similar images and suspected disease names are displayed on the user interface.

[1733] User (doctor): Makes a diagnosis based on the information provided and enters the final diagnosis and its feedback into the terminal.

[1734] Terminal: Sends the input feedback information to the server.

[1735] Server: Updates the database based on the feedback information to improve search accuracy from the next time onwards.

[1736] Specific examples

[1737] 1. Examples of data entry and saving

[1738] User (doctor): Enters the chest X-ray image, age (45 years old), gender (male), and medical history (smoking history) of patient ID 12345 into the terminal form.

[1739] Terminal: Sends data to the server.

[1740] Server: Save the information of patient ID 12345 in the database.

[1741] 2. Example of searching for similar images and suspected diseases

[1742] User (doctor): Enters the chest X-ray image, age (65 years old), and gender (female) of new patient ID 67890 into the terminal.

[1743] Terminal: Sends data to the server.

[1744] Server: Uses a generative artificial intelligence model to search the database for similar past X-ray images. Extracts the most similar set of images from the previously stored images.

[1745] Server: Identifies the suspected disease name (e.g., pneumonia) related to the extracted image and sends it to the terminal.

[1746] 3. Examples of results and feedback

[1747] Terminal: The doctor's terminal displays a similar image of a chest X-ray and the suspected disease name "pneumonia."

[1748] User (doctor): Checks similar images and suspected disease names and confirms "pneumonia" as the definitive diagnosis for patient ID 67890.

[1749] User (doctor): Enters the definitive diagnosis and additional findings into the terminal and sends them to the server.

[1750] Server: Updates the database and accumulates feedback data to improve the accuracy of the next search.

[1751] In this way, the present invention allows doctors to easily search for similar images and obtain the name of a suspected disease when making a diagnosis, thereby improving the accuracy and efficiency of diagnosis.

[1752] The processing flow will be explained below.

[1753] Program processing flow

[1754] Data entry and saving

[1755] Step 1:

[1756] User (doctor): Enters basic information such as the patient's medical images (e.g., CT scan), age, gender, medical history, and imaging findings (such as the location and shape of the lesion) into a dedicated form on the terminal.

[1757] Step 2:

[1758] Terminal: Validate that the data entered is accurate and that all required fields are completed.

[1759] Step 3:

[1760] Terminal: Sends the verified data to the server.

[1761] Step 4:

[1762] Server: Double-checks the integrity of the data received and converts it into the appropriate format.

[1763] Step 5:

[1764] Server: Stores patient information, image data, and findings data in a database with unique identifiers.

[1765] Step 6:

[1766] Server: Adds metadata (such as the date and time of storage and information about the attending physician) to the stored data to improve future search performance.

[1767] Search for similar images and suspected diseases

[1768] Step 7:

[1769] User (doctor): Enters image data and basic information of a new patient for whom a diagnosis is desired into the terminal. If necessary, additional findings are also entered.

[1770] Step 8:

[1771] Terminal: Sends the entered data to the server.

[1772] Step 9:

[1773] Server: Validates the received data and checks for missing data.

[1774] Step 10:

[1775] Server: Converts image data into a standard format and formats it so that it can be easily processed by the AI ​​model.

[1776] Step 11:

[1777] Server: Using generative AI and diffusion models, the server searches for similar images in the database. The algorithm also takes into account the patient's basic information and findings.

[1778] Step 12:

[1779] Server: Extracts the most similar images from the search results and identifies the suspected disease associated with them.

[1780] Step 13:

[1781] Server: Sends similar images and the names of suspected diseases associated with them to the terminal.

[1782] Results display and feedback

[1783] Step 14:

[1784] Terminal: Displays the similar images and suspected disease names sent from the server on the user interface.

[1785] Step 15:

[1786] User (doctor): Check the displayed information and use it as a reference for diagnosis.

[1787] Step 16:

[1788] User (doctor): Enters the final diagnosis (e.g., confirmed disease name and additional findings) into the terminal and sends it to the server.

[1789] Step 17:

[1790] Server: Updates the database based on the feedback received, specifically by linking definitive diagnoses and additional findings to existing data to help improve the accuracy of future search results.

[1791] Step 18:

[1792] Server: The feedback information is also used as training data for the AI ​​model to improve its accuracy.

[1793] This detailed process flow will enable doctors to be more confident in their diagnoses, leading to more accurate and faster diagnoses. The system aims to improve the quality of healthcare by leveraging the power of automation and AI.

[1794] Example 1

[1795] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1796] In current medical settings, doctors often lack sufficient support to make accurate and prompt diagnoses. In particular, similar image searches and suspected disease diagnoses from medical images are often performed manually, resulting in time-consuming diagnosis and inconsistent diagnostic accuracy. A system that can solve these problems and support doctors' diagnoses is needed.

[1797] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1798] In this invention, the server includes: a means for inputting patient medical images; a means for inputting basic patient information and imaging findings; an information storage means for storing the medical images, basic information, and imaging findings; a transmission means for transmitting the input information to the server; a data formatting means for formatting the received data, checking for missing data, and converting it to a standard format; a means for searching for similar images from the information storage means using a generative artificial intelligence model and a diffusion model; a means for generating a suspected disease name based on the similar images, the basic information, and imaging findings; a means for displaying the similar images and the suspected disease name; a means for inputting a diagnosis result and feedback; a transmission means for transmitting the diagnosis result and feedback information to the server; and a means for updating the information storage means based on the feedback information to improve search accuracy from the next time onward. This allows doctors to easily search for similar images and obtain a suspected disease name when making a diagnosis.

[1799] "Patient medical images" refers to image data taken of a patient using medical equipment, including, specifically, CT scans, X-ray images, and MRI images.

[1800] "Basic information" refers to basic information about the patient, such as the patient's age, gender, and medical history.

[1801] "Imaging findings" refers to findings observed by a physician, such as the location or shape of a specific lesion or abnormality seen in a medical image.

[1802] "Information storage means" refers to a database or storage system for appropriately storing the aforementioned medical images, basic information, and image findings.

[1803] "Transmission means" refers to a communication protocol or interface for transmitting input information to a server.

[1804] "Data formatting means" refers to a function for formatting received data, checking for missing data, and converting it into a standard format.

[1805] A "generative artificial intelligence model" refers to a machine learning model that learns patterns and correlations from data and makes inferences and predictions based on the results.

[1806] A "diffusion model" is a type of generative artificial intelligence model that performs probabilistic sampling within a multidimensional data space to generate new data.

[1807] "Search means" refers to the function of searching for necessary information from information storage means, specifically an interface for performing database searches and searches for similar images.

[1808] "Means for generating" refers to the process or algorithm for calculating and generating a suspected disease name based on basic information and imaging findings.

[1809] "Displaying means" refers to a user interface for visually presenting similar images and suspected disease names to the user.

[1810] "Feedback information" refers to information including a doctor's diagnosis and additional findings, and includes data used to improve the system's search accuracy.

[1811] The present invention relates to a diagnostic support system that uses medical images, basic information, and image findings. This system generates similar images and suspected disease names and supports diagnosis by having a doctor input a patient's medical images and basic information. Specific embodiments of the present invention are described below.

[1812] System configuration

[1813] This system includes a terminal operated by a user (doctor) and a server connected to it. The terminal includes an input interface and a display interface, and the server includes a data storage means, a data formatting means, and a search means using a generative artificial intelligence model and a diffusion model.

[1814] Data entry and saving

[1815] The user (doctor) inputs the patient's medical images (e.g., CT scans), basic information such as age, gender, and medical history, as well as the imaging findings, into a terminal. The terminal then sends the input information to a server. The server then stores the received data in a database. At this time, metadata (date and time, information about the person who inputted, etc.) at the time of input is also stored.

[1816] Specific examples

[1817] For example, if a user inputs "Patient ID 12345, age 45, male, smoking history, chest X-ray image," the terminal sends this data to the server, which then stores this information in a database.

[1818] Search for similar images and suspected diseases

[1819] When a user (doctor) wishes to diagnose a new patient, they again input the medical images, basic information, and imaging findings into the terminal. The terminal then sends the input data to the server. The server then performs formatting on the received data, checks for missing data, and converts it into a standard format. A generative artificial intelligence model (using, for example, TensorFlow or PyTorch) and a diffusion model are used to search for similar images in the database. Images with high similarity are extracted from the search results, and suspected diseases related to them are estimated.

[1820] Specific examples

[1821] When a user inputs "Patient ID 67890, age 65, female, chest X-ray image," the device sends this data to the server. The server uses a generative artificial intelligence model to search a database for similar past X-ray images, extracts the most similar image set, and predicts the suspected disease name (e.g., "pneumonia").

[1822] Results display and feedback

[1823] The server sends a list of similar images and suspected disease names to the terminal. The terminal displays the similar images and suspected disease names on the user interface. The user (doctor) makes a diagnosis based on the provided information and enters a definitive diagnosis and its feedback into the terminal. The terminal sends the entered feedback information to the server. The server updates the database based on the feedback information, improving the accuracy of the next search.

[1824] Specific examples

[1825] For example, the terminal display shows "Similar Image 1," "Similar Image 2," and "Suspected Disease: Pneumonia." The user confirms this and enters "Patient ID 67890, Definitive Diagnosis: Pneumonia, Additional Findings: Shadow Spread." The terminal sends this feedback information to the server. The server updates the database and accumulates diagnosis history, improving search accuracy.

[1826] This system allows doctors to easily search for similar images and obtain the name of a suspected disease when making a diagnosis, improving the accuracy and efficiency of diagnosis.

[1827] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1828] Step 1:

[1829] The user (doctor) inputs the patient's medical images (e.g., CT scan), basic information such as age, gender, medical history, and imaging findings into the terminal. The input information is used as initial data for the medical diagnosis system. Specifically, the doctor inputs "Patient ID 12345, age 45, male, smoking history, chest X-ray image" into the input form on the terminal. Input is mainly done in the form of text fields and file uploads.

[1830] Step 2:

[1831] The terminal sends the input information to the server. A communication protocol such as an HTTP POST request is used for transmission. Specifically, when the send button is pressed, the terminal puts the input data into the body of an HTTP request and sends it to the specified endpoint on the server. This request includes the input information (patient ID, age, gender, medical history, and image data).

[1832] Step 3:

[1833] The server performs a formatting process on the received data. Data formatting includes checking for missing data and converting it to a standard format. For example, it checks whether required fields such as age and gender are missing, and converts it to JSON format. Specifically, the server analyzes the data and returns an error message if any information is missing. Once the formatting process is complete, the data is unified into a standard format. The input is the initial raw data, and the output is the formatted data.

[1834] Step 4:

[1835] The server stores the received and formatted data in a database. When storing the data, the metadata at the time of input (date and time, information about the person who entered, etc.) is also stored. Specifically, the server creates a database entry and stores the patient information appropriately. This process is performed using a specific database management system (DBMS). The input is formatted data, and the output is a record stored in the database.

[1836] Step 5:

[1837] The user (doctor) inputs medical images, basic information, and imaging findings of a new patient for whom they wish to diagnose into the terminal. Specifically, the doctor inputs, for example, "Patient ID 67890, age 65, female, chest X-ray image" into the terminal. The data input format is the same as a text field or file upload.

[1838] Step 6:

[1839] The terminal again sends the entered data to the server. The sending process is the same as in step 2, using an HTTP POST request. The input is the newly entered information about the patient seeking diagnosis, and the output is the request sent to the server.

[1840] Step 7:

[1841] The server uses a generative artificial intelligence model (using, for example, TensorFlow or PyTorch) and a diffusion model to search for similar images in the database. After data shaping, the input data is provided to the generative AI model, which calculates a similarity score. Based on this score, the most similar image set is extracted from the database. Specifically, the server runs the AI ​​model and compares previously saved images with newly input images. The input is the shaped new data, and the output is a set of highly similar images and a suspected disease name.

[1842] Step 8:

[1843] The server sends a list of estimated similar images and suspected disease names to the terminal. This is done again using an HTTP response. Specifically, the server returns the generated search results to the terminal in JSON format. The input is the output result of the AI ​​model, and the output is the data sent to the terminal.

[1844] Step 9:

[1845] The terminal displays similar images and the suspected disease name on the user interface. Specifically, the terminal analyzes the received data and displays information such as "Similar Image 1," "Similar Image 2," and "Suspected Disease Name: Pneumonia" on the screen. The input is the data sent from the server, and the output is the displayed user interface.

[1846] Step 10:

[1847] The user (doctor) makes a diagnosis based on the provided information and enters the definitive diagnosis and its feedback into the terminal. Specifically, the doctor enters "Patient ID 67890, definitive diagnosis: pneumonia, additional findings: shadow spreading" into the diagnosis result input form. The input is the diagnosis result and additional findings.

[1848] Step 11:

[1849] The terminal sends the entered feedback information to the server. The sending process again uses an HTTP POST request. The input is the doctor's final diagnosis and feedback information, and the output is the request sent to the server.

[1850] Step 12:

[1851] The server updates the database based on the feedback information and takes measures to improve search accuracy from the next time onwards. Specifically, the server analyzes the feedback information and updates the corresponding entries in the database. This enables diagnostic support to be performed with higher accuracy the next time the search is performed. The input is the feedback information and the output is the updated database.

[1852] (Application example 1)

[1853] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1854] Conventional diagnostic support systems have been used primarily in the medical field, making it difficult to apply them to other fields. Furthermore, abnormal behavior detection systems that use surveillance camera footage and sensor data lack effective means for detecting abnormal behavior in real time and responding quickly. Therefore, there is a need to apply the technology of diagnostic support systems used in the medical field to abnormal behavior detection systems that use surveillance camera footage and sensor data, thereby improving the accuracy of abnormal behavior detection.

[1855] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1856] In this invention, the server includes means for inputting patient medical images, means for inputting basic information and imaging findings of the patient, database means for storing the medical images, basic information, and imaging findings, means for searching for similar images from the database using a generative artificial intelligence model and a diffusion model, means for generating a suspected disease name based on the similar images and the basic information and imaging findings, means for presenting the similar images and suspected disease name to a user, means for inputting surveillance camera footage and sensor data, database means for storing the surveillance camera footage and sensor data, means for searching for similar behavior from the database using a generative artificial intelligence model, means for detecting abnormal behavior based on the similar behavior, the surveillance camera footage, and the sensor data, and means for presenting the abnormal behavior and the reason for the abnormal behavior to a user. This enables advanced abnormal behavior detection and diagnostic support in a variety of fields.

[1857] "Patient" refers to a person receiving medical services.

[1858] "Medical imaging" refers to image data, such as CT scans, X-rays, and MRI images, acquired to visualize the internal structures of a patient's body.

[1859] "Basic information" refers to basic information necessary for diagnosis and treatment, such as the patient's age, gender, and medical history.

[1860] "Imaging findings" refers to information and findings necessary for diagnosis that a doctor intuitively obtains based on medical images.

[1861] "Database means" refers to means for systematically storing, managing, and searching input data.

[1862] A "generative artificial intelligence model" refers to an artificial intelligence model that is trained to analyze data and generate new information.

[1863] A "diffusion model" refers to a model that spreads data to obtain accurate analysis results over a wide range.

[1864] "Similar images" refer to past images that have similar characteristics and features to the input medical image.

[1865] "Suspected disease name" refers to the name of a disease that is inferred based on similar images, basic information, and imaging findings.

[1866] "Surveillance camera footage" refers to video data captured by a surveillance camera.

[1867] "Sensor data" refers to data obtained from various sensors installed in a monitoring system.

[1868] "Similar behavior" refers to behavior that has similar characteristics to past behavioral data based on surveillance camera footage and sensor data.

[1869] "Abnormal behavior" refers to behavior that deviates from normal patterns of behavior and may pose a security problem.

[1870] "User" refers to the personnel and doctors who operate the system and use the information.

[1871] The present invention provides a system for assisting diagnosis using medical images and basic information of patients, and a system for detecting abnormal behavior using surveillance camera footage and sensor data. Specific embodiments of these systems are described below.

[1872] Diagnostic support system using medical images

[1873] First, a diagnosis support system using medical images and basic information will be described.

[1874] Data entry and saving

[1875] The user (doctor) inputs the patient's medical images (e.g., CT scans or X-ray images), basic information such as age, gender, and medical history, as well as imaging findings (such as the location and shape of the lesion) into a terminal. The terminal then sends this data to a server, which then stores the received data in a database. The stored data also includes metadata from the time of input (such as date and time and information about the person who entered the data).

[1876] Search for similar images and suspected diseases

[1877] The user (doctor) inputs medical images and basic information about a new patient they wish to diagnose into the terminal. The terminal then sends the input data to the server, which receives it and performs formatting processing. The data is checked for any missing parts and converted into a standard format. Next, a generative artificial intelligence model and a diffusion model are used to search for similar images in the database. Images with high similarity are extracted from the search results, and the suspected disease names associated with them are estimated.

[1878] Results display and feedback

[1879] The server sends a list of similar images and suspected diseases to the terminal, which displays them on the user interface. The user (doctor) makes a diagnosis based on the provided information and enters a definitive diagnosis and feedback into the terminal. The terminal then sends the entered feedback information to the server, which updates the database based on it.

[1880] As a specific example, a user (doctor) enters the chest X-ray image, age (45 years old), gender (male), and medical history (smoking history) of patient ID 12345 into a form on the terminal, and the terminal sends the data to the server and stores it.The user (doctor) then enters the chest X-ray image, age (65 years old), and gender (female) of a new patient ID 67890, and the server uses a generative artificial intelligence model to search for similar images from past data, identifies the suspected disease name, and sends it to the terminal.

[1881] Abnormal behavior detection system using surveillance camera footage and sensor data

[1882] Data entry and saving

[1883] Users (security personnel) input data into the system from surveillance camera footage, motion sensors, sound sensors, etc. The terminals send this data to the server, which then stores the received data in a database.

[1884] Abnormal behavior detection

[1885] The server uses a generative AI model to analyze past abnormal behavior data stored in a database and real-time surveillance camera footage and sensor data to detect abnormal behavior. The generative AI model uses YOLO (You Only Look Once) to analyze the video data and sensor data.

[1886] Results display and alerts

[1887] The server extracts video of suspected abnormal behavior and the reason for it, and issues an alert to the terminal. The video of suspected abnormal behavior and the reason for it are displayed on a user interface (for example, a web application built with React). The user (security officer) responds based on the information provided, and inputs feedback information (confirmation of abnormal behavior and how to deal with it) into the terminal and sends it to the server.

[1888] For example, if a surveillance camera detects an intrusion or abnormal activity, it uses a generative AI model to analyze the behavior and, if it determines that the behavior is abnormal, issues an alarm. Security personnel receive the alarm and can respond promptly.

[1889] Example prompt for a generative AI model:

[1890] "Detect abnormal behavior in surveillance camera footage in real time, compare it with past abnormal behavior data, and issue an alert. Abnormal behavior includes intrusions into specific areas or unusual movements."

[1891] This system will enable advanced abnormal behavior detection and diagnostic support not only in the medical field but also in the security field.

[1892] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1893] Diagnostic support system using medical images

[1894] Step 1:

[1895] Description of the process: The user (doctor) inputs the patient's medical images, basic information, and imaging findings into the terminal.

[1896] Input: Medical images (e.g., CT scan), patient age, gender, medical history, imaging findings

[1897] Data processing: Enter data into the form on the device

[1898] Output: Formatted data

[1899] Step 2:

[1900] Processing description: The terminal sends the input medical images, basic information, and image findings to the server.

[1901] Input: Formatted data (medical images, age, gender, medical history, imaging findings)

[1902] Data calculation: Packetize the data and send it to the server via the network

[1903] Output: Data received by the server

[1904] Step 3:

[1905] Process description: The server stores the received data in a database.

[1906] Input: The data received by the server

[1907] Data processing: Save the data to the corresponding record in the database

[1908] Output: Data stored in the database

[1909] Step 4:

[1910] Process description: The user (doctor) inputs medical images and basic information of a new patient into the terminal.

[1911] Input: New medical image, age, gender, medical history

[1912] Data processing: Enter data into the form on the device

[1913] Output: The new formatted data

[1914] Step 5:

[1915] Process description: The terminal sends the new data entered to the server.

[1916] Input: Formatted new data (medical images, age, gender, basic information, imaging findings)

[1917] Data calculation: Packetize the data and send it to the server via the network

[1918] Output: New data received by the server

[1919] Step 6:

[1920] Process Description: The server analyzes the new data received and searches for similar images in its database.

[1921] Input: New medical images and basic information

[1922] Data Computing: Searching for Similar Images Using Generative Artificial Intelligence Models and Diffusion Models

[1923] Output: A list of similar images

[1924] Step 7:

[1925] Process description: The server sends similar images and the suspected disease name to the terminal.

[1926] Input: A list of similar images

[1927] Data calculation: Estimation of suspected disease and data packetization

[1928] Output: List of similar images received by the device and suspected disease name

[1929] Step 8:

[1930] Process description: The terminal displays similar images and suspected disease names on the user interface.

[1931] Input: List of similar images and suspected disease name

[1932] Data processing: Display on the user interface

[1933] Output: Information that the user (doctor) can check

[1934] Step 9:

[1935] Process description: The user (doctor) inputs the definitive diagnosis and feedback information into the terminal.

[1936] Input: definitive diagnosis, additional findings

[1937] Data processing: Enter data into the form on the device

[1938] Output: Formatted feedback data

[1939] Step 10:

[1940] Process description: The terminal sends feedback data to the server.

[1941] Input: Formatted feedback data

[1942] Data calculation: Feedback data is packetized and sent to the server via the network

[1943] Output: Feedback data received by the server

[1944] Step 11:

[1945] Process description: The server stores the feedback data in a database to improve search accuracy for future searches.

[1946] Input: Received feedback data

[1947] Data processing: Update the database

[1948] Output: Updated database

[1949] Abnormal behavior detection system using surveillance camera footage and sensor data

[1950] Step 1:

[1951] Process description: The user (security officer) inputs surveillance camera footage and sensor data into the system.

[1952] Input: Surveillance camera footage, motion sensor data, sound sensor data

[1953] Data processing: Input into the system within the terminal

[1954] Output: Formatted monitoring data

[1955] Step 2:

[1956] Process description: The terminal sends the input monitoring data to the server.

[1957] Input: Formatted monitoring data

[1958] Data calculation: Packetize the data and send it to the server via the network

[1959] Output: Monitoring data received by the server

[1960] Step 3:

[1961] Process description: The server stores the received data in a database.

[1962] Input: Received monitoring data

[1963] Data processing: Save the data to the corresponding record in the database

[1964] Output: Monitoring data stored in a database

[1965] Step 4:

[1966] Process description: The server analyzes historical abnormal behavior data and real-time monitoring data in the database using a generative artificial intelligence model (e.g., YOLO).

[1967] Input: Real-time monitoring data, historical abnormal behavior data

[1968] Data Computation: A Comparative Analysis Using Generative Artificial Intelligence Models

[1969] Output: Abnormal behavior detection results

[1970] Step 5:

[1971] Processing description: When the server detects abnormal behavior, it sends alarm data including the reason to the terminal.

[1972] Input: Abnormal behavior detection result, reason

[1973] Data Computation: Alarm data generation and packetization

[1974] Output: Alarm data received by the terminal

[1975] Step 6:

[1976] Process Description: The terminal displays the alarm data on the user interface.

[1977] Input: Alarm data

[1978] Data processing: Display on the user interface

[1979] Output: Alert information that can be checked by the user (security officer)

[1980] Step 7:

[1981] Process description: The user (security officer) inputs the confirmed abnormal behavior and the response into the terminal.

[1982] Input: Confirmed abnormal behavior, response details

[1983] Data processing: Enter data into the form on the device

[1984] Output: Formatted feedback data

[1985] Step 8:

[1986] Process description: The terminal sends feedback data to the server.

[1987] Input: Formatted feedback data

[1988] Data calculation: Feedback data is packetized and sent to the server via the network

[1989] Output: Feedback data received by the server

[1990] Step 9:

[1991] Processing description: The server stores the received feedback data in a database to improve detection accuracy in the future.

[1992] Input: Received feedback data

[1993] Data processing: Update the database

[1994] Output: Updated database

[1995] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1996] The present invention relates to a diagnostic support system that uses medical images, basic information, and imaging findings. This system generates similar images and suspected disease names when a doctor inputs a patient's medical images and information, and also recognizes the user's emotions to support diagnosis. Specific embodiments of each element of this system are described below.

[1997] Data entry and saving

[1998] User (doctor): Enters the patient's medical images (e.g., CT scan), basic information such as age, gender, and medical history, as well as imaging findings (such as the location and shape of the lesion) into the terminal.

[1999] Terminal: Sends the entered information to the server.

[2000] Server: The received data is stored in a database, and the stored data also includes metadata at the time of input (such as the date and time of storage and information about the attending physician).

[2001] Search for similar images and suspected diseases

[2002] User (doctor): Enters medical images and basic information about a new patient who wishes to be diagnosed into the terminal, along with any imaging findings.

[2003] Terminal: Sends the entered data to the server.

[2004] Server: Validates the received data, checks for missing data, and converts the image data into a standard format that is easy for the AI ​​model to process.

[2005] Server: Using a generative AI model and a diffusion model, the server searches for similar images in the database. From the search results, it extracts images with high similarity and predicts the suspected disease names associated with them.

[2006] Displaying results and using the emotion engine

[2007] Server: Sends a list of similar images and suspected disease names to the terminal.

[2008] Terminal: Similar images and suspected disease names are displayed on the user interface.

[2009] User (doctor): A diagnosis is made based on the information provided. The device's built-in emotion engine recognizes the user's emotions from facial expressions and tone of voice, and evaluates their reliability and stress level.

[2010] Feedback and Database Updates

[2011] User (doctor): Enters the final diagnosis (e.g., confirmed disease name and additional findings) into the terminal. The emotion evaluation by the emotion engine is also recorded at the same time.

[2012] Terminal: Sends the input feedback information to the server.

[2013] Server: Updates the database based on feedback information to improve search accuracy from the next time onwards. Also, adjusts the display method and presentation order based on the evaluation results of the emotion engine, allowing users to use information more efficiently.

[2014] Specific examples

[2015] 1. Examples of data entry and saving

[2016] User (doctor): Enters the chest X-ray image, age (45 years old), gender (male), and medical history (smoking history) of patient ID 12345 into the terminal.

[2017] Terminal: Sends data to the server.

[2018] Server: Save the information of patient ID 12345 in the database.

[2019] 2. Example of searching for similar images and suspected diseases

[2020] User (doctor): Enters the chest X-ray image, age (65 years old), and gender (female) of new patient ID 67890 into the terminal.

[2021] Terminal: Sends data to the server.

[2022] Server: Uses a generative artificial intelligence model to search the database for similar past X-ray images. Extracts the most similar set of images from the previously stored images.

[2023] Server: Identifies the suspected disease name (e.g., pneumonia) related to the extracted image and sends it to the terminal.

[2024] 3. Displaying the results and using the emotion engine

[2025] Terminal: The doctor's terminal displays a similar image of a chest X-ray and the suspected disease name "pneumonia."

[2026] Emotion engine: Analyzes the user's facial expressions and tone of voice to assess their confidence in the diagnosis and stress level.

[2027] User (doctor): Checks similar images and suspected disease names, and in a stable emotional state, confirms that "pneumonia" is the definitive diagnosis for patient ID 67890.

[2028] User (doctor): Enters the definitive diagnosis, additional findings, and emotional assessment during the diagnosis into the terminal and sends them to the server.

[2029] 4. Examples of feedback and database updates

[2030] Server: The received feedback information is stored in a database to improve search and emotion evaluation accuracy in the future.

[2031] Server: Based on the feedback information, the AI ​​model is trained and diagnostic support functions are enhanced.

[2032] In this way, the present invention allows doctors to easily search for similar images and identify suspected diseases when making diagnoses. Furthermore, by using an emotion engine, it becomes possible to evaluate the confidence in a diagnosis and the stress level, enabling more appropriate information to be provided. This not only improves the accuracy and efficiency of diagnoses, but also contributes to reducing the burden on doctors.

[2033] The processing flow will be explained below.

[2034] Program processing flow: Data entry and saving

[2035] Step 1:

[2036] User (doctor): Enters basic information such as the patient's medical images (e.g., CT scan), age, gender, medical history, and imaging findings (such as the location and shape of the lesion) into a dedicated form on the terminal.

[2037] Step 2:

[2038] Terminal: Validate that the data entered is accurate and that all required fields are completed.

[2039] Step 3:

[2040] Terminal: Sends the verified data to the server.

[2041] Step 4:

[2042] Server: Double-checks the integrity of the data received and converts it into the appropriate format.

[2043] Step 5:

[2044] Server: Stores patient information, image data, and findings data in a database with unique identifiers.

[2045] Step 6:

[2046] Server: Adds metadata (such as the date and time of storage and information about the attending physician) to the stored data to improve future search performance.

[2047] Program processing flow: Search for similar images and suspected disease names

[2048] Step 7:

[2049] User (doctor): Enters image data and basic information of a new patient for whom a diagnosis is desired into the terminal. If necessary, additional findings are also entered.

[2050] Step 8:

[2051] Terminal: Sends the entered data to the server.

[2052] Step 9:

[2053] Server: Validates the received data and checks for missing data.

[2054] Step 10:

[2055] Server: Converts image data into a standard format and formats it so that it can be easily processed by the AI ​​model.

[2056] Step 11:

[2057] Server: Using generative AI and diffusion models, the server searches for similar images in the database. The algorithm also takes into account the patient's basic information and findings.

[2058] Step 12:

[2059] Server: Extracts the most similar images from the search results and identifies the suspected disease associated with them.

[2060] Step 13:

[2061] Server: Sends similar images and the names of suspected diseases associated with them to the terminal.

[2062] Program processing flow: "Using the emotion engine"

[2063] Step 14:

[2064] Terminal: Displays the similar images and suspected disease names sent from the server on the user interface.

[2065] Step 15:

[2066] Emotion engine: Analyzes the user's (doctor's) facial expressions and tone of voice in real time to assess their confidence in the diagnosis and stress level.

[2067] Step 16:

[2068] User (doctor): Checks the displayed information and makes a diagnostic decision while taking into account feedback from the emotion engine. If the user feels stressed during the process, the emotion engine automatically displays a diagnostic support message.

[2069] Program flow: Feedback and database updates

[2070] Step 17:

[2071] User (doctor): Enters the final diagnosis (e.g., confirmed disease name and additional findings) into the terminal. The emotion evaluation by the emotion engine is also recorded at the same time.

[2072] Step 18:

[2073] Terminal: Sends the input feedback information to the server.

[2074] Step 19:

[2075] Server: Updates the database based on the feedback received, specifically by linking definitive diagnoses and additional findings to existing data to help improve the accuracy of future search results.

[2076] Step 20:

[2077] Server: Based on the feedback information and the evaluation results of the emotion engine, it is also used as training data for the AI ​​model to improve the accuracy of the model.

[2078] This detailed processing flow allows doctors to easily search for similar images and identify suspected diseases when making a diagnosis, and the emotional engine's support improves the reliability of the diagnosis, leading to reduced stress. This increases the accuracy and efficiency of diagnoses, and greatly contributes to reducing the burden on medical professionals.

[2079] Example 2

[2080] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2081] In modern medical settings, rapid and accurate diagnosis using medical images is required for patient diagnosis. However, it is not easy to efficiently handle large amounts of image data during the diagnostic process and accurately identify suspected diseases. Furthermore, the doctor's emotions and stress can affect the accuracy of the diagnosis, which is also a major challenge. Therefore, there is a need for a diagnostic support system that not only quickly searches for similar images and provides an appropriate suspected disease name, but also takes the doctor's emotional state into account.

[2082] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2083] In this invention, the server includes: means for inputting patient medical images; means for inputting basic patient information and imaging findings; database means for storing the medical images, basic information, and imaging findings; means for searching the database for similar images using a generative artificial intelligence model and a diffusion model; means for generating suspected disease names based on the similar images, basic information, and imaging findings; means for presenting the similar images and suspected disease names to the user; emotion engine means for analyzing the user's facial expression and tone of voice to evaluate reliability and stress level; means for inputting diagnostic feedback and storing it together with an emotion evaluation; and means for updating the database based on the stored feedback information to improve the accuracy of subsequent searches and emotion evaluations. This enables rapid and accurate diagnostic support and also enables the provision of appropriate information that takes into account the doctor's emotional state.

[2084] "Patient medical images" means image data of a patient used by a physician for diagnostic or treatment purposes, including formats such as X-rays, CT scans, and MRIs.

[2085] "Basic information" refers to basic data necessary for diagnosis and treatment, such as the patient's age, gender, and medical history.

[2086] "Imaging findings" are data showing detailed observation results, such as the location and shape of lesions identified by doctors based on diagnostic imaging.

[2087] "Database Means" means a computer system for storing and managing patient medical images, baseline information, and imaging findings.

[2088] A "generative artificial intelligence model" is a collection of algorithms that generate new data based on trained data and perform classification and prediction.

[2089] The "diffusion model" is a mathematical technique for identifying similar images by diffusing image data in a high-dimensional space and performing similarity calculations.

[2090] The "emotion engine means" is a software module for analyzing a user's facial expressions and tone of voice to assess their emotional state, such as their trustworthiness or stress level.

[2091] "Diagnostic feedback" refers to data entered by doctors regarding diagnostic results and additional findings, which is used to improve the system's learning accuracy and to support future diagnoses.

[2092] "Feedback information" refers to data such as diagnostic feedback and evaluation results from the emotion engine that the system uses to improve search accuracy and the user interface in future searches.

[2093] "Search and emotion assessment accuracy" refers to the accuracy with which the system identifies similar images and related disease names, and the accuracy with which the emotion engine assesses the user's emotional state.

[2094] MODE FOR CARRYING OUT THE INVENTION

[2095] The present invention relates to a diagnostic support system that uses a patient's medical images, basic information, and imaging findings. This system generates similar images and suspected disease names when a doctor inputs the patient's medical images and information, and further supports diagnosis by recognizing the user's emotions. Specific embodiments of each element of the present invention are described below.

[2096] System configuration

[2097] This system mainly consists of the following elements:

[2098] User (doctor): Enters the patient's medical images, basic information, and imaging findings.

[2099] Terminal: Accepts user input, sends data to the server, and displays the res...

Claims

1. a means for inputting a medical image of a patient; A means for inputting basic information and imaging findings of a patient; a database means for storing the medical images, basic information, and image findings; a means for searching for similar images from the database using a generative artificial intelligence model and a diffusion model; a means for generating a suspected disease name based on the similar image, the basic information, and image findings; means for presenting the similar image and the suspected disease name to a user; A system including:

2. The system of claim 1 , wherein the generative artificial intelligence model and the diffusion model identify the most similar medical images from a database based on basic patient information and imaging findings.

3. The system according to claim 1 , wherein the system updates a diagnostic history including diagnostic feedback based on the information stored in the database to improve search accuracy from the next time onward.

Images