Information processing device, information processing method, and information processing program

Abstract

An information processing device that assists in recalling disease names using the results of analysis of examination images. [Solution] An information processing device is provided, comprising: an analysis unit that obtains the results of analyzing an examination image; a first output unit that outputs a search term using a string of words that combines the location of the abnormal finding obtained from the analysis results of the analysis unit and at least one of the content or size of the abnormal finding; a search unit that searches a database for information on disease names or disease names using the search term output by the first output unit; and a presentation unit that presents disease names or disease names based on the search results from the search unit.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and an information processing program.

Background Art

[0002] It is required for doctors to recall the disease names of patients who visit a medical institution for examination. Medical knowledge in the medical field is constantly increasing, and it is impossible for doctors to memorize all of it. Therefore, technologies for assisting in recalling disease names that can be considered from the content of patients' findings have been disclosed (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] For the examination of patients, in addition to the doctor's inquiry, there are examinations for imaging the internal conditions of patients, such as X-ray examinations, CT (Computed Tomography) examinations, MRI (Magnetic Resonance Imaging) examinations, electrocardiograms, ultrasonic echograms, endoscopes, pathological images, infrared thermographs, and angiographies. It is considered that not only the symptoms of the patient known from the inquiry but also the examination images of the patient's body can be used to assist in recalling the disease name. However, it may be difficult for doctors to recall the disease name just by looking at the examination images.

[0005] In view of the above points, the present disclosure has been made, and an object thereof is to provide an information processing apparatus, an information processing method, and an information processing program that assist in recalling a disease name using the analysis result of an examination image.

Means for Solving the Problems

[0006] According to one aspect of this disclosure, an information processing device is provided, comprising: an analysis unit that obtains analysis results of an examination image; a first output unit that outputs a search term using a string of words that combines the location of the abnormal finding obtained from the analysis results of the analysis unit and at least one of the content or size of the abnormal finding; a search unit that searches a database for information regarding a disease name or disease name using the search term output by the first output unit; and a presentation unit that presents a disease name or disease name based on the search results from the search unit.

[0007] The above-described information processing device may further include a second output unit that outputs a search term by combining the search term output by the first output unit with one or more strings of symptom findings and disease name or disease name provided from one or more of the patient information provided by user input or stored in the medical record system.

[0008] The first output unit may output the search terms by classifying them into the location of the abnormal finding, the content of the abnormal finding, or the size of the abnormal finding.

[0009] The first output unit may further output the search term with the name of the inspection from which the inspection image was obtained.

[0010] The first output unit may output the search term with the size expressed as time-series data.

[0011] The analysis unit may obtain analysis results for multiple examination images obtained from the same subject at different times, and the first output unit may output information on the time progression of each examination image as the search term.

[0012] The analysis unit may obtain the analysis results of the inspection images by providing the inspection images to a large-scale multimodal model and obtaining the output from the large-scale multimodal model.

[0013] The display unit may also present a user interface that allows the user to select an area in the examination image that is considered to be an abnormal finding.

[0014] In another aspect of this disclosure, an information processing method is provided in which a processor obtains the results of analyzing an examination image, outputs a search term using a string that combines the location of the abnormal finding obtained from the analysis results with at least one of the content or size of the abnormal finding, searches a database for information regarding the name of a disease or medical condition using the outputted search term, and presents the name of a disease or medical condition based on the search results.

[0015] In another aspect of this disclosure, an information processing program is provided that causes a computer to obtain the results of analyzing an examination image, output a search term using a string of words that combines the location of the abnormal finding obtained from the analysis results with at least one of the content or size of the abnormal finding, use the output search term to search a database for information about the disease name or medical condition, and present the disease name or medical condition based on the search results. [Effects of the Invention]

[0016] According to this disclosure, it is possible to provide an information processing device, an information processing method, and an information processing program that support the recall of disease names using the results of analysis of examination images. [Brief explanation of the drawing]

[0017] [Figure 1] This is a diagram illustrating an information processing device according to an embodiment of the disclosed technology. [Figure 2] This is a block diagram showing the hardware configuration of an information processing device. [Figure 3] This is a block diagram showing an example of the functional configuration of an information processing device. [Figure 4] This diagram shows an example of a database structure. [Figure 5]A diagram showing an example of a user interface displayed on a user terminal by an information processing device. [Figure 6] A diagram showing an example of a user interface displayed on a user terminal by an information processing device. [Figure 7] A diagram showing an example of a time-series inspection image displayed on a user terminal by an information processing device. [Figure 8] A flowchart showing the flow of information processing by an information processing device.

Embodiments for Carrying Out the Invention

[0018] Hereinafter, an example of an embodiment of the present disclosure will be described while referring to the drawings. In each of the drawings, the same or equivalent components and parts are given the same reference numerals. Also, the dimensional ratios in the drawings are exaggerated for the convenience of explanation and may be different from the actual ratios.

[0019] FIG. 1 is a diagram for explaining an information processing device according to the present embodiment. The information processing device 10 shown in FIG. 1 receives an input of a patient's symptoms and the name of the disease state from a user terminal 20 connected via a network 1, and outputs a search term using the symptoms and the name of the disease state. The user terminal 20 is a terminal used by a doctor or other medical staff as a user. The information processing device 10 is a device that searches a database 30 storing information related to the disease state with the output search term and presents the disease name to the user terminal 20 based on the search results. The database 30 stores, as information related to the disease state, information on cases, information on suspected cases, and information on textbooks describing the disease or the disease state. The information processing device 10 according to the present embodiment outputs a search term using the analysis result of the inspection image of the patient in addition to the patient's symptoms and the name of the disease state when searching the database 30.

[0020] Furthermore, the term "pathological condition name" is an expanded concept of "disease name," referring to conditions where the pathology of the disease is consistent. Therefore, in this disclosure, "pathological condition name" may be considered equivalent to "disease name." For example, "immune-susceptible state" is a pathological condition name, but it is an example of a term that is not a disease name. In addition, as one aspect of this, when registering case information or suspected case information in database 30, considering that multiple pathological conditions may exist in a single case, the information is registered separately for each pathological condition. This prevents pathological condition names unrelated to symptoms from being displayed in the search results. It is also possible to register by combining multiple pathological conditions. This makes it possible to describe sets of symptoms that can only be explained when two or more pathological conditions are combined. For example, wheezing is seen in COPD but rarely in lung cancer, and hemoptysis is often seen in lung cancer but rarely in COPD. However, when searching for the three words "hemoptysis + wheezing + CT = tumor @ lung", the information processing device 10 can combine the two conditions, COPD + lung cancer, and display the search results on the user terminal 20.

[0021] Furthermore, in this embodiment, "subject" is not limited to humans, but is a general concept that includes animals, plants, equipment, facilities, structures, manufacturing lines, infrastructure, and other objects. When applied to non-medical fields, "pathological condition name or disease name" is understood as a broad term that includes status identification names in the target field, such as the name of the object's state, abnormality classification name, failure name, deterioration mode name, pest / disease name, etc.

[0022] Furthermore, the term "inspection image" in this embodiment is not limited to medical images, but includes visible light images, thermography, spectral images, X-ray, ultrasound, electromagnetic wave, lidar, and drone images, waveform visualization images, industrial endoscope images, etc. The term "inspection name" includes a broad definition of the acquisition mode name, referring to the means of acquiring these images (e.g., CT, MRI, fluorescence testing, thermal imaging measurement, vibration diagnosis, AE measurement, drone inspection).

[0023] Furthermore, in this embodiment, "symptom findings" include abnormal signs observed on-site in non-medical areas (e.g., temperature rise, unusual noise, increased vibration, corrosion spots, discoloration, yellowing / curling of leaves, decreased flow rate, excessive current), etc. "Patient information" may be interpreted as supplementary information concerning the object and environment, such as device specifications, operating history, inspection records, cultivation history, weather history, and pest outbreak records.

[0024] The information processing device 10 may be configured, for example, as a web server, and the acceptance of input from the user terminal 20 and the presentation of search results from the database 30 may be implemented in the form of a web page. Therefore, the user terminal 20 has a browser installed for viewing web pages, but it may also have a dedicated application installed for using the search service provided by the information processing device 10.

[0025] In this embodiment, when searching the database 30, the information processing device 10 outputs search terms using the results of the analysis of the patient's examination images in addition to the patient's symptom findings and disease name. This makes it possible to better assist physicians in recalling disease names compared to cases where the results of the examination image analysis are not used.

[0026] Network 1 can be the Internet, an intranet, or any other network, and the communication protocol, type of communication, and scale of communication may be anything. Furthermore, database 30 may be built within the information processing device 10 as shown in Figure 1, or it may be built on a device different from the information processing device 10.

[0027] Figure 2 is a block diagram showing the hardware configuration of the information processing device 10.

[0028] As shown in Figure 2, the information processing device 10 includes a CPU (Central Processing Unit) 11, ROM (Read Only Memory) 12, RAM (Random Access Memory) 13, storage 14, input unit 15, display unit 16, and communication interface (I / F) 17. Each component is connected to the others via a bus 19 so that they can communicate with each other.

[0029] The CPU 11 is a central processing unit that executes various programs and controls various parts. Specifically, the CPU 11 reads a program from the ROM 12 or storage 14 and executes the program using the RAM 13 as a working area. The CPU 11 controls each of the above components and performs various calculations according to the program recorded in the ROM 12 or storage 14. In this embodiment, the ROM 12 or storage 14 stores an information processing program that outputs a search term based on user input and the analysis results of the examination image, searches the database 30 which stores information about the pathological condition using the output search term, and presents information about the disease to the user terminal 20 based on the search results.

[0030] ROM12 stores various programs and data. RAM13 temporarily stores programs or data as a working area. Storage14 consists of a storage device such as an HDD (Hard Disk Drive), SSD (Solid State Drive), or flash memory, and stores various programs, including the operating system, and various data.

[0031] The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used for various types of input.

[0032] The display unit 16 is, for example, a liquid crystal display and displays various information. The display unit 16 may also function as an input unit 15 by employing a touch panel system.

[0033] The communication interface 17 is an interface for communicating with other devices such as the user terminal 20, and standards such as Ethernet®, FDDI, and Wi-Fi® are used.

[0034] When executing the above information processing program, the information processing device 10 uses the above hardware resources to implement various functions. The functional configuration implemented by the information processing device 10 will now be described.

[0035] Figure 3 is a block diagram showing an example of the functional configuration of the information processing device 10.

[0036] As shown in Figure 3, the information processing device 10 has the following functional configuration: acquisition unit 101, analysis unit 102, first output unit 103, second output unit 104, standardization unit 105, search unit 106, and presentation unit 107. Each functional configuration is realized by the CPU 11 reading and executing an information processing program stored in the ROM 12 or storage 14.

[0037] The acquisition unit 101 acquires user input from the user terminal 20. The user input acquired by the acquisition unit 101 includes the patient's symptom findings and disease name. The acquisition unit 101 also acquires examination images of the same patient as the patient with the symptom findings and disease name. The examination images are, for example, images obtained from examinations that photograph the inside of the patient's body, such as X-ray examinations, CT scans, and MRI scans.

[0038] The analysis unit 102 analyzes the patient's examination images acquired by the acquisition unit 101 to obtain the analysis results of the examination images. The analysis unit 102 may use any pre-trained model for the analysis of the examination images. This pre-trained model is trained to output information about any abnormal findings when an examination image is input. In other words, the analysis unit 102 provides the examination image to the pre-trained model and obtains the analysis results of the examination images by obtaining the output from the pre-trained model. The analysis unit 102 can obtain findings from the examination images by analyzing them.

[0039] In this embodiment, the analysis unit 102 may use a Large Multimodal Model (LMM) for analyzing the examination images. A Large Multimodal Model is a model that can respond in natural language text to inputs of multiple modalities, such as images, in addition to natural language text. This Large Multimodal Model is trained to output any abnormal findings in natural language text when an examination image is input. That is, the analysis unit 102 obtains the analysis results of the examination images by providing the examination images to the Large Multimodal Model and obtaining the output from the Large Multimodal Model.

[0040] The first output unit 103 outputs search terms related to abnormal findings based on the analysis results of the examination images obtained by the analysis unit 102. Specifically, the first output unit 103 outputs search terms using strings that combine the location of the abnormal finding obtained from the analysis results with at least one of the content or size of the abnormal finding. The first output unit 103 may also output search terms classified by the name of the examination from which the examination images were obtained, the location of the abnormal finding, and the content or size of the abnormal finding. In other words, the first output unit 103 outputs search terms that combine text indicating the name of the examination from which the examination images were obtained with text relating to the location of the abnormal finding, the content or size of the abnormal finding. The first output unit 103 may also output search terms according to the shape, density, or contrast of the abnormal finding with surrounding tissue. For example, suppose the analysis unit 102 found a cavity in the lung as a result of analyzing the examination images of a CT scan. In this case, the first output unit 103 outputs the search term "CT=cavity@lung" as a search term related to the abnormal finding. In this case, "CT" is an example of a phrase indicating the name of the examination from which the examination image was obtained, and "Cavity@Lung" is an example of a phrase relating to the location of the abnormal finding and the content of the abnormal finding. When the first output unit 103 outputs a search term by combining phrases relating to the size of the abnormal finding, it may output a search term that expresses the size as time-series data. Furthermore, the analysis unit 102 may use machine learning, deep learning, or image processing algorithms for analysis. Moreover, as one configuration, the analysis unit 102 and the first output unit 103 may be integrated so that the examination image is passed directly to the LMM and the search term is output. In addition, the symptom findings and disease name used as search terms may be received directly from the user via voice or keyboard input.

[0041] Furthermore, the analysis unit 102 is not limited to machine learning models (LMM, LLM, CNN, Transformer, etc.), but may use any analysis method such as threshold determination, template matching, clustering, time series statistical models, signal processing algorithms, rule-based methods, or hybrid methods.

[0042] Furthermore, the analysis results are not limited to those obtained through automated analysis, but may also include results extracted from text information such as observation reports, inspection reports, maintenance records, and agricultural guidance records created manually, using natural language processing.

[0043] Furthermore, for multiple data points acquired from the same subject at different times, the temporal changes (increase, decrease, constancy, periodicity, drift) may be statistically estimated, and a summary of these changes (PR, CR, PD, or equivalent classification terms in the industrial / agricultural domain) may be generated as the analysis output.

[0044] The second output unit 104 outputs multiple search terms by combining the search term output by the first output unit 103 with the symptom findings and disease name acquired by the acquisition unit 101. For example, if the first output unit 103 outputs the search term "CT=cavity@lung" as described above, and the acquisition unit 101 acquires the symptom findings and disease name "fever" and "body temperature 38 degrees", the second output unit 104 outputs the search terms "fever body temperature 38 degrees CT=cavity@lung".

[0045] The second output unit 104 generates search terms by combining the string of abnormal findings obtained from the analysis results of the analysis unit 102 with the symptom findings and disease name (disease name) input or provided via the user terminal 20. This includes treating the string as a logical AND search or a phrase search. For example, the second output unit 104 tags the abnormal location "lung" and the abnormal content "cavity" with the user input "fever". This allows the database 30 to be searched using the phrase "cavity@lung fever" as the search term.

[0046] This embodiment is not limited to cases where symptoms, findings, and disease names are manually entered on the input screen displayed on the user terminal 20. It also includes forms in which patient information stored in medical record systems such as electronic medical records (EMR) and picture archiving and communication systems (PACS) is referenced and used as part of the search string. For example, the user terminal 20 automatically or semi-automatically acquires symptoms and findings such as "fever," "body temperature 38.5°C," and "cough" recorded in the electronic medical record into the input area, and the second output unit 104 can generate search terms by combining the information acquired from the electronic medical record by the user terminal 20 with abnormal findings (location, content, size) obtained by the analysis unit 102. In this case, the user may select necessary items from the medical information on the electronic medical record and have them imported into the acquisition unit 101 as search terms via the user terminal 20. By using this form, the input workload of medical staff can be reduced, and searches using more accurate data can be performed.

[0047] Another example of this embodiment is the implementation of multiple search filters. In the example above, the search terms are separated by spaces, etc., but for example, one per line, Location: Right lower lung field, Findings: Cavity present, mass Size: about the size of a small bean, Location: upper right lung field Findings: Cavity-filled mass, size: about the size of a chicken egg, location: right upper lung field It is also possible to apply multiple filters, as shown above.

[0048] In this embodiment, the "search term" may be a string, a tag column, a key-value pair, an embedded vector (vector representation), or a combination thereof.

[0049] The first output unit 103 and the second output unit 104 may generate search terms by logically combining (AND / phrase) at least two items from the attributes of the abnormal finding, such as location, content, size, shape, density, texture, intensity, signal frequency, spectral peak, and temporal trend.

[0050] When using embedding vectors, the search unit 106 may obtain candidate entries by vector nearest neighbor search (kNN, ANN) and, if necessary, merge them with keyword search (Hybrid Retrieval) to perform re-ranking.

[0051] Interactive generation may be used to continuously update (narrow down) search terms based on user input, automatic integration of external information such as medical records, inspection records, and cultivation records, or selection of candidate words via the UI.

[0052] The standardization unit 105 standardizes the terminology for symptom findings and disease names in order to unify the search terms output by the second output unit 104. In unifying the terminology, the standardization unit 105 may obtain words by means of natural language processing, rule-based modification, vector search on the word list, or word search by calculating the edit distance on the word list.

[0053] As another example of this embodiment, the first output unit 103 may generate search terms that express the change in the size of abnormal findings as time-series data from the analysis results of multiple examination images obtained from the same patient at different imaging timings by the analysis unit 102. For example, the first output unit 103 may assign the increase or decrease in tumor size obtained from CT images taken every three months to the search terms in the format of "tumor 10mm (January 2024) → 12mm (April 2024) → 15mm (July 2024)". By generating search terms that express the change in the size of abnormal findings as time-series data, the first output unit 103 can be used to search for the progression over time or the effectiveness of treatment. Furthermore, the standardization unit 105 may convert the representation of the time-series information generated by the first output unit 103 into a unified standard for search terms. For example, the standardization unit 105 may convert changes in the size of abnormal findings into notations such as unchanged, increased, decreased, disappeared, CR (complete response), PR (partial response), NC (no change), and PD (progressive disease). These notations represent changes in size over time and changes in treatment interventions.

[0054] The standardization unit 105 performs lexical normalization, including a thesaurus, a technical term dictionary, a unit conversion table, normalization rules for ambiguous expressions, and edit distance / vector proximity search. It maps everyday expressions (e.g., "the size of a chicken egg," "the size of a red bean") to numerical units.

[0055] The search unit 106 searches the database 30 for information related to the pathology using the search term output by the first output unit 103 or multiple search terms output by the second output unit 104. For example, the search unit 106 outputs a query to search the database 30 using the search term output by the first output unit 103 or multiple search terms output by the second output unit 104, and searches the database 30 using the output query. Information related to the pathology includes at least one of a case or a suspected case, or a textbook. That is, the information stored in the database 30 includes at least one of a case or a suspected case, or a textbook, as information related to the pathology.

[0056] Here is an example of the data structure of database 30. Figure 4 is a diagram showing an example of the data structure of database 30. In this embodiment, database 30 has a data type column, a disease state column, a disease state findings column, an ID column, a case ID column, and a case number column.

[0057] The data type column is used to identify whether a record is a collection of multiple cases or a record of a single case. A data type of 1 indicates a record of multiple cases, while a data type of 2 indicates a record of a single case.

[0058] The "Pathology" column stores the name of the pathology. The "Pathology Findings" column stores information about the findings of the pathology. The "ID" column stores the ID that identifies the pathology. The "Case ID" column stores the case ID that identifies the case data related to the pathology. If the data type is 1, the Case ID column stores multiple case IDs, corresponding to the ID column of the case information held in a separate table in the same configuration as in Figure 4. The "Number of Cases" column stores the number of cases of the pathology; if the data type is 2, the Number of Cases column stores 1.

[0059] It should be noted that the example structure of database 30 shown in Figure 4 is merely an example, and it goes without saying that an appropriate structure can be selected depending on the information to be searched. Furthermore, database 30 may not be a single database but may consist of multiple databases. Moreover, the databases that make up database 30 may consist of a single table or multiple tables.

[0060] The processing performed by the search unit 106 is not limited to searching, but includes general search processes such as similarity search, clustering, link analysis, knowledge graph search, recommendation, and inference (rule inference / graph inference). The results may be presented as matching scores, rankings, cluster IDs, explanatory evidence, recommended actions, etc.

[0061] The presentation unit 107 presents a user interface to the user terminal 20 for searching for disease names. The presentation unit 107 then presents information about the disease to the user terminal 20 based on the search results from the search unit 106. The presentation unit 107 presents the information about the disease to the user terminal 20, for example, in the form of a web page.

[0062] Here, an example of a user interface displayed by the presentation unit 107 on the user terminal 20 is shown. Figure 5 is a diagram showing an example of a user interface displayed on the user terminal 20 by the information processing device 10. The user interface 200 shown in Figure 5 is implemented, for example, in the form of a web page.

[0063] The user interface 200 shown in Figure 5 includes an input area 201 for the user of the user terminal 20 to input the name of the symptom and pathological condition they wish to search for, an image upload button 202 for sending the examination image to the information processing device 10, and a search button 203 for causing the information processing device 10 to perform a search for information related to the pathological condition from the database 30. When the user of the user terminal 20 enters the name of the symptom and pathological condition into the input area 201, uploads the examination image to the information processing device 10 using the image upload button 202, and selects the search button 203, the information processing device 10 performs a search for information related to the pathological condition from the database 30 and presents the search results to the user interface 200.

[0064] In another embodiment, the presentation unit 107 may present a graphical user interface that allows the user to select an abnormal area highlighted on the examination image. When the user selects a candidate such as "lung field" or "frontal lobe," the area information is overwritten or added to the analysis results of the analysis unit 102 and reflected in the final generated search term. This makes it possible to improve search accuracy, including preventing misrecognition and selection from multiple candidates.

[0065] Figure 6 shows an example of a user interface displayed on a user terminal 20 by an information processing device 10. The user interface 200 shown in Figure 6 is implemented, for example, in the form of a web page. The user interface 200 shown in Figure 6 is an example of the presentation of search results for information about a disease.

[0066] The user interface 200 shown in Figure 6 includes an input area 201, an image upload button 202, a search button 203, as well as a display area 211 for a summary of search results and a display area 212 for case report search results.

[0067] The display area 211 of the search results summary displays the diseases that can be thought of based on the symptom findings and disease name entered in the input area 201 and the examination images uploaded by the image upload button 202 and analyzed by the information processing device 10, separated by medical department. The presentation unit 107 may also present the diseases to the user interface 200 without separating them by medical department. The search results summary displayed in the display area 211 of the search results summary is a summary of the search results obtained by the information processing device 10 searching the database 30 based on the content entered in the input area 201.

[0068] The case report search results display area 212 displays the number of cases of the disease that can be inferred from the symptom findings and disease name entered in the input area 201 and the examination images uploaded by the image upload button 202 and analyzed by the information processing device 10. When the user of the user terminal 20 selects a disease name presented in the case report search results display area 212, the presentation unit 107 presents detailed case information of that disease to the user terminal 20. The disease and the number of cases of the disease displayed in the case report search results display area 212 are the results obtained by the information processing device 10 searching the database 30 based on the content entered in the input area 201.

[0069] The information processing device 10 presents the user interface 200 shown in Figure 6 to the user terminal 20, thereby enabling the user of the user terminal 20 (a medical professional such as a doctor) to recall the name of the disease using the results of the analysis of the examination images.

[0070] When the presentation unit 107 presents detailed case information of a disease state to the user terminal 20, if corresponding examination images are registered in the database 30, it may also present those examination images to the user terminal 20. In this case, if the examination images are registered in the database 30 in chronological order, the presentation unit 107 may also present the examination images to the user terminal 20 along with the time when the examination images were taken. Figure 7 shows an example of chronological examination images displayed on the user terminal 20 by the information processing device 10. The chronological examination images 221 shown in Figure 7 consist of, for example, three examination images 221a, 221b, and 221c. By presenting the chronological examination images 221 together with detailed case information of the disease state, the presentation unit 107 can assist the user of the user terminal 20 in recalling the name of the disease by comparing it with the examination images uploaded to the information processing device 10. Furthermore, the display unit 107 can, in one example, overlay two or more images as a time-series image, add different colors or symbols to the lesion area, or process the boundaries to show how the size of the lesion has changed over time, making it easier for the user to intuitively grasp the changes over time. When overlaying images, the display unit 107 may also determine the color of the most recent image and the image older than the most recent to indicate whether the change in size has increased or decreased.

[0071] The display unit 107 may display not only disease names and condition names, but also non-medical items such as malfunction names, abnormality classification names, degradation mode names, pest / disease names, etc. The display format can be any format, such as text, tables, tags, thumbnail images, focus area overlays, statistical graphs, or timelines.

[0072] The presentation unit 107 may also present each candidate with explainability information (heatmap of the evidence area, referenced rules, cited cases, contributing features) and a confidence score.

[0073] The presentation unit 107 may generate and present summaries and recommendations, and such summaries may be generated by integrating the contents of multiple information sources obtained from the search unit 106 using a summarization model.

[0074] When using a large-scale multimodal model, the analysis unit 102 can obtain output expressed in natural language, such as "suspected tumor in the left lung" or "possible pneumonia with cavity formation," by inputting examination images. The first output unit 103 performs text analysis on the output from the large-scale multimodal model to extract phrases such as "lung tumor" and "cavity @ left lung," and combines these as search terms to streamline the search of the database 30.

[0075] Another example of this embodiment is that when the analysis unit 102 extracts the location, content, and size of abnormal findings using artificial intelligence (AI), particularly a large-scale multimodal model, it may also have a function to show the user the basis on which the search terms were generated. For example, the analysis unit 102 may visualize a weighted heatmap or area of ​​interest on the examination image, enclose it in an arbitrary shape such as a rectangle, and show which part was used as the basis for selecting words such as "pneumonia" or "cavity." This allows the user (doctor or medical staff) to intuitively understand why the AI ​​selected "cavity@left lung" as a search term.

[0076] In this case, the display unit 107 may also display the "site of AI judgment" and "confidence score" for each candidate search term. For example, if the abnormal finding is in the left lung field, the display unit 107 may display an explanation such as, "The image pixel values ​​in the left lung field fall within the range of XX, and it was diagnosed as a severe cavitary lesion, so 'cavitary@left lung' was generated." By displaying the information in this way, the display unit 107 makes it easier for the user to understand the AI's reasoning process, adds transparency to the generation of search terms by the AI ​​(Explainable AI), and increases the reliability of the search results.

[0077] In this embodiment, the first output unit 103 may generate text that combines at least two of the following: the location where the abnormality was observed (lungs, liver, bone, etc.), the content of the abnormality (cavity, tumor, inflammation, etc.), and the size of the abnormality (long diameter, short diameter, volume, etc.). For example, the first output unit 103 may generate text that simultaneously indicates the location and content or size, such as "tumor in the left lung, 10 mm" or "cavity in the right lung, 15 mm". By generating text in this manner, it is possible to improve the search accuracy of the database 30.

[0078] In this embodiment, the size of abnormal findings may not only be expressed using specific numerical values ​​such as "10mm" or "15mm," but may also include everyday or analog size expressions such as "the size of a chicken egg" or "the size of a red bean." When these everyday or analog size expressions are received, the standardization unit 105 may create a dictionary of correspondences such as "the size of a chicken egg = 40mm" and "the size of a red bean = 10mm," and convert them into a unified standard (numerical values, etc.) as search terms. For example, if the input is "tumor in the right lung, the size of a chicken egg," the standardization unit 105 may convert it into a phrase such as "tumor in the right lung, 40mm" before generating a search term. This makes it possible to search the database 30 more efficiently compared to not converting to a unified standard as a search term, even when various expressions are mixed together, thereby improving search accuracy and versatility.

[0079] The information processing device 10, having the configuration shown in Figure 3, can present disease names obtained from the database 30 to the user terminal 20. By presenting disease names obtained from the database 30 to the user terminal 20, the information processing device 10 can use the results of the analysis of the examination images to support the user of the user terminal 20 (a medical professional such as a doctor) in recalling the disease name.

[0080] Next, the operation of the information processing device 10 will be explained.

[0081] Figure 8 is a flowchart showing the flow of information processing by the information processing device 10. Information processing is performed when the CPU 11 reads an information processing program from the ROM 12 or storage 14, loads it into the RAM 13, and executes it.

[0082] In step S101, the CPU 11 acquires user input from the user terminal 20. The user input acquired by the CPU 11 includes the patient's symptom findings and disease name. The CPU 11 also acquires examination images of the same patient as the patient with the symptom findings and disease name. The examination images are, for example, images obtained from examinations that capture the inside of the patient's body, such as X-rays, CT scans, and MRI scans.

[0083] Following step S101, in step S102, the CPU 11 analyzes the examination image acquired in step S101. The CPU 11 may use any pre-trained model to analyze the examination image. This pre-trained model is trained to output information about any abnormal findings when an examination image is input. In other words, the CPU 11 provides the examination image to the pre-trained model and obtains the output from the pre-trained model to obtain the analysis result of the examination image. The CPU 11 can obtain findings from the examination image through its analysis.

[0084] In this embodiment, the CPU 11 may use a large-scale multimodal model (LMM) for analyzing the inspection images.

[0085] Following step S102, in step S103, the CPU 11 outputs a search term using the analysis of the examination image. For example, suppose the CPU 11 analyzes the examination image of a CT scan and finds a cavity in the lung. In this case, the CPU 11 outputs the search term "CT=cavity@lung" as a search term related to the abnormal finding.

[0086] Following step S103, in step S104, the CPU 11 outputs multiple search terms by combining the search term output in step S103 with the symptom findings and disease name obtained in step S101. For example, if the CPU 11 outputs the search term "CT=cavity@lung" as described above, and obtains the symptom findings and disease name "fever" and "body temperature 38 degrees", the CPU 11 outputs the search terms "fever body temperature 38 degrees CT=cavity@lung".

[0087] Following step S104, in step S105, the CPU 11 searches the database 30 using the search term output in step S104.

[0088] Following step S105, in step S106, the CPU 11 presents the search results to the user terminal 20 based on the search results from step S105. The CPU 11 presents the disease names obtained from the database 30 to the user terminal 20 as search results, for example in the form of a web page.

[0089] The information processing device 10 can present the disease name obtained from the database 30 to the user terminal 20 by executing the series of processes shown in Figure 8. By presenting the disease name obtained from the database 30 to the user terminal 20, the information processing device 10 can use the results of the analysis of the examination images to support the user of the user terminal 20 (a medical professional such as a doctor) in recalling the disease name.

[0090] Examples and modified versions of this embodiment are shown below.

[0091] (Example 1) Differential diagnosis by linking imaging findings and symptomatic findings (respiratory system) This example demonstrates how to generate search terms by combining the analysis results of chest CT images with symptom findings stored in medical records, and then present differential diagnoses. The subjects are outpatients, and their electronic medical records show findings of cough, low-grade fever, and weight loss.

[0092] The acquisition unit 101 acquires chest CT images uploaded from the user terminal 20, and the analysis unit 102 applies a trained model to the images to detect abnormal findings, including cavity formation, in the right upper lung field. The analysis results are structured as "Location = Right upper lung field, Content = Cavity, Longest diameter = 20 mm".

[0093] The first output unit 103 generates a search term, "CT / cavity@right upper lung," based on the analysis results, using a combination of the examination name (CT) and the location / content. Attributes such as shape and density may be added as needed to create a tag column (e.g., "shape=thin wall", "surroundings=infiltrate").

[0094] The second output unit 104 concatenates the symptom findings (cough, fever in the 38°C range, weight loss over several weeks) automatically acquired by the acquisition unit 101 from the electronic medical record with the search term as a logical AND operation, generating a composite search term "cough fever weight loss CT / cavity@right upper lung". The standardization unit 105 normalizes variations in terminology.

[0095] The search unit 106 searches the database 30 using a compound search term. The database 30 stores case information, pseudo-cases, and textbook entries, and the search unit 106 extracts candidates using a combination of keyword search and similarity search using embedding vectors, and then performs re-ranking (see Figure 6).

[0096] The display unit 107 summarizes and displays candidates that match the patient's findings, such as tuberculosis, lung abscess, and nocardiosis, by medical department, and includes a preview of the number of reference cases, confidence score, and evidence area (area of ​​interest on the image) for each candidate. When a candidate name is selected on the user interface, images and descriptions of related cases are displayed.

[0097] According to this embodiment, by integrating imaging findings and symptomatic findings in the search, differential diagnoses, including rare diseases, can be comprehensively recalled, allowing for quick access to useful information within the clinical flow. This reduces the reliance on the knowledge of individual healthcare professionals, contributing to the prevention of oversights and the efficiency of clinical practice.

[0098] (Example 2) Support for evaluating treatment effectiveness using size changes in time-series images (tumor) This example demonstrates how to use the analysis results of images acquired from the same subject at different time points to include the time-series progression of tumor size as a search term, thereby presenting insights that contribute to evaluating treatment effectiveness.

[0099] The acquisition unit 101 acquires chest CT images before treatment (January), and at the first post-treatment evaluation (April) and second evaluation (July). The analysis unit 102 extracts the tumor area and calculates the progression from 30mm to 20mm to 25mm in diameter. The standardization unit 105 normalizes the progression to "equivalent to reduction (PR) → regrowth (PD)".

[0100] The first output unit 103 generates a search term "tumor@right lung=30→20→25mm (January→April→July)" which includes the time-series size values ​​and imaging dates, and the second output unit 104 adds the treatment regimen (drug name, administration schedule) as external information.

[0101] The search unit 106 uses the trends and treatment information as clues to search the database 30 for knowledge entries related to treatment resistance, relapse, evidence for second-line drugs, indications for additional genetic testing, etc. The presentation unit 107 displays a list of summaries of candidate findings, citations of relevant guideline sections, and statistics of past cases.

[0102] According to this embodiment, time-series evaluations, which tend to rely on subjective comparisons, can be reinforced with quantitative information and retrieval, thereby improving objectivity and reproducibility, and supporting decision-making regarding changes in treatment strategies.

[0103] (Modification 1) Agricultural field: Crop disease diagnosis support (tomato) This modified example illustrates the application of the information processing device of the present disclosure to support crop disease diagnosis in the agricultural field. The subject is a tomato leaf, and the user terminal 20 uploads a visible image taken in the field.

[0104] The analysis unit 102 extracts yellowing mosaic patterns on the leaves and curling of the leaf margins as abnormal findings. The first output unit 103 generates search terms such as "yellowing mosaic @ tomato leaves" and "curled leaves @ tomato leaves".

[0105] The second output unit 104 acquires external information such as cultivation history (sowing time, fertilization), recent weather, and pest occurrence records (whiteflies visually observed) and combines it with the search term. The standardization unit 105 normalizes common expressions into standard vocabulary (disease names, pest names).

[0106] The search unit 106 searches the disease knowledge base as the database 30, and the presentation unit 107 presents a list of candidates such as tomato yellow leaf curl virus and late blight, characteristic images of each candidate, the conditions for occurrence, and recommended countermeasures (control, isolation, materials).

[0107] According to this modified version, a search combining image findings and field information allows growers to quickly access appropriate countermeasures and contributes to reducing yield losses. In this modified version, the output equivalent to "disease name" is presented as "disease name."

[0108] (Variation 2) Industrial field: Equipment maintenance and fault diagnosis support (motor bearings) This modified example illustrates how, in the maintenance of manufacturing equipment, the results of analysis of thermographic images and vibration sensor data are converted into search terms, and potential causes and recommended countermeasures are presented from a failure knowledge base. The subject of this experiment is an electric motor.

[0109] The analysis unit 102 detects the high-temperature region (40°C above normal) and the increase in broadband vibration of the bearing section. The first output unit 103 generates "Motor bearing / high temperature" and "Motor bearing / high vibration". The second output unit 104 combines equipment information such as model, cumulative operating time, and load conditions.

[0110] The search unit 106 searches for knowledge entries of failure modes and degradation modes in the database 30, and the presentation unit 107 presents candidates such as lubrication failure, misalignment, and foreign matter contamination in order of reliability, and lists recommended actions (lubrication, parts replacement, alignment adjustment) and areas of interest for the basis (thermal image heatmap) for each candidate.

[0111] According to this modified version, by projecting abnormal signs at the site onto search keys, it is possible to support early cause estimation and appropriate countermeasure planning without relying on the experience of experts. In this modified version, the output is presented as "fault name" or "abnormality classification name".

[0112] According to the information processing device 10 of the embodiment of this disclosure, it is possible to support the recall of disease names using the analysis results of examination images. Furthermore, according to the information processing device 10 of the embodiment, the analysis results can be projected onto search terms that combine with context such as on-site records, symptoms, and driving history, and candidates and evidence directly related to on-site operations such as medical treatment, maintenance, and cultivation can be presented from the knowledge base. This provides quantitative and reproducible support for decision-making processes that tend to depend on the knowledge and experience of individual experts, and enables the provision of insights with a practical response time even in situations with significant time constraints.

[0113] Although embodiments of the present disclosure have been described in detail above with reference to the attached drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person with ordinary skill in the art of the present disclosure can conceive of various modifications or alterations within the scope of the technical idea described in the claims, and these modifications or alterations are also understood to naturally fall within the technical scope of the present disclosure. For example, in one aspect of the present disclosure, the information processing device may generate text that will become a radiation report from inspection images of a radiation inspection using an LMM, and output search terms from the text generated by the LMM after natural language processing or generation processing by an LLM.

[0114] Furthermore, the effects described in the above embodiments are descriptive or illustrative, and are not limited to those described in the above embodiments. In other words, the technology relating to this disclosure may produce other effects that would be obvious to a person of ordinary skill in the art of this disclosure from the descriptions in the above embodiments, in addition to or in lieu of the effects described in the above embodiments.

[0115] Furthermore, the information processing that the CPU reads and executes in each of the above embodiments may be executed by various processors other than the CPU. Examples of such processors include PLDs (Programmable Logic Devices) such as FPGAs (Field-Programmable Gate Arrays) whose circuit configuration can be changed after manufacturing, and dedicated electrical circuits that are processors with circuit configurations specifically designed to execute specific processing, such as ASICs (Application Specific Integrated Circuits). In addition, the information processing may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (for example, multiple FPGAs, and a combination of a CPU and an FPGA). More specifically, the hardware structure of these various processors is an electrical circuit that combines circuit elements such as semiconductor elements.

[0116] Furthermore, while the above embodiments describe a configuration in which the information processing program is pre-stored (installed) in ROM or storage, the invention is not limited thereto. The program may be provided in a form recorded on a non-transitory recording medium such as a CD-ROM (Compact Disk Read Only Memory), DVD-ROM (Digital Versatile Disk Read Only Memory), or USB (Universal Serial Bus) memory. The program may also be provided in a form that is downloaded from an external device via a network. This disclosure may also apply to program products. Each functional unit may be implemented within a single device or distributed between clients / servers / edges. The database 30 may be local, remote, cloud, or hybrid in configuration, and may include data synchronization, caching, and federation. Processing can be performed in real time or asynchronous batch. [Explanation of Symbols]

[0117] 1 Network 10 Information Processing Devices 20 User Terminals 30 databases 101 Acquisition Department 102 Analysis Department 103 First Output Section 104 Second Output Section 105 Standardization Department 106 Search Section 107 Presentation section

Claims

1. An analysis unit that obtains the results of the analysis of the examination images, A first output unit outputs a search term using a string of words that combines the location of the abnormal finding obtained from the analysis results of the aforementioned analysis unit with at least one of the contents or size of the abnormal finding. A search unit that uses the search term output by the first output unit to search a database for information regarding disease names or disease names, A display unit that presents a disease name or disease name based on the search results from the aforementioned search unit, An information processing device equipped with the following features.

2. The information processing apparatus according to claim 1, further comprising a second output unit that outputs a search term by combining the search term output by the first output unit with one or more strings of symptom findings and disease name or disease name provided from one or more of the patient information provided from user input or stored in the medical record system.

3. The information processing apparatus according to claim 1, wherein the first output unit outputs the search term by classifying it into a part with an abnormal finding, the content of the abnormal finding, or the size of the abnormal finding.

4. The information processing apparatus according to claim 1, wherein the first output unit further outputs the search term with the inspection name from which the inspection image was obtained.

5. The information processing apparatus according to claim 1, wherein the first output unit outputs the search term with the size expressed as time-series data.

6. The analysis unit obtains the analysis results of multiple examination images obtained from the same subject at different time points, The information processing apparatus according to claim 1, wherein the first output unit outputs information on the time progression of each of the inspection images as the search term.

7. The information processing apparatus according to claim 1, wherein the analysis unit provides the inspection image to a large-scale multimodal model and obtains the output from the large-scale multimodal model to obtain the analysis result of the inspection image.

8. The information processing apparatus according to claim 1, wherein the display unit presents a user interface for the user to select a region considered to be an abnormal area in the examination image.

9. The processor, After obtaining the results of the analysis of the examination images, A search term is output using a string containing a combination of the location of the abnormal finding obtained from the analysis results and at least one of the characteristics or size of the abnormal finding. Using the outputted search terms, search the database for information related to the disease name or condition name. Based on the search results, present the name of the condition or disease. An information processing method that performs a process.

10. On the computer, After obtaining the results of the analysis of the examination images, A search term is output using a string containing a combination of the location of the abnormal finding obtained from the analysis results and at least one of the characteristics or size of the abnormal finding. Using the outputted search terms, search the database for information related to the disease name or condition name. Based on the search results, present the name of the condition or disease. An information processing program that executes a process.

Images