Information processing device, information processing method, and program

Abstract

An information processing device (1) is characterized by comprising: a storage unit that associates medical information for a first patient for whom treatment was completed with a feature amount of the medical information, and stores the medical information; an acquisition unit that acquires medical information for a second patient to be treated; an extraction unit that extracts the feature amount of the medical information for the second patient; a retrieval unit that retrieves the medical information of the first patient that is similar to the medical information of the second patient, on the basis of the extracted feature amount; and an output unit that outputs the retrieved medical information of the first patient.

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In the medical field, there is a technology for a search system for searching for patient cases. For example, Patent Document 1 discloses an image search device that inputs a patient's examination image into a neural network to classify it into multiple types of lesions, searches a case database for similar case images based on the classification results, and further accepts input of the position and name of a key finding in the examination image to search for similar case images based on the position and name of the specified key finding. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-82881 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the invention of Patent Document 1, the user needs to specify the position and name of the key finding that serves as a search key, which is not necessarily a very convenient method for the user.

[0005] In one aspect, an object of the present invention is to provide an information processing device or the like that can suitably search for similar cases. [Means for solving the problem]

[0006] According to one aspect, an information processing device stores medical information of a first patient who has already undergone treatment. medical information including progress information indicating whether or not complications have occurred after treatment of the first patient; a storage unit that stores the medical information in association with a feature amount of the medical information, an acquisition unit that acquires the medical information of a second patient who is a treatment target, and an extraction unit that extracts the feature amount of the medical information of the second patient. a calculation unit that calculates a similarity between the medical information of the first patient and the medical information of the second patient based on a feature amount of the medical information of the first patient and a feature amount of the medical information of the second patient;Based on the above, the medical information of the second patient is similar to that of the Multiple a search unit that searches for medical information of the first patient; According to the search ranking determined according to the similarity, Searched Multiple an output unit that outputs the medical information of the first patient; If the searched first patient has a complication, the search unit lowers the search ranking of the medical information of the first patient. It is characterized by: [Effects of the Invention]

[0007] In one aspect, similar cases can be suitably searched for. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is an explanatory diagram illustrating an example of the configuration of a case search system. [Figure 2] FIG. 2 is a block diagram illustrating an example of the configuration of a server. [Figure 3] FIG. 2 is an explanatory diagram illustrating an example of a record layout of a case DB. [Figure 4] FIG. 1 is an explanatory diagram showing an overview of a first embodiment. [Figure 5] FIG. 10 is an explanatory diagram showing an example of a display screen of similar cases. [Figure 6] 10 is a flowchart showing the steps of a learning model generation process. [Figure 7] 10 is a flowchart showing a procedure for searching for similar cases. [Figure 8] FIG. 10 is an explanatory diagram of a learning model according to the second embodiment. [Figure 9] 10 is a flowchart showing the procedure of a process for generating a learning model according to the second embodiment. [Figure 10] 10 is a flowchart showing the procedure of a search process for similar cases according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] The present invention will be described in detail below with reference to the drawings showing embodiments thereof. (Embodiment 1) Fig. 1 is an explanatory diagram showing an example of the configuration of a case retrieval system. In this embodiment, a case retrieval system is described which retrieves, from a database storing the medical information of a first patient who has already undergone vascular treatment, medical information of a first patient that is similar to the medical information of a second patient who will newly undergo vascular treatment. The case retrieval system includes an information processing device 1 and an imaging diagnostic device 2. The information processing device 1 and the imaging diagnostic device 2 are communicatively connected to a network N such as a LAN (Local Area Network) or the Internet.

[0010] Although the present embodiment will be described taking vascular treatment as an example, the target hollow organ is not limited to blood vessels, and may be other hollow organs such as the bile duct, pancreatic duct, bronchi, and intestines.

[0011] The diagnostic imaging device 2 is a device unit that acquires medical images of a patient's hollow organs, and is, for example, an IVUS (Intravascular Ultrasound) device that performs ultrasound examinations using a catheter 21. The diagnostic imaging device 2 includes the catheter 21, an image processing device 22, and a display device 23. The catheter 21 is a medical instrument that is inserted into the blood vessels of a subject, and includes a piezoelectric element that transmits ultrasound waves and receives reflected waves from within the blood vessels. The diagnostic imaging device 2 generates a tomographic image (medical image) of the inside of the blood vessels based on the signal of the reflected waves received by the catheter 21. The image processing device 22 is a processing device that processes data of the reflected waves received by the catheter 21 to generate a tomographic image, displays the generated tomographic image on the display device 23, and includes an input interface for receiving input of various setting values ​​when performing an examination.

[0012] In this embodiment, an IVUS device is given as an example of the imaging diagnostic device 2, and an ultrasound tomographic image is generated as the medical image, but the imaging diagnostic device 2 may also generate an optical coherence tomographic image (OCT image). The imaging diagnostic device 2 is not limited to an IVUS device, and may be, for example, an angiography device, a CT (Computed Tomography) device, an MRI (Magnetic Resonance Imaging) device, etc. In other words, the medical image is not limited to an ultrasound tomographic image, and may include an optical coherence tomographic image, an X-ray fluoroscopic image (an angiography image, a CT (Computed Tomography) image, etc.), a magnetic resonance imaging (MRI) image, etc.

[0013] The information processing device 1 is an information processing device capable of various information processing and transmitting and receiving information, such as a server computer or a personal computer. In this embodiment, the information processing device 1 is assumed to be a server computer, and for simplicity, will be referred to as server 1 below. Note that the server 1 may be a local server installed in the same facility (hospital, etc.) as the imaging diagnostic device 2, or may be a cloud server communicatively connected to the imaging diagnostic device 2 via the Internet, etc. The server 1 functions as a search device that searches for medical information of a first patient similar to medical information of a second patient who is the target of treatment by the imaging diagnostic device 2, and outputs the search results to the imaging diagnostic device 2.

[0014] Specifically, as described below, the server 1 performs machine learning in advance to learn predetermined training data, and prepares a learning model 50 that receives medical information including medical images as input and outputs support information for supporting patient treatment (see FIG. 4). The support information is information for supporting vascular treatment such as catheter surgery (e.g., PCI (Percutaneous Coronary Intervention)), and is, for example, a detection result indicating a lesion (e.g., plaque) in a blood vessel that appears in a medical image. The server 1 inputs the medical information of a second patient into the learning model 50, extracts features of the medical information, and detects a lesion in the blood vessel of the second patient based on the extracted features.

[0015] In this embodiment, the server 1 searches the database for medical information of a first patient that is similar to the medical information of a second patient as a similar case, based on the features extracted by the learning model 50. The server 1 outputs the searched medical information of the first patient to the image diagnostic device 2 and presents it to the user (medical worker).

[0016] 2 is a block diagram showing an example of the configuration of the server 1. The server 1 includes a control unit 11, a main memory unit 12, a communication unit 13, and an auxiliary memory unit . The control unit 11 has one or more arithmetic processing devices such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit), etc., and performs various information processing, control processing, etc. by reading and executing a program P stored in the auxiliary storage unit 14. The main storage unit 12 is a temporary storage area such as an SRAM (Static Random Access Memory), a DRAM (Dynamic Random Access Memory), or a flash memory, and temporarily stores data necessary for the control unit 11 to execute arithmetic processing. The communication unit 13 is a communication module for performing communication-related processing, and transmits and receives information to and from the outside.

[0017] The auxiliary storage unit 14 is a non-volatile storage area such as a large-capacity memory or a hard disk, and stores the program P and other data required for the control unit 11 to execute processing. The auxiliary storage unit 14 also stores a learning model 50 and a case DB 141. The learning model 50 is a machine learning model that has learned training data as described above, and is a model that inputs medical information and outputs support information for medical support. The learning model 50 is expected to be used as a program module that constitutes artificial intelligence software. The case DB 141 is a database that stores patient cases, and stores the medical information of a first patient in association with the feature quantities of the medical information.

[0018] The auxiliary storage unit 14 may be an external storage device connected to the server 1. The server 1 may be a multi-computer consisting of multiple computers, or may be a virtual machine virtually constructed by software.

[0019] Furthermore, in this embodiment, the server 1 is not limited to the above configuration, and may include, for example, an input unit that accepts operation input, a display unit that displays images, etc. The server 1 may also include a reading unit that reads a portable storage medium 1a such as a CD (Compact Disk), a DVD (Digital Versatile Disc), or a USB (Universal Serial Bus) memory, and may read and execute the program P from the portable storage medium 1a. Alternatively, the server 1 may read the program P from a semiconductor memory 1b.

[0020] 3 is an explanatory diagram showing an example of a record layout of the case DB 141. The case DB 141 includes a case ID column, a patient information column, a treatment information column, an examination information column, an image information column, and a feature column. The case ID column stores a case ID for identifying the medical information of each first patient, which is a case. The patient information column, the treatment information column, the examination information column, the image information column, and the feature column store the patient information, treatment information, examination information, medical images, and features of the medical information of the first patient, respectively, in association with the case ID.

[0021] The patient information is basic information about the first patient who received treatment and is the medical record of the first patient. The patient information includes, for example, the patient's age, sex, diagnosis, risk factors (presence or absence of lifestyle-related diseases, etc.), medical history, medication history, etc.

[0022] The treatment information is information indicating the details of the treatment performed on the first patient, and is a record of vascular treatment such as PCI. The treatment information includes, for example, the date of treatment, the location of the treated lesion (hereinafter referred to as the "lesion site"), the characteristics of the lesion, the puncture site of the catheter 21, the amount of contrast agent administered, the time taken to capture the fluoroscopic image, the presence or absence and details of additional procedures before stent placement, the presence or absence and details of additional procedures after stent placement, the name, diameter, length, total number, total stent length of the placed stents, the maximum balloon inflation diameter, the maximum inflation pressure, the treatment method for the bifurcation lesion, and post-treatment progress information (for example, the presence or absence of complications). Note that, although the stent diameter is expressed as a diameter in this embodiment, it may also be expressed as a radius.

[0023] The examination information is a record of examinations of the first patient other than diagnostic imaging, and includes, for example, blood test results, the number of affected blood vessels, the left ventricular ejection fraction, and a history of cardiovascular emergencies (such as myocardial infarction).

[0024] Medical images are images of a patient's blood vessels, and as described above, include intravascular tomographic images (ultrasound tomographic images, optical coherence tomographic images), X-ray fluoroscopic images (angiographic images), CT images, MRI images, etc. In this embodiment, an intravascular tomographic image (ultrasound tomographic image) acquired by the image diagnostic device 2 will be mainly described as an example of a medical image.

[0025] Fig. 4 is an explanatory diagram showing an overview of the first embodiment. Fig. 4 conceptually illustrates how medical information of a first patient similar to medical information of a second patient is searched for using a learning model 50. The overview of this embodiment will be described based on Fig. 4.

[0026] The learning model 50 is a machine learning model that takes a patient's medical information as input and outputs support information for assisting the patient's vascular treatment. For example, the learning model 50 is a neural network model generated by deep learning, and is a convolutional neural network (CNN) that extracts features of input data using multiple convolution layers. The learning model 50 includes an intermediate layer (hidden layer) in which convolution layers that convolve the input data and pooling layers that map the convolved data are alternately connected, and extracts features of the input data.

[0027] In this embodiment, the learning model 50 is described as a CNN, but it may also be another neural network model such as an RNN (Recurrent Neural Network), or a model based on another learning algorithm such as a GAN (Generative Adversarial Network), an SVM (Support Vector Machine), or a decision tree.

[0028] In this embodiment, when the server 1 receives an input of a medical image (tomogram), which is one of the medical information, the server 1 generates a learning model 50 that detects a lesion in the medical image. For example, the server 1 generates a semantic segmentation model (such as U-net) or a Faster R-CNN (Region CNN) as the learning model 50.

[0029] Semantic segmentation is a type of CNN, a type of Encoder / Decoder model that generates output data from input data. In addition to a convolutional layer that compresses input image data, a semantic segmentation model also has a deconvolutional layer that maps (expands) the features obtained through compression back to the original image size. The deconvolutional layer identifies which objects exist in which positions in the image on a pixel-by-pixel basis based on the features extracted in the convolutional layer, and generates a binarized label image indicating which object each pixel corresponds to.

[0030] Faster R-CNN is a CNN primarily used for object detection, and in addition to the intermediate layer that extracts the features of the input image, it also includes an RPN (Region Proposal Network) that estimates the image region where an object may exist. Faster R-CNN inputs the features extracted from the image and the coordinate range of the image region estimated by the RPN into the output layer, ultimately detecting the object in the input image.

[0031] The server 1 generates these models as a learning model 50 and uses them to detect lesions. Note that all of the above models are merely examples, and the learning model 50 may be any model that can identify the position and type of lesions in medical images. In this embodiment, as an example, the learning model 50 will be described as a semantic segmentation model.

[0032] In Figure 4, the lesions contained in the medical image are conceptually illustrated by hatching on the right side of the learning model 50. The learning model 50 receives input of intravascular tomographic images (medical images) acquired by the image diagnostic device 2, and detects image regions corresponding to lesions such as plaque. Note that plaque is an example of a lesion, and other parts such as calcified tissue, dissections (flaps) in the blood vessel wall, neointima, and the like may also be detected.

[0033] The server 1 performs learning using training data in which image regions corresponding to lesions are labeled for training medical images. Specifically, in the training data, labels (metadata) indicating the coordinate ranges corresponding to the lesions and the types of the lesions are assigned to the training medical images.

[0034] The training data is not limited to the data of the first patient stored in the case DB 141, but may be data of other patients. Furthermore, the training data is not limited to the data of actual patients, but may be virtual data that has been inflated using a data generation means such as GAN.

[0035] The server 1 inputs training tomographic images into the learning model 50 and obtains the detection results of the lesion as an output. Specifically, the server 1 obtains as an output a labeled image in which each pixel in the image area corresponding to the lesion is labeled with a value indicating the type of the lesion.

[0036] The server 1 compares the detection results output from the learning model 50 with the coordinate range of the correct image area indicated by the training data and the type of lesion, and optimizes parameters such as the weights between neurons so that the two are similar. In this way, the server 1 generates the learning model 50.

[0037] In addition to medical images, the server 1 may also use text data such as patient information and examination information as input to the learning model 50. In this case, for example, the server 1 converts the text data using a method such as one-hot encoding, and inputs the converted text data to the learning model 50 as a categorical variable representing the category of the medical image. This allows lesion detection to be performed using medical information other than images.

[0038] It is not essential that the input to the learning model 50 includes text data such as patient information and examination information, and the medical information input to the learning model 50 may consist only of medical images.

[0039] Furthermore, although the above description has been given assuming that the learning model 50 detects a lesion, the present embodiment is not limited to this. For example, the learning model 50 may be a model that takes medical information as input and predicts a correct treatment method (e.g., the length, diameter, or balloon expansion diameter of a stent to be used in vascular treatment). Alternatively, the learning model 50 may be a model that takes medical information as input and predicts post-treatment progress information (e.g., the probability of complications occurring). In this way, the learning model 50 may be any model that takes medical information as input and outputs support information for supporting treatment, and the content of the support information is not particularly limited.

[0040] In this embodiment, the server 1 extracts features of the medical information of the second patient using the above-described learning model 50, and uses the extracted features to search for the medical information of the first patient. Specifically, the server 1 uses the features of a tomographic image in which a lesion is detected, among multiple tomographic images of the inside of the blood vessels of the second patient, to search for the medical information of the first patient.

[0041] For example, when treating a second patient, the server 1 acquires intravascular tomographic images of the second patient from the imaging diagnostic device 2. Specifically, the server 1 acquires a moving image consisting of a plurality of frame images captured along the longitudinal direction of the blood vessel in accordance with the scanning of the catheter 21 from the imaging diagnostic device 2. The server 1 sequentially inputs the tomographic images corresponding to each frame into the learning model 50 and detects the lesion.

[0042] The timing for performing the series of processes is not limited to when treatment is performed, and the recorded video may be input to the learning model 50 afterwards to detect the lesion.

[0043] When a lesion is detected from the tomographic image, the server 1 identifies the feature extracted from the tomographic image in which the lesion is detected as the feature of the medical information of the second patient. The server 1 searches for medical information of the first patient that is similar to the medical information of the second patient based on the identified feature and the feature of the medical information of each first patient stored in the case DB 141.

[0044] Specifically, the server 1 calculates the similarity based on the feature quantities of both the documents and performs a search based on the calculated similarity. There are no particular limitations on the method for calculating the similarity, but for example, the server 1 calculates the cosine similarity of the feature quantities expressed as vectors.

[0045] For example, the server 1 determines whether the degree of similarity is equal to or greater than a predetermined threshold, and searches for the medical information of the first patient determined to be equal to or greater than the threshold as a similar case. The server 1 determines a search order for the medical information of each first patient according to the degree of similarity, and outputs the medical information of each first patient to the display device 23 according to the search order.

[0046] When determining the search ranking for the first patient, it is preferable that the server 1 change the search ranking according to the progress information of the first patient after treatment included in the medical information. The progress information is data indicating the condition of the patient after treatment, such as data indicating whether or not a complication has occurred. For example, if the searched first patient has developed a complication, the server 1 lowers the search ranking. It is also possible to change the search ranking by referring not only to whether or not a complication has occurred, but also to the type of complication that has occurred.

[0047] In the above, the search ranking is changed according to the progress information, but the present embodiment is not limited to this. For example, even if a complication has occurred, the server 1 may not change the search ranking, but may simply indicate that a complication has occurred on a display screen described below. This allows the user to decide whether or not to refer to the medical information of the first patient searched as a similar case.

[0048] Fig. 5 is an explanatory diagram showing an example of a display screen for similar cases. Fig. 5 shows an example of a screen displaying medical information for a first patient found as a similar case. For example, the display device 23 displays a subject image 501, a case image 502, a case information field 503, a list of similar cases 504, a field for specifying an examiner 505, and a summary field 506.

[0049] The subject image 501 is a medical image of a second patient, and is an ultrasound tomographic image of the patient currently undergoing an ultrasound examination. The case image 502 is a medical image of a first patient found to be a similar case, and is an ultrasound tomographic image of the blood vessels of the first patient that has been determined to be similar to the tomographic image of the second patient. The case information field 503 is a display field that displays medical information of the first patient other than the medical image displayed as the case image 502, such as text data of patient information, examination information, treatment information, etc. The treatment information includes not only the surgical record of the treatment of the first patient but also progress information of the first patient after treatment (e.g., complications). The server 1 reads out the medical image and patient information of the first patient from the case DB 141 and displays them on the display device 23.

[0050] In the example screen of FIG. 5, only ultrasonic tomographic images are displayed as the subject image 501 and the case image 502, but it goes without saying that other medical images such as X-ray fluoroscopic images may also be displayed.

[0051] The similar case list 504 is a display field showing the search results for similar cases in a list, and is a display field that displays a summary of the medical information of each first patient found as a similar case and the degree of similarity with the medical information of each first patient. For example, the display device 23 displays thumbnail images of reduced medical images of each first patient and the degree of similarity in order of search ranking. The display device 23 accepts a selection input for selecting one of the thumbnail images from the similar case list 504. The display device 23 displays the medical image of the first patient corresponding to the selected case as a case image 502, and also displays patient information, etc. of the first patient in the case information field 503.

[0052] When displaying a list of medical information of each first patient in the similar case list 504, the display device 23 may change the display according to the progress information (whether or not a complication has occurred). For example, if the first patient has developed a complication, the display device 23 displays a predetermined icon indicating the occurrence of the complication in association with the thumbnail image. This can improve user convenience.

[0053] The practitioner designation field 505 is an input field for inputting search conditions for similar cases, and is an input field for accepting input of designation of the treatment provider (e.g., doctor) of the first patient to be searched. For example, the display device 23 accepts input of designation of the name of the practitioner in the practitioner designation field 505. When input of designation of the practitioner is accepted in the practitioner designation field 505, the server 1 narrows down the search to cases of the designated practitioner, i.e., the medical information of the first patient treated by the practitioner.

[0054] In this embodiment, the practitioner can be specified as a search condition, but other search conditions such as the patient's age, diagnosis, medical history, etc. may also be specified in addition to the practitioner.

[0055] The tally column 506 is a display column that displays the tally results obtained by tallying predetermined information items for the medical information of multiple first patients searched for as similar cases. There are no particular limitations on the tally items, but for example, the server 1 tally up the treatment methods of multiple first patients searched for as similar cases and output the tally results to the display device 23. Specifically, the server 1 tally up the number of cases by treatment method, such as catheter surgery, bypass surgery, or drug therapy, and displays the tally results.

[0056] 6 is a flowchart showing the procedure of the process of generating the learning model 50. The process of generating the learning model 50 from training data will be described with reference to FIG. The control unit 11 of the server 1 acquires training data in which correct values ​​of support information are assigned to the training medical information (step S11). For example, the control unit 11 acquires training data in which image regions corresponding to lesions in the medical images and label data indicating the types of lesions are assigned to the training medical images. As described above, the training input data may include text data such as patient information, and a learning model 50 may be generated that receives medical images and text data as inputs and outputs support information.

[0057] Based on the training data, the control unit 11 generates a learning model 50 that outputs support information for supporting patient treatment when medical information is input (step S12). Specifically, as described above, the control unit 11 generates a neural network such as a CNN as the learning model 50. The control unit 11 inputs training medical images into the learning model 50 and obtains the detection results of lesions in the medical images as output. The control unit 11 compares the detection results of the lesions with correct label data, and optimizes parameters such as the weights between neurons so that the two are similar to each other, thereby generating the learning model 50. The control unit 11 ends the series of processes.

[0058] 7 is a flowchart showing the procedure for the process of searching for similar cases. The process of searching for medical information on a first patient who is a similar case will be described with reference to FIG. The control unit 11 of the server 1 receives input specifying the provider who performed the treatment of the first patient for the medical information of the first patient to be searched (step S31). If the provider is not specified, the control unit 11 skips step S31.

[0059] The control unit 11 acquires medical information of the second patient to be treated (step S32). As described above, the medical information may include text data such as patient information and examination information in addition to the medical images acquired from the image diagnostic apparatus 2.

[0060] The control unit 11 inputs the acquired medical information of the second patient into the learning model 50, extracts features, and outputs support information based on the extracted features (step S33). Specifically, the control unit 11 sequentially inputs multiple tomographic images captured consecutively along the longitudinal direction of the blood vessel into the learning model 50, and detects the lesion from each tomographic image.

[0061] The control unit 11 calculates the similarity between the medical information of the first patient and the second patient based on the feature amount of the tomographic image in which the lesion was detected and the feature amount of the medical information of each first patient stored in the case DB 141 (step S34). The control unit 11 searches for medical information of the first patient that is similar to the medical information of the second patient based on the calculated similarity (step S35). Note that if a designated practitioner input has been received in step S31, the control unit 11 narrows the search to first patients who were treated by the designated practitioner.

[0062] The control unit 11 changes the search ranking of the medical information of each retrieved first patient according to the progress information of the first patient after treatment (step S36). For example, if a complication occurs in the first patient after treatment, the control unit 11 lowers the search ranking of the first patient.

[0063] The control unit 11 also tally up the medical information of the retrieved multiple first patients (step S37). For example, the control unit 11 tally up the number of cases of catheter surgery, bypass surgery, and drug therapy for the first patient's treatment methods. The control unit 11 outputs the medical information of the first patient retrieved from the case DB 141 to the imaging diagnostic apparatus 2 (step S38), and ends the series of processes.

[0064] As described above, according to the first embodiment, it is possible to suitably search for similar cases that are similar to the second patient to be treated.

[0065] Furthermore, according to the first embodiment, by presenting the similarity together with the medical information of the first patient, the user can easily understand similar cases.

[0066] Furthermore, according to the first embodiment, by presenting not only the records of the first patient's treatment but also progress information after treatment as the medical information of the first patient, it is possible to judge the validity of the first patient's treatment records.

[0067] Furthermore, according to the first embodiment, by changing the search order of the medical information of the first patients depending on the progress information, it is possible to present the medical information of each first patient in a more appropriate order.

[0068] Furthermore, according to the first embodiment, by using the learning model 50 that receives medical information as input and outputs support information, it is possible to suitably extract the feature amount of the medical information of the second patient.

[0069] Furthermore, according to the first embodiment, by using a model for detecting a lesion as the learning model 50, it is possible to detect a lesion in the second patient while simultaneously searching for similar cases.

[0070] Furthermore, according to the first embodiment, functions for specifying practitioners and aggregating treatment methods (medical information) are also provided, which can improve user convenience.

[0071] (Embodiment 2) In this embodiment, features are extracted from image data (medical images) and text data included in medical information, and the extracted features are used for case search. Note that the same reference numerals are used to designate the same parts as in the first embodiment, and the description thereof will be omitted.

[0072] Fig. 8 is an explanatory diagram of a learning model 50 according to the second embodiment. Fig. 8 illustrates a so-called multimodal model that includes a network (first extractor 51) for extracting features of image data and a network (second extractor 52) for extracting features of text data. An outline of this embodiment will be described with reference to Fig. 8.

[0073] The learning model 50 according to this embodiment includes a first extractor 51, a second extractor 52, and a detector 53. The first extractor 51 is an extractor that extracts features of image data, and is, for example, a CNN network. The second extractor 52 is an extractor that extracts features of text data such as patient information and examination information, and is, for example, an RNN network. The first extractor 51 receives input of a medical image of a patient and extracts features of the medical image. The second extractor 52 receives input of text data such as patient information and extracts features of the text data.

[0074] The detector 53 is a network that detects lesions using the feature amounts extracted by the first extractor 51 and the second extractor 52 as input. For example, the detector 53 includes a combination layer that combines the feature amounts of the medical image and the text data, and a decoder (deconvolution layer) that generates a labeled image by labeling an image region corresponding to the lesion based on the combined feature amounts. The detector 53 combines the feature amounts of the medical image and the text data, inputs them to the decoder, and outputs a labeled image that is the detection result of the lesion.

[0075] When a lesion is detected, the server 1 searches the case DB 141 for medical information of a first patient, which is a similar case, based on the feature amounts extracted by the first extractor 51 and the second extractor 52 at the time of detection. For example, the server 1 calculates the similarity individually based on the feature amounts of the medical images and the text data. That is, the server 1 calculates the similarity of the medical images based on the feature amounts of the medical images of the first patient and the medical images of the second patient, and calculates the similarity of the text data (patient information, etc.) based on the feature amounts of the text data of the first patient and the text data of the second patient.

[0076] The server 1 calculates a total similarity for comprehensively evaluating the similarity between the first patient and the second patient from the similarities between the medical images and the text data. For example, the server 1 calculates the total similarity by calculating the sum of the similarities between the medical images and the text data.

[0077] For example, the server 1 may use the average value of both data as the total similarity. Also, for example, the server 1 may weight one of the data (e.g., medical images) to calculate the total similarity.

[0078] The server 1 determines whether the calculated total similarity is equal to or greater than a threshold, and searches for the medical information of the first patient determined to be equal to or greater than the threshold as a similar case. The server 1 then outputs the searched medical information of the first patient to the display device 23. In this case, the server 1 may display not only the total similarity but also the similarity of each of the medical image and text data on the display device 23. This allows the user to understand whether the images or the text data are similar.

[0079] 9 is a flowchart showing the procedure of the process of generating the learning model 50 according to embodiment 2. After acquiring training data for generating the learning model 50 (step S11), the server 1 executes the following process. The control unit 11 of the server 1 generates a learning model 50 based on the training data (step S201). In this embodiment, as described above, a multimodal model is generated that extracts features from both medical images and text data. The control unit 11 inputs training medical images to a first extractor 51 to extract features, inputs training text data to a second extractor 52 to extract features, and inputs both features to a detector 53 to detect lesions. The control unit 11 compares the lesion detection results with the correct label data, and generates the learning model 50 by optimizing parameters such as the weights between neurons so that the two are similar. The control unit 11 then completes the series of processes.

[0080] 10 is a flowchart showing the procedure of the search process for similar cases according to Embodiment 2. After acquiring the medical information of the second patient (step S32), the server 1 executes the following process. The control unit 11 of the server 1 inputs the medical images included in the medical information of the second patient to the first extractor 51 and the text data other than the medical images to the second extractor 52, extracts feature amounts from each of the medical images and the text data, and detects the lesion (step S221). Specifically, as described above, the control unit 11 inputs both feature amounts to the detector 53, combines the feature amounts, and detects the lesion based on the combined feature amount.

[0081] If a lesion is detected, the control unit 11 calculates the similarity between the medical information of each first patient based on the feature amount of the medical image extracted by the first extractor 51 and the feature amount of the text data extracted by the second extractor 52 (step S222). Specifically, the control unit 11 calculates the similarity between the first patient and the second patient for each of the medical image and the text data, and calculates the total similarity by, for example, summing the similarities of the medical image and the text data. The control unit 11 searches the case DB 141 for the medical information of the first patient based on the calculated similarity (step S223), and proceeds to step S36.

[0082] Although a multimodal model has been given above as an example of the learning model 50, for example, a model that outputs support information from medical images and a model that outputs support information from text data may be prepared independently, and each model may be used to extract feature quantities from the medical images and text data, respectively. In other words, the learning model 50 is not limited to a single model, but may be multiple models.

[0083] Furthermore, although the above description has been given assuming that only one type of image is input to the learning model 50, multiple types of medical images may be input to the learning model 50 to extract features and use them to search for similar cases. For example, the server 1 inputs X-ray fluoroscopic images in addition to ultrasound tomographic images to the learning model 50 and extracts features from each image. In this case, the learning model 50 may be provided with separate extractors for extracting features from ultrasound tomographic images and X-ray fluoroscopic images, and these may be connected in parallel to the detector 53. In this way, multiple types of medical images may be input to the learning model 50.

[0084] As described above, according to the second embodiment, by extracting the features of the medical images and text data contained in the medical information separately, the similarity between the medical images and the text data can be evaluated separately, and similar cases can be searched for more effectively.

[0085] The embodiments disclosed herein are illustrative in all respects and should not be considered limiting. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0086] 1. Server (information processing device) 11 Control section 12 Main memory 13 Communications Department 14 Auxiliary storage 141 Case DB 1a Portable storage media 1b semiconductor memory 2. Diagnostic imaging equipment 21 Catheter 22 Image processing device 23 Display device 50 Learning Models 501 Subject images 502 Case Images 503 Case Information Column 504 List of similar cases 505 Implementer designation field 506 Summary column 51 1st extractor 52 Second extractor 53 Detector N Network P Program

Claims

1. a storage unit that stores medical information of a first patient who has already been treated, the medical information including progress information indicating whether or not complications have occurred after the treatment of the first patient, in association with a feature amount of the medical information; an acquisition unit that acquires medical information of a second patient to be treated; an extraction unit that extracts features of the medical information of the second patient; a calculation unit that calculates a similarity between the medical information of the first patient and the medical information of the second patient based on a feature amount of the medical information of the first patient and a feature amount of the medical information of the second patient; a search unit that searches for a plurality of pieces of medical information of the first patient that are similar to the medical information of the second patient based on the calculated similarity; an output unit that outputs the searched medical information of the plurality of first patients in accordance with a search order determined according to the similarity; The search unit lowers the search ranking of medical information of the first patient when a complication occurs in the searched first patient.

1. An information processing device comprising:

2. The output unit outputs the degree of similarity between the searched medical information of the first patient and the medical information of the second patient.

2. The information processing apparatus according to claim 1, wherein:

3. The extraction unit extracts the feature amounts of the medical information of the second patient using a trained model that extracts feature amounts of the medical information when the medical information is input and outputs support information for medical support based on the extracted feature amounts.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

4. the model is a model that, when a medical image of a patient's hollow organ is input, extracts feature amounts from the medical image, detects a lesion in the medical image based on the extracted feature amounts, and outputs a detection result; The extraction unit inputting the medical image of the second patient into the model to detect the lesion; extracting a feature of the medical image in which the lesion is detected as a feature of the medical information of the second patient; The information processing device according to claim 3 .

5. the medical image is a cross-sectional image of a patient's blood vessel; The extraction unit inputs a plurality of the tomographic images taken along a longitudinal direction of the blood vessel of the second patient into the model, and detects the lesion area from any of the plurality of tomographic images.

5. The information processing apparatus according to claim 4,

6. the medical information of the first patient and the second patient includes medical images and text data; the model includes a first extractor that extracts a feature amount of the medical image and a second extractor that extracts a feature amount of the text data; The search unit searches for medical information of the first patient that is similar to medical information of the second patient based on feature amounts of the medical image and text data.

6. The information processing device according to claim 3, wherein the information processing device is a computer.

7. The medical image includes at least one of an ultrasonic tomographic image, an optical coherence tomographic image, an X-ray fluoroscopic image, a CT image, and an MRI image of the blood vessel of the first patient.

7. The information processing device according to claim 4, wherein:

8. a reception unit that receives an input of a designated practitioner who has performed medical care on the first patient; The search unit searches for medical information of the first patient corresponding to the specified practitioner, The medical information of the first patient includes information of the practitioner.

8. The information processing device according to claim 1, wherein the information processing device is a computer.

9. a counting unit that counts the medical information of the searched first patients, the search unit searches for a plurality of pieces of medical information of the first patient that are similar to the medical information of the second patient; The output unit outputs a compilation result of the medical information of the first patient.

9. The information processing device according to claim 1, wherein the information processing device is a computer.

10. The medical information of the first patient is information about an endovascular treatment performed on the first patient.

10. The information processing device according to claim 1, wherein the information processing device is a computer.

11. Obtain medical information of a second patient to be treated; extracting features of medical information of the second patient; calculating a similarity between the medical information of the first patient and the second patient based on a feature amount of the medical information of the first patient who has already been treated and a feature amount of the medical information of the second patient; based on the calculated similarity, searching for a plurality of pieces of medical information of the first patient that are similar to the medical information of the second patient from a storage unit that stores medical information including progress information indicating whether or not complications have occurred after treatment of the first patient in association with feature amounts of the medical information; outputting the searched medical information of the plurality of first patients in accordance with a search order determined according to the similarity; If the searched first patient has a complication, the search ranking of the medical information of the first patient is lowered. An information processing method characterized in that the processing is executed by a computer.

12. Obtain medical information of a second patient to be treated; extracting features of medical information of the second patient; calculating a similarity between the medical information of the first patient and the second patient based on a feature amount of the medical information of the first patient who has already been treated and a feature amount of the medical information of the second patient; based on the calculated similarity, searching for a plurality of pieces of medical information of the first patient that are similar to the medical information of the second patient from a storage unit that stores medical information including progress information indicating whether or not complications have occurred after treatment of the first patient in association with feature amounts of the medical information; outputting the searched medical information of the plurality of first patients in accordance with a search order determined according to the similarity; If the searched first patient has a complication, the search ranking of the medical information of the first patient is lowered. A program that causes a computer to execute a process.

Images