Medical information processing device, medical information processing method, and program

Abstract

To reduce the burden on users in preparing training data. [Solution] The medical information processing device according to the embodiment includes: a clustering unit that clusters a plurality of labels, each assigned to a plurality of medical images, based on the similarity of the labels, which are labels indicating analysis results based on medical images; an acquisition unit that receives a modification to at least one target label among the plurality of labels clustered by the clustering unit and acquires a relationship between the target label and a modified target label in which the modification is reflected for the target label; and a modification unit that modifies a label assigned to a second medical image different from the first medical image corresponding to the target label, based on the relationship.

Description

Technical Field

[0001] The embodiments disclosed in this specification and the drawings relate to a medical information processing apparatus, a medical information processing method, and a program.

Background Art

[0002] In order to perform analysis of medical information by AI (Artificial Intelligence), first, medical information to be used as learning data is collected, and the machine learning model is trained by inputting the medical information. The trained model is used in medical facilities such as hospitals and clinics. Specifically, the trained model supports the formulation of various diagnoses and treatment plans by receiving the input of medical information collected in medical facilities such as hospitals and outputting an analysis result.

[0003] Here, due to various factors, the data has a tendency bias, and the tendency of the medical information used for learning data does not always match the medical information input to the model during use in medical facilities. As a result, there are cases where the model cannot exhibit sufficient performance depending on the medical facility.

[0004] As a measure to improve the performance of the model, it is conceivable to perform additional learning of the model in the medical facility. However, preparing learning data for such additional learning is laborious and burdensome for users such as doctors.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

[0006] One of the problems that the embodiments disclosed in this specification and drawings aim to solve is to reduce the burden on the user in preparing training data. However, the problems that the embodiments disclosed in this specification and drawings aim to solve are not limited to the above problem. Problems corresponding to the effects of each configuration shown in the embodiments described later can also be positioned as other problems. [Means for solving the problem]

[0007] The medical information processing device according to this embodiment includes: a clustering unit that clusters a plurality of labels, each assigned to a plurality of medical images, based on the similarity of the labels, which are labels indicating analysis results based on medical images; an acquisition unit that receives a modification to at least one target label among the plurality of labels clustered by the clustering unit and acquires a relationship between the target label and a modified target label in which the modification is reflected for the target label; and a modification unit that modifies a label assigned to a second medical image different from the first medical image corresponding to the target label, based on the relationship. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is a block diagram showing an example of a medical information processing system according to the first embodiment. [Figure 2] Figure 2 shows an overview of the training and use of the model according to the first embodiment. [Figure 3] Figure 3 shows an overview of the additional learning process according to the first embodiment. [Figure 4] Figure 4 shows an example of a label generation method according to the first embodiment. [Figure 5]FIG. 5 is a diagram showing the relationship between the number of additional learning data and the performance of the model according to the first embodiment. [Figure 6A] FIG. 6A is a diagram showing an example of active learning according to the first embodiment. [Figure 6B] FIG. 6B is a diagram showing an example of required active learning according to the first embodiment. [Figure 7] FIG. 7 is a diagram showing an overview of the processing of the medical information processing device according to the first embodiment. [Figure 8] FIG. 8 is a diagram showing an example of clustering according to the first embodiment. [Figure 9A] FIG. 9A is a diagram for explaining the label modification reception process according to the first embodiment. [Figure 9B] FIG. 9B is a diagram for explaining the label modification process according to the first embodiment. [Figure 9C] FIG. 9C is a diagram for explaining the label preprocessing according to the first embodiment. [Figure 10] FIG. 10 is a flowchart showing a series of processes of the processing circuit according to the first embodiment. [Figure 11] FIG. 11 is a diagram for explaining a method of selecting a target label according to the second embodiment. [Figure 12] FIG. 12 is a diagram showing an example of similarity calculation processing according to the second embodiment. [Figure 13] FIG. 13 is a flowchart showing a series of processes of the processing circuit according to the second embodiment. [Figure 14] FIG. 14 is a diagram showing an example of the shape of a target site according to another embodiment.

MODE FOR CARRYING OUT THE INVENTION

[0009] Hereinafter, embodiments of a medical information processing device, a medical information processing method, and a program will be described in detail with reference to the accompanying drawings.

[0010] (First Embodiment) In the first embodiment, the medical information processing system 1 in FIG. 1 will be described as an example. The medical information processing system 1 includes a medical image diagnostic device 10, a medical information processing device 20, and an image storage device 30. The medical image diagnostic device 10, the medical information processing device 20, and the image storage device 30 are communicably connected via a network NW.

[0011] The medical image diagnostic device 10 is a device that collects medical images from a subject. The medical image diagnostic device 10 is also called a modality. In this embodiment, medical images will be described as an example of medical information.

[0012] The specific type of the medical image diagnostic device 10 is not limited. As an example, an X-ray diagnostic device, an X-ray computed tomography (CT) device, a magnetic resonance imaging (MRI) device, a positron emission tomography (PET) device, a SPECT (Single Photon Emission Computed Tomography) device, an ultrasonic imaging device, an optical coherence tomography (OCT) device, etc. can be mentioned.

[0013] In the embodiment, the medical image may be a 2D image or a 3D image. Also, the medical image may be a still image or a plurality of images (moving images) collected over time. For example, the medical image may be a plurality of 3D images (i.e., 4D images) collected over time. Also, the medical image may be raw data before reconstruction processing or data after reconstruction processing. In the case of an X-ray image, the data before reconstruction processing is also called projection data.

[0014] The image storage device 30 is a device that stores medical images collected by the medical imaging diagnostic device 10. Although only one medical imaging diagnostic device 10 is shown in Figure 1, the image storage device 30 may be connected to multiple medical imaging diagnostic devices 10 and store medical images collected by these multiple medical imaging diagnostic devices 10. The image storage device 30 is, for example, a PACS (Picture Archiving and Communication System) server.

[0015] The specific hardware configuration of the image storage device 30 is not limited in any way, but one example is that it can be implemented using the cloud. For example, the image storage device 30 can be realized by configuring cloud storage with any group of servers and building a database for managing medical images on the cloud storage. Of course, it is not necessary to configure the image storage device 30 using the cloud; it may also be storage installed within a specific facility.

[0016] The medical information processing device 20 supports various diagnoses and the formulation of treatment plans by inputting medical images into a model and obtaining analysis results. Furthermore, the medical information processing device 20 prepares training data for additional learning based on the medical images and performs additional training of the model. The medical information processing device 20 reduces the user's burden in preparing the training data through processes described in detail later.

[0017] For example, as shown in Figure 1, the medical information processing device 20 includes a communication interface 21, an input interface 22, a display 23, a memory 24, and a processing circuit 25.

[0018] The communication interface 21 controls the transmission and communication of various data sent and received between the medical information processing device 20 and other devices and systems connected via the network NW. Specifically, the communication interface 21 is connected to the processing circuit 25 and outputs data received from other devices and systems to the processing circuit 25, or transmits data output from the processing circuit 25 to other devices and systems. For example, the communication interface 21 can be implemented by a network card, network adapter, NIC (Network Interface Controller), etc.

[0019] The input interface 22 receives various input operations from the user of the medical information processing device 20, converts the received input operations into electrical signals, and outputs them to the processing circuit 25. For example, the input interface 22 can be implemented using a mouse, keyboard, trackball, switch, button, joystick, touchpad for input operations by touching the operating surface, touchscreen with an integrated display screen and touchpad, non-contact input circuit using an optical sensor, voice input circuit, etc. The input interface 22 may also consist of the medical information processing device 20 main unit and a tablet terminal that can communicate wirelessly. Furthermore, the input interface 22 may be a circuit that receives input operations from the user via motion capture. For example, the input interface 22 can receive the user's body movements and gaze as input operations by processing signals acquired via a tracker and images collected about the user. Moreover, the input interface 22 is not limited to those equipped with physical operating components such as a mouse or keyboard. For example, an electrical signal processing circuit that receives an electrical signal corresponding to an input operation from an external input device, which is provided separately from the medical information processing device 20, and outputs this electrical signal to the processing circuit 25, is also included as an example of an input interface 22.

[0020] The display 23 displays various types of information. For example, the display 23 displays medical images and analysis results based on medical images. Alternatively, the display 23 may display a GUI (Graphical User Interface) for receiving various instructions and settings from the user via the input interface 22. For example, the display 23 may be an LCD or a CRT (Cathode Ray Tube) display. The display 23 may be a desktop type, or it may be composed of the medical information processing device 20 and a wirelessly connected tablet terminal.

[0021] The medical information processing device 20 may also include a projector in place of or in addition to the display 23. The projector can project onto a screen, wall, floor, etc., under the control of the processing circuit 25. For example, the projector can also project onto any plane, object, space, etc., using projection mapping.

[0022] Memory 24 can be implemented using, for example, semiconductor memory elements such as RAM (Random Access Memory) or flash memory, a hard disk, or an optical disc. For example, memory 24 stores medical images and analysis results based on medical images. Memory 24 also stores models that are configured to output analysis results based on input medical images. Memory 24 also stores programs for circuits included in the medical information processing device 20 to realize their functions. Memory 24 may also be implemented using the cloud.

[0023] The processing circuit 25 includes a control function 25a, a labeling function 25b, a clustering function 25c, an acquisition function 25d, a correction function 25e, a learning function 25f, and an evaluation function 25g. The labeling function 25b is an example of a labeling unit. The clustering function 25c is an example of a clustering unit. The acquisition function 25d is an example of an acquisition unit. The correction function 25e is an example of a correction unit. The evaluation function 25g is an example of an evaluation unit.

[0024] For example, the processing circuit 25 functions as the control function 25a by reading and executing a program corresponding to the control function 25a from the memory 24. Similarly, the processing circuit 25 can function as a labeling function 25b, a clustering function 25c, an acquisition function 25d, a correction function 25e, a learning function 25f, and an evaluation function 25g. Details of the functions provided by the processing circuit 25 will be described later.

[0025] In the medical information processing device 20 shown in Figure 1, each processing function is stored in memory 24 in the form of a program that can be executed by a computer. The processing circuit 25 is a processor that realizes the function corresponding to each program by reading and executing the program from memory 24. In other words, the processing circuit 25, when a program has been read, has the function corresponding to the read program.

[0026] In Figure 1, the control function 25a, labeling function 25b, clustering function 25c, acquisition function 25d, correction function 25e, learning function 25f, and evaluation function 25g are described as being realized by a single processing circuit 25. However, the processing circuit 25 may be configured by combining multiple independent processors, and each processor may realize the functions by executing a program. Furthermore, each processing function of the processing circuit 25 may be appropriately distributed or integrated across one or more processing circuits.

[0027] Furthermore, the processing circuit 25 may also implement its functions by utilizing the processor of an external device connected via a network NW. For example, the processing circuit 25 reads and executes programs corresponding to each function from the memory 24, and implements the functions shown in Figure 1 by using cloud computing, which utilizes a group of servers connected to the medical information processing device 20 via a network NW as computing resources.

[0028] In the above explanation, the term "processor" refers to circuits such as CPUs (Central Processing Units), GPUs (Graphics Processing Units), Application Specific Integrated Circuits (ASICs), and Programmable Logic Devices (e.g., Simple Programmable Logic Devices (SPLDs), Complex Programmable Logic Devices (CPLDs), and Field Programmable Gate Arrays (FPGAs)). A processor performs its functions by reading and executing programs stored in memory.

[0029] In Figure 2, a single memory 24 was described as storing programs corresponding to each processing function. However, the embodiments are not limited to this. For example, multiple memories 24 may be distributed, and the processing circuit 25 may be configured to read the corresponding programs from individual memories 24. Alternatively, instead of storing programs in memory, the processor may be configured to directly incorporate programs into its circuitry. In this case, the processor realizes its functions by reading and executing the programs incorporated into the circuitry.

[0030] The above describes an example of the overall configuration of the medical information processing system 1. Under this configuration, the medical information processing device 20 included in the medical information processing system 1 reduces the burden on the user in preparing training data.

[0031] First, we will explain the training and use of a model as it is commonly practiced, with reference to Figure 2. Figure 2 is a diagram illustrating the overview of the training and use of a model according to the first embodiment.

[0032] First, the model is trained in the learning phase, and then in the inference phase, the trained model analyzes the medical images. The trained model is also referred to as the pre-trained model. In this embodiment, the pre-trained model is an analysis model that outputs analysis results based on the input medical images. While the machine learning algorithm of the model is not particularly limited, one example is its implementation using a neural network.

[0033] Figure 2 illustrates the training data, specifically "medical images and labels collected in Domain D1." Labels are data assigned to medical images as correct data for analysis processing based on those images. For example, when training a segmentation model, the results of segmentation processing performed on the medical images are assigned to those images as labels. The process of assigning labels is also called annotation. Domain D1, for example, is the range of data used by those who train the model (AI developers).

[0034] In the inference phase, medical images collected in Domain D2 are used as input data for the model, and the medical images are analyzed. Domain D2 is, for example, the range of data used by those who use the model (users within a medical facility). For example, Domain D2 includes medical images collected within the medical facility and medical images collected at related facilities of that medical facility.

[0035] Here, the data trends may differ between the learning phase and the inference phase due to the different domains. For example, differences in the region, time period, and methods used to collect medical images between the learning phase and the inference phase may result in different biases in the characteristics of the subjects from whom the medical images were collected (e.g., age, sex, height, weight, race, etc.).

[0036] For example, if the medical images used in the learning phase were collected in areas with high obesity rates, the model will be trained to analyze medical images collected from subjects with high obesity rates with particularly high accuracy. If such a model were to be used as is in areas with low obesity rates, its performance may be insufficient.

[0037] To ensure the model performs optimally, additional training may be performed on the model in a medical setting. This additional training is explained using Figure 3. Figure 3 shows an overview of the additional training according to the first embodiment.

[0038] The "pre-trained model" shown in Figure 3 is a model that has completed the training phase shown in Figure 2, but has not undergone additional training in a medical facility. The pre-trained model will also be referred to as the initial model.

[0039] A "further trained model" is a model after further training has been performed. As will be explained later, further training may be repeated until the required performance is achieved. Regardless of whether further training is being performed, any model that has undergone at least one round of further training will be described as a further trained model.

[0040] As explained in Figure 2, the pre-trained model corresponds to the domain during training. To improve inference performance, it is preferable to perform additional training using data corresponding to the domain during inference. For example, as shown in Figure 3, the database of medical facilities using the model contains medical images, and by assigning labels to each medical image and using them for additional training, the model can be further trained to correspond to the domain of the medical facility.

[0041] However, labeling each medical image can be a burden for the user. Ideally, the labels assigned to each medical image should be used as ground truth data, and higher accuracy is preferable. For example, when performing additional training on a segmentation model, it is preferable to use labels that accurately segment the target area from the medical image.

[0042] Highly accurate segmentation results can be generated by users, such as physicians, performing segmentation manually. However, manual segmentation is burdensome for users, and it is difficult to collect a sufficient number of labels.

[0043] Therefore, as shown in Figure 4, it is conceivable to reduce the user's burden by using a pre-trained model. Specifically, this method involves first obtaining labels inferred from medical images using a pre-trained model, having the user review the inferred labels and make corrections if there are any shortcomings, and then using the corrected labels for further training. Figure 4 shows an example of a label generation method according to the first embodiment. Although the method shown in Figure 4 alleviates the burden compared to manually generating all labels, it can still be said that the user's burden is still significant.

[0044] Generally, the more training data a model has, the better its performance. That is, generating more labels and increasing the amount of data available for further training improves the performance of the newly trained model. However, the user's workload increases with the number of labels generated. Therefore, it is preferable to generate an appropriate number of labels, considering the balance between the performance of the newly trained model and the user's workload.

[0045] Ideally, as shown in Figure 5, it is preferable to increase the number of data points for additional training until the required performance is reached at "N1". That is, if "N1" is known, it can be determined that labeling is necessary until the number of data points for additional training reaches "N1", thereby raising the performance of the further trained model to the required level. On the other hand, after exceeding "N1", it can be determined that labeling is no longer necessary, thereby reducing the burden on the user. Figure 5 is a diagram showing the relationship between the number of data points for additional training and the performance of the model according to the first embodiment.

[0046] However, it is difficult to identify "N1" in advance. Therefore, as shown in Figure 6A, one can consider a method to minimize the number of labeling steps by sequentially increasing the amount of additional training data while evaluating the model's performance. Note that the method of repeating training while checking the model's performance after the initial training is complete is also called active learning. Figure 6A is a diagram showing an example of active learning according to the first embodiment.

[0047] Specifically, first, additional training data is added using the method shown in Figure 4 (Step S1). Next, additional training is performed based on the added data to improve the model's performance (Step S2). Then, the user checks the model's performance (Step S3), and if the required performance is not met, steps S1 to S3 are repeated.

[0048] The step in which the user verifies the model's performance may be performed in a clinical setting or using samples. For example, the user may use the newly trained model in a clinical setting to verify whether its analysis results are suitable for diagnostic purposes. Alternatively, the user may have the newly trained model analyze sample images for performance evaluation to verify whether its analysis results are suitable for diagnostic purposes.

[0049] As shown in Figure 6A, the model's performance gradually improves by repeating steps S1 to S3, eventually achieving the required performance. In other words, active learning allows for achieving the required performance and minimizing the number of labeling steps, even if "N1" cannot be determined in advance.

[0050] However, while the method shown in Figure 6A reduces the user burden for labeling, it still requires the task of verifying model performance, as shown in step S3. If the number of repetitions of steps S1 to S3 increases and the model performance needs to be verified many times, it will become a significant burden on the user.

[0051] Therefore, the medical information processing device 20 according to this embodiment, as shown in Figure 6B, increases the difference in model performance improvement due to a single additional learning session. In other words, the medical information processing device 20 improves the learning speed for each labeling session. This reduces the user burden for labeling, reduces the number of times the model performance needs to be checked, and thus reduces the overall burden on the user.

[0052] The processing performed by the medical information processing device 20 will be explained using Figure 7. Figure 7 is a diagram showing an overview of the processing performed by the medical information processing device 20 according to the first embodiment. First, the medical information processing device 20 performs model inference on the medical image, similar to the case shown in Figure 4. For example, the medical information processing device 20 segments the medical image to a desired region by applying a segmentation model to the medical image.

[0053] Here, the medical information processing device 20 performs inference for each of the multiple medical images collected from multiple subjects and obtains multiple inferred labels. Next, the medical information processing device 20 accepts a correction request from the user for at least one of the multiple inferred labels. Then, based on the relationship between the corrected label and its pre-correction state, the medical information processing device 20 corrects the labels assigned to medical images that are different from the medical images corresponding to the label corrected by the user.

[0054] In other words, the medical information processing device 20 accepts user modifications for some labels and automatically modifies other labels based on the user modifications. This reduces the user burden for labeling while increasing the amount of data available for additional training, thereby making the difference in performance improvement through additional training larger, as shown in Figure 6B.

[0055] The details of the processing performed by the medical information processing device 20 are described below. First, the control function 25a acquires multiple medical images collected from multiple subjects. For example, the control function 25a acquires multiple medical images stored in the image storage device 30 via the network NW. Alternatively, the control function 25a may directly acquire medical images collected by the medical image diagnostic device 10 without going through the image storage device 30.

[0056] Next, the labeling function 25b assigns a label to each of the multiple medical images. For example, the labeling function 25b assigns labels by performing inference using a model, as shown in Figure 4. That is, the labeling function 25b applies a model that is configured to output analysis results based on the input medical image to the medical image, and assigns the analysis results output from the model as a label to the medical image.

[0057] Next, the clustering function 25c clusters the multiple labels assigned to each of the multiple medical images based on the similarity of those labels. Clustering is a method of classifying data into multiple groups (also called clusters) based on similarity.

[0058] The clustering performed by the clustering function 25c will be explained using Figure 8. Figure 8 is a diagram showing an example of clustering according to the first embodiment. In Figure 8, an example is described in which a label is assigned to each of several medical images, including the medical image of subject P1, the medical image of subject P2, and the medical image of subject P3, and these multiple labels are clustered.

[0059] For example, when performing additional training on a segmentation model, label similarity refers to the similarity of the shapes of the regions segmented as target areas. In Figure 8, because the shapes of the segmented regions are similar, the labels assigned to the medical image of subject P1 and the medical image of subject P3 are classified into the same group G1. On the other hand, the labels assigned to the medical image of subject P2 are classified into a different group G2.

[0060] A typical example of similar labels is when the subjects themselves are similar. For instance, if subjects P1 and P3 have similar characteristics such as height and weight, as well as the presence and condition of diseases, then the shape and size of the target area will also be similar, and it is expected that the shapes of the regions segmented as target areas will be similar.

[0061] Furthermore, if the subjects themselves are similar, it is expected that factors such as scattered radiation affecting the image quality of X-ray CT images and the distance from the skin surface to the target area affecting the image quality of ultrasound images will also be similar. As a result, the characteristics of the medical images themselves (contrast, noise level, etc.) will also be similar, and as a result of segmentation being performed with a similar level of accuracy, it is expected that the shapes of the segmented regions will be similar.

[0062] The clustering performed by clustering function 25c may be non-hierarchical clustering or hierarchical clustering. Furthermore, overlaps in multiple groups may or may not be permitted. If overlaps are permitted, a single label may be classified into multiple groups. If overlaps are not permitted, a single label will be classified into only one group. Additionally, labels that are not similar to any other label may not be classified into any group.

[0063] The clustering function 25c may cluster based solely on label similarity, or it may also cluster based on medical image similarity. For example, the clustering function 25c calculates a first similarity score indicating label similarity. Furthermore, the clustering function 25c calculates a second similarity score indicating medical image similarity. Medical image similarity refers to, for example, the similarity of the characteristics of the medical images themselves (contrast, noise level, etc.). Alternatively, medical image similarity may refer to, for example, the similarity of the imaging conditions when the medical images were collected.

[0064] The clustering function 25c then performs clustering of labels based on the first similarity and the second similarity. For example, the clustering function 25c calculates a weighted average of the first similarity and the second similarity, and performs clustering using the calculated weighted average as the similarity.

[0065] Next, the acquisition function 25d accepts a modification to at least one of the multiple labels clustered by the clustering function 25c. Specifically, the acquisition function 25d accepts a modification to at least one of the multiple labels grouped together by the clustering function 25c. The processing by the acquisition function 25d will be explained using Figure 9A. Figure 9A is a diagram illustrating the label modification acceptance process according to the first embodiment.

[0066] First, the acquisition function 25d selects the group with the largest number of subjects as a result of clustering by the clustering function 25c. In Figure 9A, the labels assigned to the medical images "Subject P1, Subject P3, ..., Subject PX" are grouped into group G1, and it is explained that group G1 was the group with the largest number of subjects.

[0067] Next, the acquisition function 25d selects the most typical label in group G1. The most typical label is, for example, the label with the smallest distance from the group center among multiple labels grouped together. For example, the acquisition function 25d calculates the average similarity of each label in group G1 with other labels, and selects the label with the highest average value as the most typical label. Figure 9A illustrates an example in which the label assigned to the medical image of subject P3 is selected as the most typical label.

[0068] Next, the acquisition function 25d accepts user modifications to the selected label. The label that accepts modifications will also be referred to as the target label. The medical image corresponding to the target label will also be referred to as the first medical image. Furthermore, the target label after user modifications have been applied will be referred to as the modified target label.

[0069] For example, the acquisition function 25d displays the segmentation results by the model as the target label. To give one example, the acquisition function 25d superimposes on the display 23 the medical image of the subject P3 and the region segmented as the target area by the model that received the input of the medical image.

[0070] The user, referring to the display, inputs an operation to modify the region via the input interface 22. For example, as shown in Figure 9A, if a portion of the region that should be extracted as a target area is not extracted in the segmentation results by the model, the user inputs an operation to expand the region.

[0071] Here, the acquisition function 25d records the modifications received from the user by describing them in conversion f. Conversion f is an example of a first relationship that shows the relationship between the target label and the modified target label.

[0072] An example of transformation f is an affine transformation, which transforms the shape of the region before modification is accepted to the shape of the region after modification is accepted. An affine transformation is an example of shape transformation. The acquisition function 25d may calculate transformation f based on the shape of the region before modification and the shape of the region after modification, or it may record the operation entered by the user as transformation f.

[0073] Next, the modification function 25e modifies labels other than the target label for which modification has been accepted, based on the conversion f. For example, as shown in Figure 9A, the acquisition function 25d accepts modification of the label assigned to the medical image of subject P3 from among the multiple labels grouped in group G1, and also acquires the conversion f. Then, as shown in Figure 9B, the modification function 25e modifies the label assigned to the medical image of subject P1, the label assigned to the medical image of subject PX, and the other labels grouped in group G1, based on the conversion f. Figure 9B is a diagram illustrating the label modification process according to the first embodiment.

[0074] In Figure 9B, the labels assigned to each medical image represent regions segmented by the model. At this time, the modification function 25e can, for example, perform an affine transformation on the segmented regions of the medical images of subject P1 and subject PX based on the transformation f.

[0075] In Figure 9B, the medical image of subject P3 is an example of the first type of medical image. In addition, the other medical images grouped in Group G1 (excluding the medical image of subject P3, the medical image of subject PX, etc.) are examples of the second type of medical image.

[0076] As shown in Figure 9A, the acquisition function 25d accepts user modifications to the target label assigned to the medical image of subject P3 (the first medical image). The learning function 25f uses the modified target label, modified by the user, to perform additional training on the model and improve its performance.

[0077] Furthermore, as shown in Figure 9B, the correction function 25e corrects the labels assigned to second medical images, such as the medical image of subject P1 and the medical image of subject PX, based on the transformation f. The learning function 25f further trains the model using the labels corrected based on the transformation f, thereby improving the model's performance. Specifically, the learning function 25f performs further training of the model using the corrected target labels corrected by the user and other labels automatically corrected by the correction function 25e based on the relationship between the target labels and the corrected target labels.

[0078] The evaluation function 25g evaluates the performance of an additionally trained model after further training has been performed. For example, the additionally trained model is used in a clinical setting to support various diagnoses and the formulation of treatment plans. If the accuracy of the analysis results from the additionally trained model is high, those results are used as is. On the other hand, if the accuracy of the analysis results from the additionally trained model is low, the user makes corrections to those results. Therefore, the evaluation function 25g can evaluate the performance of the additionally trained model based on the frequency of corrections to the analysis results. Alternatively, the evaluation function 25g may accept evaluation input from the user regarding the performance of the additionally trained model.

[0079] As shown in Figure 9B, the medical information processing device 20 can perform additional training using the corrected target labels that have been modified by the user, as well as the labels modified based on the conversion f. This allows the medical information processing device 20 to increase the difference in performance improvement during additional training and reduce the number of times the model performance is checked, as shown in Figure 6B.

[0080] Furthermore, as shown in Figure 9A, the medical information processing device 20 selects the group with the largest number of subjects, accepts the correction for the target label and obtains the conversion f, and corrects the other labels included in the group with the largest number of subjects based on the conversion f. This increases the number of labels corrected based on the conversion f, and makes the difference in performance improvement through additional learning larger.

[0081] In Figure 9B, it is assumed that the clustered labels are sufficiently similar to each other, ensuring the reliability of the corrections when user-submitted corrections to a target label are applied to other labels. However, if clustering is performed to achieve a similarity level sufficient to satisfy this assumption, the number of labels per group will likely decrease, depending on the number of medical images available. A smaller number of labels per group would result in smaller performance improvements through additional training, potentially increasing the number of times the model performance needs to be checked, thus burdening the user.

[0082] Therefore, the correction function 25e may perform preprocessing on the labels assigned to the second medical image to improve similarity. For example, as shown in Figure 9C, the correction function 25e obtains a transformation g1 that shows the relationship between the target label assigned to the medical image of subject P3 and the label assigned to the medical image of subject P1. The correction function 25e also obtains a transformation g that shows the relationship between the target label assigned to the medical image of subject P3 and the label assigned to the medical image of subject PX. X This is obtained. Figure 9C is a diagram illustrating the label preprocessing according to the first embodiment.

[0083] Conversion g1 and conversion g X This is an example of a second relationship, showing the relationship between the target label assigned to the first medical image and the label assigned to the second medical image. Transformation g1 and transformation g X This is achieved, for example, by an affine transformation, similar to the case of transformation f. Then, the correction function 25e preprocesses the labels assigned to the second medical image based on the second relationship, and corrects the preprocessed labels based on the first relationship that shows the relationship between the target label and the corrected target label.

[0084] For example, the correction function 25e performs preprocessing on the labels assigned to the medical image of subject P1 based on transformation g1. This improves the similarity between the labels assigned to the medical image of subject P1 and the target labels. Furthermore, the correction function 25e corrects the labels that have been preprocessed based on transformation g1 based on transformation f. This series of processes using transformation g1 and transformation f can be expressed as a composite transformation in the following equation (1).

[0085]

number

[0086] Similarly, the correction function 25e converts the labels assigned to the medical images of the subject PX. X Preprocessing is performed based on this. This improves the similarity between the labels assigned to the medical images of the subject PX and the target labels. Furthermore, the correction function 25e performs conversion g X The labels, preprocessed based on [formula], are modified based on transformation f. Transformation g X And this series of operations using transformation f can be expressed as a composite transformation in the following equation (2).

[0087]

number

[0088] In the embodiment shown in Figure 9C, conversion g1 and conversion g X The preprocessing is performed to increase the similarity with the target label, and then the correction is made based on transformation f. Therefore, even if a group contains labels that do not have a very high similarity to the target label, the reliability of the correction based on transformation f is guaranteed. In other words, in the embodiment shown in Figure 9C, the similarity criterion in clustering can be relaxed while guaranteeing reliability, and the number of labels included in each group can be increased. In turn, the difference in performance improvement through additional training can be enlarged, and the burden on the user to check the model performance can be reduced.

[0089] Next, the sequence of processes of the medical information processing device 20 will be explained using the flowchart in Figure 10. Figure 10 is a flowchart of the sequence of processes of the processing circuit 25 according to the first embodiment.

[0090] Steps S101 and S108 correspond to the labeling function 25b. Step S102 corresponds to the clustering function 25c. Steps S103, S104, and S105 correspond to the acquisition function 25d. Step S106 corresponds to the correction function 25e. Step S107 corresponds to the learning function 25f. Step S109 corresponds to the evaluation function 25g.

[0091] First, the processing circuit 25 performs labeling by performing inference using an initial model (a pre-trained model) (step S101). For example, the processing circuit 25 applies a segmentation model to a medical image and assigns the obtained segmentation result to the medical image as a label. Next, the processing circuit 25 clusters the multiple labels assigned to each of the multiple medical images based on the similarity of the labels (step S102).

[0092] Next, the processing circuit 25 selects one of the multiple groups resulting from the clustering (step S103). For example, as shown in Figure 9A, the processing circuit 25 selects the group with the largest number of subjects. The processing circuit 25 also accepts a request to modify the target label assigned to the first medical image included in the selected group (step S104) and obtains the relationship between the labels before and after the modification (step S105).

[0093] Next, the processing circuit 25 modifies the labels assigned to the second medical image based on the relationships acquired in step S105 (step S106). Then, the processing circuit 25 performs additional learning using the modified target labels accepted in step S104 and the labels modified in step S106 (step S106). In other words, the processing circuit 25 performs additional learning using labels automatically modified by the device in addition to labels modified by the user. This makes it possible to increase the difference in performance improvement due to additional learning compared to the case where only labels modified by the user are used.

[0094] Next, the processing circuit 25 performs inference using the newly trained model (step S108), and determines whether the required performance has been achieved based on the results of the inference (step S109). If the required performance has not been achieved, the process returns to step S102 (step S109 negative), and if the required performance has been achieved, the process terminates (step S109 positive).

[0095] Furthermore, after the transition from step S109 to step S102, the results from step S108 may be added to the multiple labels that are to be clustered. That is, the analysis results output from the additionally trained model in step S108 can be used to determine whether or not the required performance has been achieved, and can also be attached to the medical images as labels.

[0096] Furthermore, after transitioning from step S109 to step S102, a different group may be selected when executing step S103 again. For example, after selecting the group with the largest number of subjects and executing a series of processes, the group with the second largest number of subjects may be selected when executing step S103 again.

[0097] Furthermore, after the transition from step S109 to step S102, when step S104 is executed again, a different label may be selected as the target label and the modification may be accepted. For example, as explained in Figure 9A, after executing a series of processes with the label with the smallest distance from the group center as the target label, when step S104 is executed again, the label with the second smallest distance from the group center may be selected as the target label.

[0098] Furthermore, although Figure 10 explains that the process proceeds to step S102 if the required performance is not achieved in step S109, this can be changed as appropriate. For example, if the processing circuit 25 does not achieve the required performance in step S109, it may proceed to step S103. In this case, the results of the clustering performed once are repeatedly used to execute the processing from step S103 onwards. Alternatively, for example, if the processing circuit 25 does not achieve the required performance in step S109, it may proceed to step S104. In this case, the processing from step S104 onwards is repeatedly executed for the group that was selected once.

[0099] As described above, the labeling function 25b assigns a label to each of the multiple medical images that indicates the analysis result based on the medical image. The clustering function 25c clusters the multiple labels assigned to each of the multiple medical images based on the similarity of the labels. The acquisition function 25d accepts a modification to at least one target label among the clustered multiple labels and acquires the relationship between the target label and the modified target label. The modification function 25e modifies the label assigned to a second medical image that is different from the first medical image corresponding to the target label, based on the relationship between the target label and the modified target label. This allows the medical information processing device 20 to reduce the burden on the user in preparing training data.

[0100] (Second embodiment) In the first embodiment, as explained in Figure 9A, an example was described in which a typical label within a group is selected as the target label and modifications are accepted. In the second embodiment, another method for selecting the target label is described.

[0101] The medical information processing system 1 according to the second embodiment can be configured in the same manner as the medical information processing system 1 shown in Figure 1. Hereinafter, the same reference numerals are used for parts described in the first embodiment, and their descriptions are omitted.

[0102] In the first embodiment, it was explained that the model's performance gradually improves with repeated additional training, eventually meeting the required performance. However, due to various factors, there may be cases where the model's performance deteriorates with additional training. For example, the model's performance may degrade if additional training is performed using inappropriate training data.

[0103] Therefore, the acquisition function 25d selects a target label according to the evaluation result of the model's performance. For example, as shown in Figure 11, the acquisition function 25d determines whether the model's performance has improved or worsened as a result of additional training, and selects a target label based on the determination result.

[0104] For example, the acquisition function 25d calculates the similarity between the previous label, which was selected as the target label in the previous additional learning, and each of the multiple labels clustered into the same group as the previous label. The similarity calculation for selecting the target label will be explained using Figure 12. Figure 12 is a diagram showing an example of the similarity calculation process according to the second embodiment. Figure 12 illustrates the case when the label assigned to the medical image of subject P3 is selected as the target label and the final additional learning is performed. In this case, the acquisition function 25d calculates the similarity between the label assigned to the medical image of subject P3 and each of the other labels included in group G1.

[0105] Here, if the model's performance deteriorated after the previous additional training, the acquisition function 25d selects the label with the minimum similarity as the target label for the current additional training. This is because if the model's performance deteriorated after the previous additional training, it is possible that the target label selected in the previous additional training was not valid. In other words, an invalid target label was selected, an invalid transformation f was obtained based on that target label, and as a result of correcting each label based on that transformation f, the generation of data for additional training may not have been successful, and the additional training may have also failed. By selecting the label with the minimum similarity as the target label for the current additional training, the case where the additional training fails as in the previous instance can be avoided, and it is expected that the model's performance will improve.

[0106] Furthermore, if the model's performance improved as a result of the previous additional training, the acquisition function 25d selects the label with the highest similarity as the target label for the current additional training. In other words, if the model's performance improved as a result of the previous additional training, it is presumed that the target label selected in the previous additional training was appropriate. Therefore, by selecting a similar target label as the target label for the current additional training, it is expected that the model's performance will improve, similar to the previous additional training.

[0107] Next, the flowchart in Figure 13 will be used to explain the sequence of processes in the medical information processing device 20. Figure 13 is a flowchart showing the sequence of processes in the processing circuit 25 according to the second embodiment.

[0108] Steps S201 and S208 correspond to the labeling function 25b. Step S202 corresponds to the clustering function 25c. Steps S203, S204, S205, S210, S211, and S212 correspond to the acquisition function 25d. Step S206 corresponds to the correction function 25e. Step S207 corresponds to the learning function 25f. Step S209 corresponds to the evaluation function 25g.

[0109] The processing in steps S201 to S208 is the same as the processing in steps S101 to S108 shown in Figure 10, so a detailed explanation is omitted. The processing circuit 25 determines in step S209 whether the required performance has been achieved, and if it has not been achieved (negation of step S209), it determines whether the model's performance has improved due to the previous additional learning (step S210).

[0110] If the model's performance has improved as a result of the previous additional training, the processing circuit 25 proceeds to step S211 and selects the label with the highest similarity. On the other hand, if the model's performance has improved as a result of the previous additional training, the processing circuit 25 proceeds to step S212 and selects the label with the lowest similarity. The processing circuit 25 then proceeds to step S204 again, using the label selected in step S211 or step S212 as the target label, and accepts modifications to the target label. If the required performance is achieved in step S209, the processing circuit 25 terminates the process.

[0111] (Other embodiments) In addition to the first and second embodiments described above, various modifications may be made to carry out the invention.

[0112] For example, in the embodiment described above, we explained an example of segmenting parts with relatively simple shapes, as shown in Figure 4, etc. Many organs have relatively simple shapes, and as explained in Figure 9A, user modifications can be described using affine transformations, etc.

[0113] However, for complex shapes such as vascular structures with multiple branches, it may be difficult to describe them using affine transformations. In other words, in the case of organs, user modifications involve changing the shape of existing regions, making description using affine transformations possible. On the other hand, in the case of vascular structures with multiple branches, as shown in Figure 14, for example, user modifications may involve adding regions rather than changing the shape of existing regions.

[0114] Therefore, the acquisition function 25d may describe the relationship between the target label and the modified target label using a machine learning model such as a neural network. For example, Figure 14 illustrates an example of segmenting the blood vessels of both lungs from a medical image of the chest. In the label inferred by the segmentation model, only the blood vessel region of one lung is segmented, but the user's modification adds the blood vessel region of the other lung.

[0115] Here, the acquisition function 25d inputs the medical image, the label before the user added blood vessels, and the label after the user added blood vessels into a neural network for training. This allows the neural network to learn the symmetry of the blood vessels to be segmented, as well as the pixel values ​​of the blood vessels and surrounding regions, enabling it to segment the blood vessels in both lungs when similar labels are input. Such a neural network is an example of a transformation f.

[0116] Furthermore, although the above-described embodiment was explained in which a labeling function 25b is used to assign a label to each of multiple medical images, the embodiment is not limited to this. That is, the assignment of labels may be performed by a device other than the medical information processing device 20. For example, medical images with labels assigned may be stored in an image storage device 30, and the control function 25a may acquire such medical images via a network NW. In this case, the medical information processing device 20 does not need to have a labeling function 25b.

[0117] Furthermore, additional learning based on labels may be performed on a device other than the medical information processing device 20. For example, the control function 25a may transmit the modified target labels that have been modified by the user, or the labels modified by the modification function 25e, as learning data to an external device, and additional learning may be performed on that external device. In this case, the medical information processing device 20 does not need to have a learning function 25f.

[0118] Furthermore, while we have described an example of active learning in which additional learning is repeated while checking the performance of the model, the embodiments are not limited to this. For example, the configuration may simply involve correcting the label using the correction function 25e and performing additional learning using the corrected label each time the user corrects the target label. In other words, the process of checking the performance of the model may be omitted, and the configuration may simply involve performing additional learning when the user corrects the target label. In this case, the medical information processing device 20 does not need to have an evaluation function 25g.

[0119] Furthermore, while we have described examples of collecting training data for additional training, the embodiments are not limited to these. For example, the above embodiments may be applied when collecting training data to generate the pre-trained model shown in Figure 3, and the collection of training data may be supported by the automatic label correction function 25e.

[0120] Furthermore, the embodiments described above mainly explained examples of performing additional training on a segmentation model. However, the embodiments are not limited to this. That is, the embodiments described above can be similarly applied to other models that perform analyses other than segmentation. For example, in a model that detects lesion candidates from input medical images, the location information of the lesion candidates can be assigned as labels.

[0121] Furthermore, the embodiments described above illustrate an example of performing additional training on a model that uses medical images as input. However, the embodiments are not limited to this. For example, the embodiments described above can be similarly applied to other models that analyze other medical information that is not images, such as text, audio data, or numerical data. For example, there is a known model that analyzes audio recorded during surgery and estimates the stage of the surgery at each point in time. In such a model, the stage of the surgery can be assigned to the audio data as a label.

[0122] Each component of the apparatus according to the above embodiment is a functional concept and does not necessarily have to be physically configured as shown in the illustration. That is, the specific form of distribution and integration of each apparatus is not limited to that shown in the illustration, and all or part of it can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions. Furthermore, each processing function performed by each apparatus can be implemented in whole or in any part by a CPU and a program that is analyzed and executed by the CPU, or by hardware using wired logic.

[0123] Furthermore, the medical information processing method described in the above-mentioned embodiments can be implemented by executing a pre-prepared program on a computer such as a personal computer or workstation. This program can be distributed via a network such as the Internet. Alternatively, this program can be recorded on a computer-readable non-transient recording medium such as a hard disk, flexible disk (FD), CD-ROM, MO, or DVD, and executed by reading it from the recording medium by a computer.

[0124] According to at least one embodiment described above, the burden on the user in preparing training data can be reduced.

[0125] While several embodiments have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These embodiments can be implemented in a variety of other forms, and various omissions, substitutions, modifications, and combinations of embodiments can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of symbols]

[0126] 1: Medical Information Processing System 10: Medical imaging diagnostic equipment 20: Medical Information Processing Device 21: Communication Interface 22: Input Interface 23: Display 24: Memory 25: Processing Circuit 25a: Control function 25b: Labeling function 25c: Clustering function 25d: Acquisition function 25e: Correction function 25f: Learning function 25g: Evaluation function 30: Image storage device

Claims

1. A label indicating the analysis results based on medical images, comprising a clustering unit that clusters multiple labels assigned to each of the multiple medical images based on the similarity of the labels, An acquisition unit receives a modification to at least one target label among a plurality of labels clustered by the clustering unit, and acquires the relationship between the target label and the modified target label in which the modification is reflected for that target label. A correction unit that corrects a label attached to a second medical image, which is different from the first medical image corresponding to the aforementioned target label, based on the aforementioned relationship. A medical information processing device equipped with [a specific feature].

2. The medical information processing apparatus according to claim 1, wherein the acquisition unit determines the shape transformation as the relationship.

3. The medical information processing apparatus according to claim 1, wherein the target label is the label with the smallest distance from the group center among a plurality of clustered labels.

4. The acquisition unit acquires a first relationship indicating the relationship between the target label and the modified target label, The medical information processing apparatus according to claim 1, wherein the modification unit further acquires a second relationship indicating the relationship between the target label and the label assigned to the second medical image, performs preprocessing on the label assigned to the second medical image based on the second relationship, and modifies the preprocessed label based on the first relationship.

5. The medical information processing apparatus according to claim 1, wherein the acquisition unit selects the group with the largest number of subjects from among a plurality of groups resulting from clustering by the clustering unit, and accepts a modification to at least one of the target labels included in the group with the largest number of subjects.

6. The medical information processing apparatus according to claim 1, further comprising a labeling unit for assigning labels to each of the multiple medical images.

7. The medical information processing apparatus according to claim 6, wherein the labeling unit assigns a label to each of the multiple medical images by applying a model that is configured to output analysis results based on the input medical images to each of the multiple medical images.

8. The medical information processing apparatus according to claim 7, further comprising a learning unit that performs additional training of the model using the modified target label and the label modified based on the relationship as training data.

9. The medical information processing apparatus according to claim 8, further comprising an evaluation unit for evaluating the performance of the model on which additional learning has been performed.

10. The medical information processing apparatus according to claim 9, wherein the acquisition unit selects the target label based on the evaluation result of the model's performance when the previous additional learning was performed.

11. The acquisition unit calculates the similarity between the previous label that was selected as the target label in the previous additional learning and each of the multiple labels clustered into the same group as the previous label, and if the performance of the model has deteriorated as a result of the evaluation, it selects the label with the smallest similarity as the target label for the current additional learning, as described in claim 10.

12. The medical information processing apparatus according to claim 1, wherein the clustering unit clusters a plurality of labels based on a first similarity indicating the similarity of the labels and a second similarity indicating the similarity of the medical images corresponding to the labels.

13. The medical information processing apparatus according to claim 1, wherein the target label is at least one label from a plurality of labels that have been grouped into the same group by the clustering unit.

14. A label indicating the analysis results based on medical images, wherein multiple labels assigned to each of the multiple medical images are clustered based on the similarity of the labels. The system accepts a modification to at least one target label among a clustered set of labels, and obtains the relationship between the target label and the modified target label in which the modification is reflected. The label assigned to a second medical image, which is different from the first medical image corresponding to the aforementioned target label, is modified based on the aforementioned relationship. A medical information processing method, including the following.

15. A label indicating the analysis results based on medical images, wherein multiple labels assigned to each of the multiple medical images are clustered based on the similarity of the labels. The system accepts a modification to at least one target label among a clustered set of labels, and obtains the relationship between the target label and the modified target label in which the modification is reflected. The label assigned to a second medical image, which is different from the first medical image corresponding to the aforementioned target label, is modified based on the aforementioned relationship. A program that instructs a computer to perform various processes.

Images