Medical image processing apparatus and medical image processing method
The medical image processing apparatus addresses the challenge of high learning time and load by dynamically adjusting epochs and using pseudo teacher data, achieving efficient and accurate image processing.
Patent Information
- Application Number
- JP2020175594
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-10-19
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2040-10-19
AI Technical Summary
Existing medical image processing technologies using machine learning face challenges in reducing learning time and processing load.
A medical image processing apparatus that dynamically determines the number of epochs in machine learning based on an initial value and the previous model's epoch number, using a semi-supervised learning approach with pseudo teacher data to reduce learning time and load.
This approach reduces learning time and processing load while maintaining accuracy by dynamically adjusting the number of epochs and utilizing pseudo teacher data, thereby minimizing unnecessary training and user workload.
Smart Images

Figure 0007591781000002 
Figure 0007591781000003 
Figure 0007591781000004
Abstract
Description
[Technical field]
[0001] The embodiments disclosed in this specification and the drawings relate to a medical image processing apparatus and a medical image processing method. [Background technology]
[0002] Conventionally, techniques using machine learning such as deep learning have been known in the field of image processing for medical images. In such techniques, there is a demand for reducing the learning time and the load of the learning process. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2020-93076 A Summary of the Invention [Problem to be solved by the invention]
[0004] One of the problems to be solved by the embodiments disclosed in this specification and the drawings is to reduce the learning time and the learning process load of machine learning in image processing of medical images. However, the problems to be solved by the embodiments disclosed in this specification and the drawings are not limited to the above problems. Problems corresponding to the effects of each configuration shown in the embodiments described later can also be positioned as other problems. [Means for solving the problem]
[0005] The medical image processing apparatus according to the embodiment includes a determination unit and an output unit. , No. Used in model 1 Ta Based on the first epoch number and the specified initial value , No. The number of second epochs used in the model 2 is dynamically determined. The output unit outputs the determined number of second epochs. The first model is a trained model generated by machine learning for a first number of epochs using a first medical image set including a prescribed number of medical images from a medical image group including a plurality of consecutive medical images of a subject. The second model is a trained model generated by further performing machine learning for a second number of epochs on the first model using a second medical image set including some medical images from the first medical image set and a medical image next to the last medical image in the first medical image set from the medical image group. [Brief description of the drawings]
[0006] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a medical image processing apparatus according to the first embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of image processing for a microscope image according to the first embodiment. [Diagram 3] FIG. 3 is a diagram for explaining details of image processing for a microscope image according to the first embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of a learning process and an estimation process using a model according to the first embodiment. [Diagram 5] FIG. 5 is an example of an algorithm for image processing using the model according to the first embodiment. [Figure 6] FIG. 6 is a flowchart showing an example of the flow of the semi-automatic region extraction process according to the first embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of the configuration of a medical image processing apparatus according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0007] Hereinafter, embodiments of a medical image processing apparatus and a medical image processing method will be described in detail with reference to the drawings.
[0008] (First embodiment) 1 is a diagram showing an example of the configuration of a medical image processing device 100 according to this embodiment. The medical image processing device 100 is, for example, a server device or a PC (Personal Computer). The medical image processing device 100 may be connected to other information processing devices such as a medical image storage device via a network such as an in-hospital LAN (Local Area Network) (not shown).
[0009] The medical image processing device 100 executes image processing on medical images by machine learning. In this embodiment, an electron microscope image captured by a scanning electron microscope (SEM) will be described as an example of a medical image. Hereinafter, the electron microscope image will be simply referred to as a microscope image.
[0010] FIG. 2 is a diagram showing an example of image processing for microscopic images 201 to 20N according to this embodiment. As shown in FIG. 2, the microscopic images 201 to 20N are consecutive images of slices cut out from a specimen captured by a scanning electron microscope. In this embodiment, the consecutive images mean that the slices that are the subjects of the imaging of each image are arranged adjacent to each other. The adjacent arrangement of the slices is not limited to the slices being cut out completely in contact with each other. For example, there may be a gap between the cut out positions of adjacent slices. The consecutive images are images of slices of adjacent tissues, and therefore have a strong correlation with each other. In this embodiment, the specimen is a human body or other living organisms, but is not limited thereto.
[0011] The microscope images 201-20N depict, for example, the nervous system of the brain or sensory organs of a living subject. In the field of connectomics, a method of reconstructing such microscope images 201-20N into three dimensions is generally used for detailed observation of neural structures.
[0012] The microscopic images 201-20N are medical images related to a subject. Hereinafter, when there is no need to distinguish between the individual microscopic images 201-20N, they will simply be referred to as microscopic images 20. Furthermore, when referring to a group of the microscopic images 201-20N, they will be referred to as a microscopic image group 2. Furthermore, the microscopic image group 2 is an example of a medical image group in this embodiment.
[0013] The number of the microscopic images 201 to 20N is not particularly limited, but may be, for example, about several hundred. In the following embodiment, the number of medical images included in the medical image group is set to N. The microscopic images 201 to 20N are also called serial slice images.
[0014] In order to reconstruct the microscopic image group 2 in three dimensions while clarifying the continuity of the neural structures depicted in the individual microscopic images 201-20N, for example, a method called an image processing pipeline can be adopted. In this method, image processing such as region extraction (segmentation) and grouping is performed on each of the microscopic images 201-20N.
[0015] 3 is a diagram for explaining details of image processing for a microscope image 20 according to this embodiment. In FIG. 3, one microscope image 20 included in the microscope image group 2 is illustrated as an example.
[0016] Region extraction is a process for distinguishing between nerve regions and other tissues in the microscopic image 20. A region-extracted image 30 shown in Fig. 3 is an image in which the microscopic image 20 has been region-extracted.
[0017] The region extraction is performed, for example, by a researcher manually drawing the outline of a nerve region. In this embodiment, an example in which the region extraction is performed automatically using a model described later will be described.
[0018] A researcher is an example of a user of the medical image processing device 100 in this embodiment. The user of the medical image processing device 100 is not limited to a researcher. The operations described below as being performed by a researcher may be performed by a doctor, a technician, or other workers.
[0019] Moreover, grouping refers to classifying a plurality of nerve regions in the region extraction image 30 into groups of the same nerve region. As in the grouping image 40 shown in Fig. 3, the same nerve region may be color-coded. In this embodiment, the image processing is performed manually by a researcher, but it may also be performed automatically by a model.
[0020] When the area extraction described in Fig. 3 is performed on each of the microscope images 201-20N included in the microscope image group 2, N area extraction images 301-30N corresponding to each of the microscope images 201-20N are generated, as shown in Fig. 2. Hereinafter, when there is no need to distinguish between the individual area extraction images 301-30N, they will simply be referred to as area extraction images 30.
[0021] Furthermore, when the individual area extraction images 301 to 30N are grouped, N grouped images 401 to 40N corresponding to the individual area extraction images 301 to 30N are generated as shown in Fig. 2. Hereinafter, when there is no need to distinguish between the individual grouped images 401 to 40N, they will simply be referred to as grouped image 40.
[0022] 2 is a result of three-dimensional reconstruction of grouping images 401 to 40N. In the three-dimensional image 50, the continuity of the nerve structure depicted in the successive microscope images 201 to 20N is represented three-dimensionally.
[0023] In this embodiment, a method for efficiently performing such image processing will be described.
[0024] 1, a description will be given of the configuration of the medical image processing device 100. The medical image processing device 100 includes a NW interface 110, a storage circuitry 120, an input interface 130, a display 140, and a processing circuitry 150.
[0025] The NW interface 110 is connected to the processing circuit 150, and controls the transmission and communication of various data between the medical image processing device 100 and other information processing devices. The NW interface 110 is realized by a network card, a network adapter, a NIC (Network Interface Controller), or the like.
[0026] The memory circuitry 120 stores in advance various types of information used by the processing circuitry 150. The memory circuitry 120 also stores various programs. The memory circuitry 120 may be provided outside the medical image-processing device 100.
[0027] The input interface 130 is realized by a trackball, a switch button, a mouse, a keyboard, a touch pad that performs an input operation by touching the operation surface, a touch screen in which a display screen and a touch pad are integrated, a non-contact input circuit using an optical sensor, a voice input circuit, etc. The input interface 130 is connected to the processing circuit 150, and converts an input operation received from an operator into an electric signal and outputs it to the processing circuit 150. Note that in this specification, the input interface 130 is not limited to an interface having physical operation parts such as a mouse and a keyboard. For example, an electric signal processing circuit that receives an electric signal corresponding to an input operation from an external input device provided separately from the device and outputs the electric signal to the processing circuit 150 is also included as an example of the input interface 130.
[0028] The display 140 is a liquid crystal display, an organic electro-luminescence (OEL) display, or the like. The input interface 130 and the display 140 may be integrated. For example, the input interface 130 and the display 140 may be realized by a touch panel. The display 140 is an example of a display unit in this embodiment. The display 140 may be provided outside the medical image-processing device 100.
[0029] The processing circuit 150 is a processor that reads out programs from the memory circuit 120 and executes them to realize functions corresponding to each program. The processing circuit 150 of this embodiment includes an acquisition function 151, a learning function 152, an estimation function 153, a decision function 154, an output function 155, a reception function 156, and a reconstruction function 157. The acquisition function 151 is an example of an acquisition unit. The learning function 152 is an example of a learning unit. The estimation function 153 is an example of an estimation unit. The decision function 154 is an example of a decision unit. The output function 155 is an example of an output unit. The reception function 156 is an example of a reception unit. The reconstruction function 157 is an example of a reconstruction unit.
[0030] Here, for example, each processing function of the acquisition function 151, learning function 152, estimation function 153, decision function 154, output function 155, acceptance function 156, and reconstruction function 157, which are components of the processing circuit 150, is stored in the storage circuit 120 in the form of a program executable by a computer. The processing circuit 150 is a processor. For example, the processing circuit 150 realizes a function corresponding to each program by reading the program from the storage circuit 120 and executing it. In other words, the processing circuit 150 in a state in which each program is read has each function shown in the processing circuit 150 in FIG. 1. Note that, in FIG. 1, it has been described that the processing functions performed by the acquisition function 151, learning function 152, estimation function 153, decision function 154, output function 155, acceptance function 156, and reconstruction function 157 are realized by a single processor, but it is also possible to configure the processing circuit 150 by combining multiple independent processors, and realize the functions by each processor executing a program. In addition, in FIG. 1, a single memory circuit 120 is described as storing a program corresponding to each processing function, but a configuration may be adopted in which multiple memory circuits are distributed and the processing circuit 150 reads out the corresponding program from each individual memory circuit.
[0031] In the above description, an example has been described in which the "processor" reads out and executes a program corresponding to each function from a storage circuit, but the embodiment is not limited to this. The term "processor" refers to a circuit such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an Application Specific Integrated Circuit (ASIC), a programmable logic device (e.g., a Simple Programmable Logic Device (SPLD), a Complex Programmable Logic Device (CPLD), and a Field Programmable Gate Array (FPGA)). When the processor is, for example, a CPU, the processor realizes a function by reading out and executing a program stored in the storage circuit 120. On the other hand, when the processor is an ASIC, instead of storing a program in the storage circuit 120, the function is directly incorporated as a logic circuit in the circuit of the processor. Note that each processor in the present embodiment is not limited to being configured as a single circuit for each processor, and may be configured as one processor by combining multiple independent circuits to realize the function. Furthermore, multiple components in FIG. 1 may be integrated into one processor to realize the functions thereof.
[0032] The acquisition function 151 acquires the microscope image group 2. For example, the acquisition function 151 acquires the microscope image group 2 from an image storage device (not shown) via the NW interface 110, but the acquisition source is not limited to this.
[0033] Moreover, the acquisition function 151 acquires training data to be input into a model, which will be described later.
[0034] The teacher data in this embodiment are area extraction images 301-303 corresponding to the first M microscopic images 201-203 in the microscopic image group 2. M is an example of a specified number in this embodiment. M is a number less than N, which is the number of microscopic images 20 included in the microscopic image group 2. In this embodiment, M=3 as an example. N may be referred to as the first number, and M may be referred to as the second number.
[0035] That is, in this embodiment, the first M of the microscopic images 20 included in the microscopic image group 2 are associated with area extraction images 30 created by a researcher as teacher data. The M area extraction images 30 corresponding to the first M of the microscopic images 20 are an example of a first teacher data set in this embodiment. Moreover, the microscopic images 20 associated with the area extraction images 30, which are the teacher data, are called labeled samples.
[0036] The source of the region extraction image 30 created by the researcher may be, for example, an information processing device used by the researcher. When the researcher generates the region extraction image 30 in the medical image processing device 100, the acquisition function 151 may read out the region extraction image 30 stored in the memory circuitry 120, for example.
[0037] The learning function 152 causes the model to execute the first learning using the first medical image set and the first teacher data set corresponding to the first medical image set, for the number of epochs output by the output function 155 described below. In addition, the learning function 152 causes the model to re-learn using pseudo teacher data generated by the estimation function 153 described below using the model.
[0038] 4 is a diagram showing an example of a learning process and an estimation process by the models 601 to 60N according to this embodiment. Hereinafter, the models 601 to 60N will be simply referred to as the model 60 when no particular distinction is made between them.
[0039] The model 60 is a model generated by a machine learning technique. For example, the model 60 is generated by a deep learning technique such as DCN (Deep Contextual Network), but other machine learning techniques may be adopted. The model 60 is stored in the storage circuitry 120. In this case, the learning function 152 and the estimation function 153 described later read the model 60 from the storage circuitry 120 and execute the learning process or the estimation process. The model 60 may be incorporated in the learning function 152 or the estimation function 153 described later.
[0040] In this embodiment, the number of the model 60 is updated every time learning is performed using a pair of a medical image set and a teacher data set. For example, the learning function 152 causes the first model 601 to perform the first learning for the first number of epochs using the first medical image set 21a and the first teacher data set 31a. The model 60 after the first learning is called the second model 602. Then, the learning function 152 causes the second model 602 to perform the second learning for the second number of epochs using the second medical image set 21b and the second teacher data set 31b. The second model 602 after learning becomes a third model. In other words, the learning function 152 generates a third model by causing the second model 602 to perform the second learning.
[0041] The first model 601 shown in Fig. 4 is an example of the first model in this embodiment, and the second model 602 is an example of the second model in this embodiment. In Fig. 1, the first model 601 is the leading model 60 of all the models 601 to 60N, but the first model is not limited to this. The first model and the second model simply mean two models adjacent to each other.
[0042] In the first learning and the second learning, the number of epochs of the learning process repeated for each set of medical images and teacher data set is dynamically determined by a later-described determination function 154. In the first learning, the learning function 152 causes the first model 601 to execute learning for the number of epochs determined by the determination function 154 using the first medical image set 21a and the first teacher data set 31a.
[0043] In addition, in the second learning, the learning function 152 causes the second model 602 to perform learning for the second number of epochs determined by the determination function 154 using the second medical image set 21b and the second teacher data set 31b.
[0044] Here, the medical image set and the teacher data set will be described in more detail. As shown in Fig. 4, the microscopic images 201-203 are associated with area extraction images 301-303 created by a researcher. The area extraction image 30 used as teacher data is called a label. Moreover, a label created manually is called a true label, and a label estimated by the model 60 is called a pseudo label. The pseudo label is an example of pseudo teacher data in this embodiment.
[0045] Moreover, a set of three microscopic images 20 included in the microscopic image group 2 is an example of a first medical image set 21a in this embodiment. In the example shown in FIG. 4, the microscopic images 201 to 203 are the first medical image set 21a. Moreover, the microscopic images 202 to 204 are the second medical image set 21b. The second medical image set 21b includes the microscopic images 202 and 203 that are part of the first medical image set 21a, and the microscopic image 204 that is not included in the first medical image set 21a. The microscopic image 204 is located next to the microscopic image 203 among the successive microscopic images 201 to 20N included in the microscopic image group 2.
[0046] Furthermore, the microscopic images 203-205 constitute a third medical image set 21c. In FIG. 4, the fourth and subsequent medical image sets are not shown, but the microscopic images 201-20N included in the microscopic image group 2 are used for learning by the model 60 in sets of M images each. Hereinafter, when the individual medical image sets 21a-21c are not distinguished from each other, they are simply referred to as medical image sets 21. In addition, in FIG. 4, teacher data sets 31a-31c corresponding to the respective medical image sets 21 are shown, but when they are not distinguished from each other, they are simply referred to as teacher data sets 31.
[0047] When the learning function 152 finishes learning using one medical image set 21, it generates the next medical image set 21 by replacing the microscopic images 20 included in the medical image set 21 one by one. Specifically, of the consecutive microscopic images 201 to 203, the one closest to the beginning is excluded, and a microscopic image 204 not included in the first medical image set 21a is added. The added microscopic image 204 is the microscopic image 204 next to the last microscopic image 203 in the first medical image set 21a. By such replacement, as described above, the microscopic images 202 to 204 are selected as the second medical image set 21b.
[0048] The first teacher data set 31a shown in FIG. 4 corresponds to the first medical image set 21a. The first teacher data set 31a includes area extraction images 301-303 that are correct labels. The second teacher data set 31b corresponds to the second medical image set 21b. The second teacher data set 31b includes area extraction images 302 and 303 that are part of the first teacher data set 31a, and an area extraction image 304 that is a pseudo label. The area extraction images 302 and 303 are teacher data that correspond to the microscopic images 202 and 203, as described above. The microscopic images 202 and 203 are two images included in the second medical image set 21b among the microscopic images 201-203 included in the first medical image set 21a.
[0049] In the first learning, the first teacher data set 31a is composed of only correct labels, but in the second learning and onwards, the teacher data set 31 includes pseudo labels generated by the model 60. In this way, the learning function 152 of this embodiment causes the model 60 to perform re-learning using the pseudo labels that are the estimation results by the model 60. This type of learning method is called semi-supervised learning.
[0050] Next, the estimation process by the model 60 will be described. Returning to FIG. 1, the estimation function 153 inputs a microscopic image 20, of the microscopic image group 2, to which no teacher data is associated, to the model 60 that has already undergone one or more rounds of learning, and causes the model 60 to output pseudo teacher data corresponding to the microscopic image 20. For example, the estimation function 153 inputs a microscopic image 20, of the microscopic image group 2, to which no teacher data is associated, to a second model generated as a result of learning from the first model, and causes the model 60 to output pseudo teacher data corresponding to the microscopic image 20.
[0051] More specifically, the estimation function 153 inputs the microscope image 204 to the second model 602 generated by the first learning, thereby outputting the region extraction image 304 from the second model 602. As described above, the microscope image 204 is the image next to the last microscope image 203 included in the first medical image set 21a. The region extraction image 304 output from the second model 602 is used as teacher data in the next learning by the learning function 152. Furthermore, when the generation of the region extraction images 301 to 30N corresponding to all the microscope images 201 to 20N included in the microscope image group 2 is completed, the region extraction images 301 to 30N are used to generate the grouping images 401 to 40N as described in FIG. 2.
[0052] Here, the algorithm of image processing using the model 60 by the learning function 152 and the estimation function 153 will be described with reference to FIG. 5 in addition to FIG.
[0053] FIG. 5 shows an example of an algorithm for image processing using the model 60 according to this embodiment.
[0054] In the algorithm shown in FIG. 5, a group of microscope images 2, which are continuous images composed of N slices, is called "serial images X." Furthermore, M labeled microscope images 20 are called "labeled samples M." Furthermore, microscope images 201 to 20N included in the group of microscope images 2 are called "X1, ..., X N ". Also, the region extraction images 301 to 30N corresponding to the microscopic images 201 to 20N are denoted by "T1, . . . , T N "
[0055] Moreover, “i” is a number for distinguishing the model 60. Every time learning is performed using a set of medical image sets and a teacher data set, “1” is added to “i” by the learning function 152. At the start of the first learning, the learning function 152 sets “i=1”. The value of the parameter “i” is stored in, for example, the storage circuitry 120, and is updated by the learning function 152. In other words, every time learning is performed using a set of medical image sets and a teacher data set, the version of the model 60 is updated, and “i” is a number indicating the version of the model 60. Note that the number of epochs of learning is performed for each set of medical image sets and a teacher data set. The number of epochs in this embodiment differs for each version of the model 60.
[0056] Specifically, the learning function 152 and the estimation function 153 execute the following processes (1) to (4) in order.
[0057] (1) In the first learning, the learning function 152 performs learning of the model (i) for the number of epochs using the (i+M-1)th labeled image from the i-th slice. The model (i) is the i-th model 60, and in the first learning, it is the first model 601. In the example shown in FIG. 4, the "(i+M-1)th labeled image from the i-th slice" is the microscopic images 201 to 203. The number of epochs is determined by the determination function 154, which will be described later. As an example, in the first learning, the number of epochs is determined to be 200 epochs. In this case, the learning function 152 performs learning of the first model 601 for 200 epochs using the microscopic images 201 to 203 and the area extraction images 301 to 30N.
[0058] (2) The estimation function 153 uses the model (i) learned in (1) to perform estimation for the (i+M)th image. In the example shown in Fig. 4, the "i+Mth image" is the fourth microscope image 204 from the top. The estimation function 153 inputs the microscope image 204 to the trained first model 601.
[0059] (3) The estimation function 153 sets the estimation result of (2) as the label of the (i+M)th image. That is, the estimation function 153 sets the region extraction image 304 output from the first model 601 as the pseudo label corresponding to the fourth microscope image 204.
[0060] (4) The learning function 152 adds 1 to the number of the model 60, making it "i=i+1". That is, the first model 601 after learning becomes the second model 602. The next learning is the second learning. In this way, the learning function 152 generates the second model 602 by learning the first model 601.
[0061] The learning function 152 and the estimation function 153 repeat the above processes (1) to (4) until the model (NM) is learned and the Nth image is estimated. The number of epochs used in (1) varies depending on the value of "i". However, the models (1) to (NM) are similar machine learning models, and the learned parameters are not initialized. That is, in the second learning, the second model 602 is used on the premise of the first learning. In the third learning, the third model 603 is used on the premise of the first learning and the second learning. The model (1) is the first model 601, and the model (NM) is the last model 60. For example, when N=300 and M=3, the model (NM) is the 297th model.
[0062] As the version of the model 60 increases, the amount of teacher data included in the teacher dataset 31 decreases, and instead the amount of pseudo teacher data increases. Then, from model (M+1) onwards, the teacher dataset 31 does not include teacher data, but only pseudo teacher data.
[0063] 1, the determination function 154 dynamically determines the number of epochs to be used in a certain model 60 based on a specified initial value and the number of epochs of the model 60 immediately preceding the certain model 60. More specifically, the determination function 154 determines the number of epochs for each model 60 by equation (1).
[0064]
number
[0065] As explained in FIG. 5, "i" in formula (1) is a number that distinguishes the model 60. Also, "init" is a prescribed initial value (hereinafter simply referred to as the initial value). The value of the initial value is not particularly limited, but may be, for example, 200 epochs. When "i=1", that is, in the first learning, the determination function 154 determines the initial value as the first epoch number.
[0066] According to formula (1), when "i ≠ 1", the determination function 154 determines the larger of one hundredth of the initial value or one half of the number of epochs in the "i-1"th learning as the number of epochs. The "i-1"th learning is the previous learning. For example, in the second learning, since "i = 2", the determination function 154 compares "2", which is one hundredth of the initial value, with "100", which is one half of the first number of epochs in the first learning. In this case, the determination function 154 determines "100" as the second number of epochs in the second learning. When formula (1) is used, the determination function 154 halves the number of epochs each time the value of "i" increases as long as the number of epochs is "2" or more.
[0067] In addition, "one hundredth of the initial value" in formula (1) is an example of the minimum number of epochs. The minimum number of epochs is the number of epochs at which the learning of the model 60 satisfies a specified accuracy. The minimum value may be, for example, a value obtained by prior verification. In addition, "one half of the number of epochs in the previous learning" is an example of a reduction rate of the number of epochs.
[0068] In other words, the determination function 154 reduces the number of second epochs used in the second model 602 by a specified reduction rate compared to the number of first epochs used in the previous first model 601, as long as the number of epochs is equal to or greater than a specified minimum value.
[0069] The initial value, the minimum value, and the decrease rate may be determined in advance, or may be specified by the user.
[0070] The determination function 154 sends the determined epoch number to the output function 155 .
[0071] Returning to FIG. 1, the output function 155 outputs the number of epochs determined by the determination function 154. For example, when a second number of epochs is determined by the determination function 154, the output function 155 outputs the second number of epochs. In this embodiment, "output" includes setting the number of epochs to the model 60. As a specific example of setting, for example, the number of epochs of the current learning may be stored in the storage circuitry 120 by the output function 155 as a parameter of the learning process. Furthermore, when the learning process is executed by an information processing device different from the medical image processing device 100, "output" includes transmission of the number of epochs to the information processing device.
[0072] The reception function 156 receives various operations by the user. For example, the reception function 156 receives an operation to start image processing by the user. The reception function 156 also receives a grouping operation by the user for the area extraction images 301-303 acquired as a correct model and the area extraction images 304-30N output from the model 60 by the estimation function 153. The results of grouping the area extraction images 301-30N by the user are grouped images 401-40N.
[0073] The reconstruction function 157 reconstructs the grouping images 401 to 40N to generate a three-dimensional image 50. The reconstruction function 157 stores the generated three-dimensional image 50 in, for example, the storage circuitry 120. Alternatively, the reconstruction function 157 may transmit the generated three-dimensional image 50 to another information processing device, or may display it on the display 140.
[0074] Next, the flow of the semi-automatic region extraction process executed by the medical image processing apparatus 100 configured as above will be described.
[0075] 6 is a flowchart showing an example of the flow of semi-automatic region extraction processing according to this embodiment. The processing of this flowchart starts, for example, when the reception function 156 receives an operation to start image processing by the user.
[0076] First, the acquisition function 151 acquires the microscopic image group 2 to be processed (S1). The acquisition function 151 also acquires the region extraction images 301 to 303, which are the first teacher data set 31a (S2).
[0077] Next, the learning function 152 sets "i" to "1" (S3).
[0078] Next, the decision function 154 judges whether "i=1", that is, whether the current learning is the first time (S4). In the first learning, since the parameter "i" is set to "1" in S3, "i=1" is obtained (S4 "Yes").
[0079] Then, the determination function 154 determines the number of epochs using equation (1). When "i=1", the determination function 154 determines a prescribed initial value as the number of epochs (S5). In this flowchart, the initial value is "200". The output function 155 sets the number of epochs "200" determined by the determination function 154 as the number of epochs of the first model 601.
[0080] Then, the learning function 152 inputs the first M microscope images 20 in the microscope image group 2 and the first teacher data set 31a to the first model 601 (S6). In this flowchart, the first M microscope images 20 are microscope images 201 to 203.
[0081] Then, the learning function 152 executes the learning process of the first model 601 using the microscopic images 201 to 203 and the area extracted images 301 to 303 for 200 epochs (S7).
[0082] Next, the estimation function 153 causes the first model 601 after learning, i.e., the second model 602, to output a pseudo label corresponding to the next microscope image 20 (S8). Specifically, the estimation function 153 inputs the microscope image 204 to the first model 601, and sets the region extraction image 304 output from the first model 601 as the pseudo label corresponding to the microscope image 204.
[0083] Next, the estimation function 153 judges whether or not labeling has been completed for all the microscopic images 20 included in the microscopic image group 2 (S9). The estimation function 153 judges that labeling has not been completed until an area extraction image 30N corresponding to the Nth microscopic image 20N is generated (S9 "No"). In this case, the estimation function 153 sends the generated pseudo label to the learning function 152, for example.
[0084] Next, the learning function 152 adds "1" to "i" (S10), and then returns to the process of S4.
[0085] The decision function 154 again determines whether or not "i=1" (S4). In the case of the second learning, "i=2" is satisfied, and therefore the decision function 154 determines that "i≠1" (S4 "No").
[0086] Then, the determination function 154 determines the number of epochs using formula (1) (S11). For example, in the second learning, the determination function 154 determines the number of epochs in the second learning to be “100”, which is half the number of epochs in the first learning. In addition, the output function 155 sets the number of epochs “100” determined by the determination function 154 as the number of epochs of the second model 602.
[0087] Then, the learning function 152 inputs the M microscopic images 20 that are the subject of this learning and the second teacher data set 31b to the second model 602 (S12). In this flowchart, the M microscopic images 20 that are the subject of the second learning are the microscopic images 202 to 204. Then, the process proceeds to S7, where the second learning is executed.
[0088] After the processes of S7 and S8, the estimation function 153 again judges whether or not labeling has been completed for all the microscopic images 20 included in the microscopic image group 2 (S9). When an area extraction image 30N corresponding to the Nth microscopic image 20N has been generated, it is judged that labeling has been completed (S9 "Yes"). In this case, the process of this flowchart ends.
[0089] After the process of this flowchart is completed, the reception function 156 receives a grouping operation from the user, and the reconstruction function 157 performs three-dimensional reconstruction processing.
[0090] In this way, according to the medical image processing device 100 of this embodiment, the number of epochs used in the model 60 is dynamically determined based on the initial value and the number of epochs of the previous model 60, thereby making it possible to reduce the learning time and the learning processing load of machine learning compared to the case of applying a fixed number of epochs. For example, compared to the case of executing 200 epochs, which is the same as the initial value, regardless of the version of the model 60, when the number of epochs is dynamically changed, learning of a number of epochs more than necessary is not executed, so that it is possible to reduce the learning time and the processing load.
[0091] Furthermore, in this embodiment, since multiple consecutive medical images are used as the learning subject, even if the number of epochs is reduced depending on the version of model 60, the deterioration in learning accuracy can be reduced by accumulating learning data from previous times.
[0092] Furthermore, the medical image processing apparatus 100 of this embodiment reduces the second number of epochs by a specified reduction rate from the first number of epochs used in the previous model 60, as long as the second number of epochs is equal to or greater than a specified minimum value. In other words, the medical image processing apparatus 100 of this embodiment reduces the number of epochs based on a specified reduction rate each time a version of the model 60 is increased, as long as the number of epochs is equal to or greater than a specified minimum value. Therefore, according to the medical image processing apparatus 100 of this embodiment, it is possible to reduce the learning time and the processing load while maintaining the accuracy of the model 60 to a certain extent.
[0093] In addition, the medical image processing apparatus 100 of this embodiment generates a second model 602 by executing the first learning for the first number of epochs in the first model 601 using the first medical image set 21a, which is the first M microscopic images 20 in the microscopic image group 2, and the first teacher data set 31a corresponding to the first medical image set 21a. Then, in the next learning, the medical image processing apparatus 100 of this embodiment uses the pseudo teacher data estimated by the second model 602 to cause the second model 602 to relearn. Therefore, according to the medical image processing apparatus 100 of this embodiment, if there is teacher data corresponding to the first M microscopic images 20 among the consecutive microscopic images 201 to 20N, it is possible to automatically generate pseudo teacher data corresponding to the remaining microscopic images 20 and perform relearning. Therefore, the user does not need to prepare teacher data corresponding to all the microscopic images 201 to 20N, and the user's workload and work time for creating teacher data are reduced.
[0094] In addition, by adopting such a method, the microscopic image group 2 itself, which is the target of image processing, can be used for learning, even if the user does not prepare a learning-only data set. In general, it is difficult even for a skilled expert to obtain a beautiful continuous image using an electron microscope, so the quality of the captured image may not be stable. For this reason, it is difficult to obtain multiple data sets of the same type. Due to such circumstances, it may be difficult to apply a method of creating a model for one learning-only data set including multiple continuous microscopic images and using the created model for other data that is the target of image processing. In contrast, in the medical image processing device 100 of this embodiment, it is not necessary to prepare a learning-only microscopic image group and a teacher image data group corresponding to the learning-only microscopic image group, in addition to the microscopic image group 2 that is the target of image processing. Therefore, according to the medical image processing device 100 of this embodiment, the workload for the user to obtain or create learning data and teacher data can be reduced.
[0095] Moreover, the medical image processing apparatus 100 of this embodiment causes the first model 601 after the first learning, i.e., the second model 602, to output pseudo teacher data corresponding to the next microscopic image 204 of the last microscopic image 203 of the first medical image set 21a among the microscopic image group 2. Moreover, the medical image processing apparatus 100 of this embodiment causes the second model 602 to execute the second learning for the number of epochs using the second medical image set 21b and the second teacher data set 31b. In this embodiment, the microscopic image 20 to be learned among the microscopic image group 2 moves one by one, and the medical image set 21 and the teacher data set 31 used for learning include a part of the microscopic image 20 and the teacher data used in the previous learning, so that even when the pseudo teacher data is used, the deterioration of the learning accuracy can be reduced.
[0096] In addition, the consecutive microscopic images 201-20N included in the microscopic image group 2 in this embodiment are physically consecutive images. Therefore, adjacent microscopic images 20 among the microscopic images 201-20N are similar to each other, so that pseudo teacher data corresponding to the next microscopic image 20 can be generated with high accuracy from the learning results up to the previous time.
[0097] Second embodiment In the second embodiment, the medical image processing apparatus 100 evaluates the confidence level of pseudo teaching data generated by the model 60.
[0098] Fig. 7 is a diagram showing an example of the configuration of a medical image processing device 100 according to this embodiment. As shown in Fig. 7, the medical image processing device 100 includes a NW interface 110, a storage circuitry 120, an input interface 130, a display 140, and a processing circuitry 150, similar to the first embodiment described in Fig. 1.
[0099] The processing circuit 150 of this embodiment includes an acquisition function 151, a learning function 152, an estimation function 153, a determination function 154, an output function 155, a reception function 156, a reconstruction function 157, a calculation function 158, and a display control function 159. The calculation function 158 is an example of a calculation unit. The display control function 159 is an example of a display control unit. The acquisition function 151, the learning function 152, the estimation function 153, the determination function 154, the output function 155, the reception function 156, and the reconstruction function 157 include functions similar to those of the first embodiment.
[0100] The calculation function 158 calculates the confidence of the pseudo label output by the model 60. A known technique can be adopted as the method for calculating the confidence. The calculation function 158 sends the calculated confidence to the display control function 159.
[0101] The display control function 159 causes the display 140 to display a warning when the confidence level falls below a threshold value.
[0102] According to the medical image-processing apparatus 100 of this embodiment, by providing such a function, it is possible to make the user aware of the decrease in confidence level caused by repeated relearning.
[0103] Furthermore, the learning function 152 and the estimation function 153 may stop the area extraction process when the confidence level becomes equal to or lower than a threshold value.
[0104] For example, when a warning is issued, the user may manually generate the region extraction image 30 that is the teacher data correct label to be used in the next learning. In this case, the acquisition function 151 acquires the teacher data newly generated by the user. Furthermore, the learning function 152 can improve the accuracy of the learning again by executing learning of the model 60 using the newly acquired teacher data. Note that the user may completely generate the teacher data manually, or may create the teacher data by correcting the pseudo teacher data whose confidence level falls below a threshold.
[0105] In this embodiment, the confidence level has been described as an example, but other indices may be used.
[0106] (Variation 1) In the above-described embodiments, the microscope image 20 is taken as an example of a medical image, but the medical image is not limited to this. For example, the medical image may be a magnetic resonance image taken by a magnetic resonance imaging (MRI) device, or a CT image taken by an X-ray computed tomography (CT) device. Alternatively, the medical image may be an image taken by other modalities.
[0107] Furthermore, when the medical image is, for example, a magnetic resonance image or a CT image, the configuration of the medical image processing device 100 of each of the above-mentioned embodiments can be applied to measurement of the three-dimensional volume of a certain tissue (for example, a tumor) of a subject. For example, if a user generates teacher data indicating the tissue portion for M of N consecutive cross-sectional images in which the target tissue is depicted, the part in which the target tissue is depicted by the model 60 can be specified for the remaining cross-sectional images. The volume of the tissue can be specified from a three-dimensional image obtained by three-dimensionally reconstructing the N consecutive cross-sectional images in which the part in which the target tissue is depicted is specified. Therefore, according to the medical image processing device 100 of this modification, the user does not need to specify the part in which the target tissue is depicted for all N cross-sectional images, and the user's workload can be reduced.
[0108] (Variation 2) In the above-mentioned embodiments, the slices that are the imaging targets of the multiple microscopic images 201-20N are described as being physically continuous, but the continuity of the medical images is not limited to being physically continuous. For example, multiple medical images that are continuous in time and captured with a fixed imaging field of view may be an example of "multiple continuous medical images."
[0109] (Variation 3) In each of the above-described embodiments, the model 60 performs up to extracting the region of the microscope image 20, but it may also perform grouping. In this case, for example, the acquisition function 151 acquires grouping images 401 to 403 as teacher data. In addition, grouping images 404 to 40N are generated by semi-automatic image processing by the model 60.
[0110] Furthermore, the image processing performed by the model 60 is not limited to these. For example, when the medical image is a magnetic resonance image or a CT image, the model 60 may perform image processing such as segmentation of a lesion area.
[0111] (Variation 4) In the above-described second embodiment, an example has been described in which the user creates new teacher data when the confidence level falls below a threshold value, but the user may create new teacher data at a desired timing regardless of the confidence level value. As described in the second embodiment, the creation of new teacher data includes the correction of the pseudo teacher data.
[0112] (Variation 5) In each of the above-mentioned embodiments, the medical image processing apparatus 100 executes the region extraction process in sequence from the top of the microscopic image group 2, but the execution sequence is not limited to this. For example, the medical image processing apparatus 100 may divide the microscopic image group 2 into a plurality of groups and execute the region extraction process for each of the plurality of divided microscopic image groups.
[0113] (Variation 6) In the above-described embodiments, the entire image of the microscope image 20 is the target of image processing, but only a part of the image region may be the target. For example, when the medical image is a magnetic resonance image or a CT image, the target of image processing may be only a region of interest (ROI).
[0114] (Variation 7) The medical image processing apparatus 100 does not need to have all of the functions described in the above-mentioned embodiments. For example, the learning function 152 and the estimation function 153 may be executed by another information processing apparatus different from the medical image processing apparatus 100. That is, a configuration in which the model 60 is not provided in the medical image processing apparatus 100 may be adopted. When adopting such a configuration, the medical image processing apparatus 100 determines the number of epochs by the determination function 154, and transmits the determined number of epochs to the other information processing apparatus by the output function 155.
[0115] Moreover, the processing such as grouping and three-dimensional reconstruction may also be executed by an information processing device other than the medical image processing device 100.
[0116] Furthermore, some or all of the functions of the medical image processing apparatus 100 described in each of the above embodiments may be executed in a cloud environment.
[0117] The various data handled in this specification are typically digital data.
[0118] According to at least one of the embodiments described above, it is possible to reduce the learning time and the load of the learning process in machine learning in image processing of medical images.
[0119] Although some embodiments have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, and combinations of embodiments can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope of the invention and its equivalents as described in the claims, as well as in the scope and spirit of the invention. [Explanation of symbols]
[0120] 2. Microscope images 20,201~20N Microscope image 21, 21a-21c Medical image set 30,301~30N region extraction image 31, 31a-31c Teacher dataset 40,401~40N Grouping images 50 3D Images 60,601~60N models 100 Medical image processing device 110 Network Interface 120 Memory circuit 130 Input Interface 140 Display 150 Processing circuit 151 Acquisition Function 152 Learning Function 153 Estimation Function 154 Decision Function 155 Output Function 156 Reception Function 157 Reconfiguration function 158 Calculation Function 159 Display control function
Claims
1. A method for generating a first model, comprising: a determination unit that dynamically determines a second number of epochs to be used in a second model based on a first number of epochs used in machine learning to generate a first model and a specified initial value; an output unit that outputs the determined second epoch number; Equipped with The first model is a trained model generated by machine learning for a first number of epochs using a first medical image set including a prescribed number of medical images from a medical image group including a plurality of consecutive medical images related to a subject; The second model is a trained model generated by further performing machine learning for the second number of epochs on the first model using a second medical image set including some medical images of the first medical image set and a medical image next to the last medical image of the first medical image set among the medical image group. Medical imaging equipment.
2. the determination unit reduces the second number of epochs by a specified reduction rate from the first number of epochs as long as the second number of epochs is equal to or greater than a specified minimum value. The medical image processing device according to claim 1 .
3. An acquisition unit that acquires the medical image group and a first teacher data set including a plurality of teacher data corresponding to the first medical image set including a first medical image of the medical image group; a learning unit that generates the second model by causing the first model to execute a first learning for the second number of epochs. The medical image processing device according to claim 1 .
4. an estimation unit that inputs a medical image to which no teacher data is associated among the group of medical images into the second model, and outputs pseudo teacher data corresponding to the medical image; The learning unit causes the second model to perform second learning using the pseudo teacher data. The medical image processing device according to claim 3 .
5. the estimation unit inputs a medical image next to a last medical image in the first medical image set among the medical image group into the second model generated by the first learning, thereby outputting pseudo teacher data corresponding to the next medical image; the learning unit generates the second model by causing the first model to perform second learning for the second number of epochs output by the output unit, using the second medical image set and a second teacher dataset; The second teacher data set includes teacher data corresponding to the part of medical images in the first teacher data set, and the pseudo teacher data estimated by the second model. The medical image processing device according to claim 4 .
6. A calculation unit that calculates a certainty factor of the pseudo-teacher data output by the first model; a display control unit that displays a warning on a display unit when the certainty degree becomes equal to or less than a threshold value; Further comprising:
6. The medical image processing device according to claim 4.
7. The plurality of medical images included in the medical image group are physically or temporally continuous. The medical image processing device according to claim 1 .
8. The medical image is an electron microscope image captured by an electron microscope. The medical image processing device according to claim 1 .
9. The first model and the second model are trained on a partial image region of each medical image included in the medical image group. The medical image processing device according to claim 1 .
10. A method for dynamically determining a second number of epochs to be used in a second model based on a first number of epochs used in a first model and a specified initial value; an output step of outputting the determined second epoch number; Including, The first model is a trained model generated by machine learning for a first number of epochs using a first medical image set including a prescribed number of medical images from a medical image group including a plurality of consecutive medical images related to a subject; The second model is a trained model generated by further performing machine learning for the second number of epochs on the first model using a second medical image set including some medical images of the first medical image set and a medical image next to the last medical image of the first medical image set among the medical image group. Medical image processing method.
Citation Information
Patent Citations
Learning device, learning method, and program
JP2007280054A
Learning method, learning program, and information processing device
JP2017097807A
Robot device
JP2020044590A
Medical image processing device, learned model, medical image processing method and program
JP2020093076A
