Medical image processor, program, and method

Abstract

To reduce load on edition processing of medical image data.SOLUTION: A medical image processor includes an acquisition part and a learning part. The acquisition part acquires: a first outline representing a region of an anatomical tissue of a first subject drawn to first medical image data; a second outline different from the first outline representing a region of the anatomical tissue; and a position of at least one anatomical landmark in the first subject in the first medical image data. The learning part learns a difference between the first outline and the second outline on the basis of respective relative positions of the first outline and the second outline based on an anatomical landmark.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The embodiments disclosed in this specification and the drawings relate to a medical image processing apparatus, a program, and a method.

Background Art

[0002] Conventionally, a technique for automatically extracting, by an image processing application or the like, a region in which an anatomical tissue such as an organ to be diagnosed is depicted from medical image data has been known. In such a technique, it is common to detect the contour of an anatomical tissue based on the pixel values of medical image data. For this reason, in such a conventional technique, when the pixel values are similar even between different anatomical tissues, the detection accuracy of the boundary between the anatomical tissues may decrease. Further, in such a case, the workload of the user for correcting the contour of the anatomical tissue on the medical image data has been high.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] One of the problems to be solved by the embodiments disclosed in this specification and the drawings is to reduce the workload of the editing process of medical image data. However, the problems to be solved by the embodiments disclosed in this specification and the drawings are not limited to the above problems. The problems corresponding to the respective effects of the respective configurations shown in the embodiments described later can also be regarded as other problems.

Means for Solving the Problems

[0005] The medical image processing apparatus according to the embodiment includes an acquisition unit and a learning unit. The acquisition unit acquires a first contour representing a region of an anatomical tissue of a first subject depicted in first medical image data, a second contour different from the first contour representing the region of the anatomical tissue, and positions of at least one anatomical landmark in the first subject in the first medical image data. The learning unit learns the difference between the first contour and the second contour based on the relative positions of the first contour and the second contour with respect to the anatomical landmark.

Brief Description of the Drawings

[0006]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Best Mode for Carrying Out the Invention

[0007] Hereinafter, embodiments of a medical image processing apparatus, a program, and a method will be described in detail with reference to the drawings.

[0008] (First Embodiment) FIG. 1 is a diagram showing an example of the overall configuration of a medical image processing system S according to the first embodiment. As shown in FIG. 1, the medical image processing system S includes, as an example, a medical image processing apparatus 100 and a medical image diagnostic apparatus 200. The medical image processing system S is provided, for example, in a medical institution such as a hospital.

[0009] The medical image diagnostic apparatus 200 is an apparatus for taking a medical image of a subject, and examples thereof include an X-ray CT (Computed Tomography) apparatus, an X-ray diagnostic apparatus, an ultrasonic diagnostic apparatus, a PET (Positron Emission Tomography) apparatus, an MRI (Magnetic Resonance Imaging) apparatus, a SPECT (Single Photon Emission Computed Tomography) apparatus, etc., but are not limited thereto. The medical image diagnostic apparatus 200 is also referred to as a modality. In FIG. 1, one medical image diagnostic apparatus 200 is illustrated, but a plurality of medical image diagnostic apparatuses 200 may be provided. Hereinafter, in this embodiment, the case where the medical image diagnostic apparatus 200 is an X-ray CT apparatus that takes an X-ray CT image will be described as an example. The medical image diagnostic apparatus 200 is also referred to as a modality.

[0010] The medical image processing device 100 is a computer such as a server device or a PC (Personal Computer), for example. The medical image processing device 100 and the medical image diagnostic device 200 are communicably connected via a network 300 such as an in-hospital LAN (Local Area Network).

[0011] The medical image processing device 100 includes a NW (network) interface 110, a memory circuit 120, an input interface 130, a display 140, and a processing circuit 150.

[0012] The NW interface 110 is connected to the processing circuit 150 and controls the transmission and communication of various data performed between the medical image processing device 100 and the medical image diagnostic device 200. The NW interface 110 is realized by a network card, a network adapter, a NIC (Network Interface Controller), or the like.

[0013] The memory circuit 120 stores various information used by the processing circuit 150 in advance. The memory circuit 120 also stores various programs. The memory circuit 120 is, for example, a storage device such as an HDD (Hard disk Drive), an SSD (Solid State Drive), or an integrated circuit storage device that stores various information. In addition to an HDD or an SSD, etc., the memory circuit 120 may also be a drive device that reads and writes various information to and from a portable storage medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), a flash memory, or a semiconductor memory element such as a RAM (Random Access Memory).

[0014] The input interface 130 is realized by a drawing tablet (a combination of a touch pen and a tablet for receiving operations by the user), a trackball, a switch button, a mouse, a keyboard, a touch pad for performing input operations by touching the operation surface, a touch screen in which the display screen and the touch pad are integrated, a non-contact input circuit using an optical sensor, and an audio input circuit, etc. The input interface 130 may include a plurality of devices for receiving operations by the user. The input interface 130 is connected to the processing circuit 150, converts the input operation received from the user into an electrical signal, and outputs it to the processing circuit 150. Note that in this specification, the input interface is not limited to those equipped with physical operation components such as a mouse and a keyboard. For example, an electrical signal processing circuit that receives an electrical signal corresponding to an input operation from an external input device provided separately from the device and outputs this electrical signal to the processing circuit 150 is also included in the example of the input interface.

[0015] The display 140 displays various types of information under the control of the processing circuit 150. For example, the display 140 outputs a radiography viewer including a medical image generated by the processing circuit 150, a GUI (Graphical User Interface) for receiving various operations from the user, etc. Specifically, the display 140 is a liquid crystal display, a CRT (Cathode Ray Tube) display, etc. Note that the input interface 130 and the display 140 may be integrated. For example, the input interface 130 and the display 140 may be realized by a touch panel. The display 140 is an example of a display unit.

[0016] The processing circuit 150 is a processor that reads and executes a program from the memory circuit 120 to realize the functions corresponding to each program. The processing circuit 150 of the present embodiment includes an acquisition function 151, a reception function 152, an image processing function 153, a recording function 154, a learning function 155, an estimation function 156, and a display control function 157. The acquisition function 151, the image processing function 153, and the recording function 154 are examples of an acquisition unit. Also, the image processing function 153 is also an example of an image processing unit. Also, the recording function 154 is an example of a recording unit. The reception function 152 is an example of a reception unit. The learning function 155 is an example of a learning unit. The estimation function 156 is an example of an estimation unit. The display control function 157 is an example of a display control unit.

[0017] Here, for example, each processing function of the acquisition function 151, the reception function 152, the image processing function 153, the recording function 154, the learning function 155, the estimation function 156, and the display control function 157, which are components of the processing circuit 150, is stored in the memory circuit 120 in the form of a program executable by a computer. The processing circuit 150 is a processor. For example, the processing circuit 150 reads and executes a program from the memory circuit 120 to realize the functions corresponding to each program. In other words, the processing circuit 150 in the state of having read each program will have each function shown in the processing circuit 150 of FIG. 1. Note that in FIG. 1, it has been described that the processing functions performed by the acquisition function 151, the reception function 152, the image processing function 153, the recording function 154, the learning function 155, the estimation function 156, and the display control function 157 are realized by a single processor, but it is also possible to configure the processing circuit 150 by combining a plurality of independent processors, and each processor realizes the function by executing a program. Also, in FIG. 1, it has been described that a single memory circuit 120 stores the programs corresponding to each processing function, but it is also possible to configure the processing circuit 150 to read the corresponding programs from individual memory circuits by dispersing and arranging a plurality of memory circuits.

[0018] In the above description, an example has been described in which the "processor" reads and executes a program corresponding to each function from the storage circuit. However, the embodiment is not limited to this. The term "processor" means, for example, a circuit such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an application specific integrated circuit (ASIC), or a programmable logic device (for example, a simple programmable logic device (SPLD), a complex programmable logic device (CPLD), and a field programmable gate array (FPGA)). When the processor is, for example, a CPU, the processor realizes its function by reading and executing a program stored in the storage circuit. On the other hand, when the processor is an ASIC, instead of storing the program in the storage circuit 120, the function is directly incorporated as a logic circuit in the circuit of the processor. Note that each processor of the present embodiment is not limited to being configured as a single circuit for each processor, and a plurality of independent circuits may be combined to be configured as one processor to realize its function. Further, a plurality of components in FIG. 1 may be integrated into one processor to realize its function.

[0019] (Learning phase) The processing executed by the medical image processing apparatus 100 in the present embodiment includes a learning phase in which learning data is learned by a model and an estimation phase in which inference processing is executed by the learned model. First, the processing of each function in the learning phase will be described.

[0020] The acquisition function 151 acquires medical image data obtained by photographing a subject from the medical image diagnostic apparatus 200 via the network 300 and the NW interface 110. The medical image data is, for example, X-ray CT image data. The medical image data may be, for example, two-dimensional image data or three-dimensional image data (volume data).

[0021] Note that the source of the medical image data acquired by the acquisition function 151 is not limited to the medical image diagnostic apparatus 200, and may be another server device or the storage circuit 120 of the medical image processing apparatus 100. The medical image data acquired by the acquisition function 151 in the learning phase is an example of the first medical image data in the present embodiment. Also, the subject that is the imaging target of the first medical image data is an example of the first subject.

[0022] The reception function 152 receives various operations of the user via the input interface 130. For example, the reception function 152 receives a user operation to correct a mask indicating the range of an anatomical tissue in the medical image data described later, and a user operation to set various parameters in the learning process. The reception function 152 sends the received operation to the recording function 154.

[0023] In the learning phase, the image processing function 153 acquires a first contour representing the region of the anatomical tissue of the subject depicted in the medical image data, and the position of at least one anatomical landmark (ALD: Anatomical Landmark Detection) in the subject in the medical image data.

[0024] For example, the image processing function 153 executes predetermined image processing on the medical image data acquired from the medical image diagnostic apparatus 200. Specifically, the image processing function 153 is an image processing application that executes image processing for extracting a predetermined region included in the medical image. The image processing function 153 acquires, for example, the region of the anatomical tissue of the subject depicted in the medical image data by segmentation by rule-based image recognition processing or the like. The image processing function 153 may acquire the contour of the anatomical tissue based on the difference in pixel values (CT values in the case of X-ray CT image data) between pixels in the medical image data. The image processing function 153 associates the information indicating the extracted range with the medical image data as a mask indicating the range of the anatomical tissue of the subject in the medical image data. The mask indicating the region of the anatomical tissue of the subject set by the image processing function 153 is an example of the first contour in the present embodiment. Further, the mask set by the image processing function 153 is also referred to as an initial contour.

[0025] Anatomical tissues are, for example, organs, blood vessels, bones, and tumors, etc. Further, the anatomical tissue may be a part of an organ rather than the whole organ.

[0026] Note that the method for acquiring the region of the anatomical tissue by the image processing function 153 is not limited to the above example, and known methods can be applied.

[0027] Anatomical landmarks are characteristic local structures in the subject, such as specific locations of organs or bones, etc. The anatomical landmarks extracted by the image processing function 153 only need to be photographed together with the target anatomical tissue on the medical image data, and may be located within the anatomical tissue or outside the anatomical tissue. When there are a plurality of anatomical landmarks on the medical image data, the image processing function 153 may acquire, for example, the anatomical landmark closest to the target anatomical tissue on the medical image data.

[0028] Here, generally, in segmentation processing or the like based on the difference in pixel values, among other tissues adjacent to the target anatomical tissue, those with pixel values similar to the target anatomical tissue may be misrecognized as part of the target anatomical tissue. For example, when the target anatomical tissue is the liver, the image processing function 153 may misrecognize the chest wall and the large intestine wall around the liver as part of the liver. In this case, the mask set by the image processing function 153 extends not only to the liver but also to the surrounding chest wall and large intestine wall. Thus, in this embodiment, a region that is not actually the target anatomical tissue but is included in the mask as the target anatomical tissue is called a false positive (FP) region.

[0029] FIG. 2 is a diagram showing an example of a mask 51 indicating the region of an anatomical tissue set by the image processing function 153 according to the first embodiment. In the medical image data 41 shown in FIG. 2, the liver is photographed as an example of an anatomical tissue.

[0030] In the example shown in FIG. 2, there is an error in the mask 51 set by the image processing function 153. More specifically, the mask 51 set for the medical image data 41 by the image processing function 153 includes a first region 511 that overlaps with the correct data described later and a second region 512 that does not overlap with the correct data.

[0031] Since the correct data shows the contour indicating the area where the liver is actually depicted on the medical image data 41, in other words, the first area 511 is the area where the liver is actually depicted, and the second area 512 is the area where tissues other than the liver are depicted. That is, the first area 511 is detected as the liver by the image processing function 153 and is the area where the liver is actually depicted, so in other words, it is a true positive (TP: True Positive) area. The second area 512 is an area that is included in the mask 51 as the liver even though it is not actually the liver, and such an area is called a false positive area. Also, although not illustrated in FIG. 2, when an area that is actually the liver but is not included in the mask 51 occurs, such an area is a false negative (FN: False Negative) area.

[0032] In FIG. 2, for the sake of explanation, the first area 511 and the second area 512 are displayed separately, but in the process of the image processing function 153, the first area 511 and the second area 512 are not discriminated. Therefore, in the mask 51 generated by the image processing function 153, the first area 511 and the second area 512 are not distinguished.

[0033] Also, in the example shown in FIG. 2, the image processing function 153 acquires the position of the anatomical landmark 60 depicted in the medical image data 41. The anatomical landmark 60 shown in FIG. 2 is the center of the liver. The method for acquiring the position of the anatomical landmark 60 by the image processing function 153 is not particularly limited, and known image processing such as image segmentation may be applied. Note that the image processing function 153 may acquire a plurality of anatomical landmarks 60 from one medical image data 41. Also, the anatomical landmark 60 may be a bone located outside the target anatomical tissue or a part of another anatomical tissue.

[0034] In the present embodiment, the case where the image processing function 153 detects the region of the target anatomical tissue based on the difference in pixel values is described as an example. However, even when the region of the target anatomical tissue is detected by other methods, false positive and false negative regions may occur, resulting in a correction operation by the user. The medical image processing apparatus 100 of the present embodiment performs learning of the correction content in order to streamline the correction of such false positive and false negative regions.

[0035] In the present embodiment, the user manually corrects the mask 51 generated by the image processing function 153 to generate the correct data used for learning. The user of the medical image processing apparatus 100 in the present embodiment is, for example, a radiologist or the like. The correct data is also referred to as an annotation.

[0036] Returning here to FIG. 1, the recording function 154 records the operation of the user who corrects the mask 51 indicating the range of the anatomical tissue of the subject in the medical image data. In the present embodiment, the information indicating the correction content of the mask 51 by the user is referred to as correction information. The correction information includes the locus of the movement of the image editing tool that changes the mask 51 and information regarding the size of the image area that can be changed by one operation of the image editing tool.

[0037] In addition, the mask 51 after correction by the user is the correct data representing the region of the anatomical tissue and is an example of the second contour in the present embodiment. The recording function 154 also records the position of the correction location by the user based on the relative position with the anatomical landmark 60. In other words, the recording function 154 acquires correction information including the recording of the temporal change of the relative position with the anatomical landmark 60 of the portion to be changed in the operation of the user who corrects the first contour to the second contour.

[0038] Here, the recording of the user's correction operation by the recording function 154 will be described with reference to FIGS. 3 and 4.

[0039] FIG. 3 is a diagram showing an example of a correction operation by a user according to the first embodiment. In FIG. 3, a GUI-based editing screen of medical image data 41 displayed on the display 140 by a display control function 157 described later is shown.

[0040] The user can operate the brush 70 on the GUI by operating an input interface 130 such as a tablet or a mouse. In FIG. 3, brushes 70a to 70c are shown at three locations for the sake of explanation, but actually only one brush 70 can be operated by the user at a time. The brush 70 is an example of an image editing tool capable of editing the mask 51. Further, when the medical image data 41 is three-dimensional image data, the brush 70 is spherical as an example, and the size of the sphere can be changed by the user's operation. The size of the brush 70 is an example of the size of an image area that can be changed by one operation of the image editing tool. Note that the image editing tool is not limited to the brush 70, and may be, for example, a pointer capable of selecting a correction range.

[0041] It is assumed that the brush 70 can erase (eraser function) and add the mask 51. For example, the user can erase the second region 512 in the range through which the brush 70 passes by moving the brush 70 on the second region 512. In the example shown in FIG. 3, the user moves the brush 70 to erase the second region 512, such as the trajectories 71a to 71c of the brush 70 indicated by the broken lines, to correct the mask 51.

[0042] As described above, in the mask 51 generated by the image processing function 153, the first region 511 and the second region 512 are not distinguished. Therefore, in the correction operation, the user determines the first region 511 and the second region 512 based on his or her own judgment. Then, the user erases, at his or her own discretion, the portion that protrudes from the region of the mask 51 where it is considered that areas other than the liver are actually depicted, that is, the second region 512. In addition, when the user determines that there is a range in the region where the liver is considered to be actually depicted and is not included in the mask 51, the user performs a correction to add the mask 51 so as to include the range.

[0043] In the above-described image processing function 153, for example, the area where the liver is depicted is extracted based on the difference in pixel values. In contrast, in the correction of the mask 51 by the user, the area where the liver is depicted is determined not only by pixel values but also by the user's anatomical knowledge.

[0044] The recording function 154 records the corrected portions of the mask in the operation of the user for correcting the mask 51 in chronological order based on the relative positions with respect to the anatomical landmarks 60. Specifically, the recording function 154 records the distances in the X direction, Y direction, and Z direction between the brush 70 and the anatomical landmark 60 in association with the time. When the user moves the brush 70 in any of the X direction, Y direction, and Z direction, the distances in the X direction, Y direction, and Z direction between the brush 70 and the anatomical landmark 60 change over time. Such a history of the distances in the X direction, Y direction, and Z direction between the brush 70 and the anatomical landmark 60 becomes the record of the movement trajectory of the brush 70. The distances in the X direction, Y direction, and Z direction of the brush 70 from the anatomical landmark 60 are the coordinates of the relative position of the brush 70 with respect to the anatomical landmark 60.

[0045] In the present embodiment, the recording function 154 records values obtained by converting the distances in the X direction, Y direction, and Z direction on the medical image data 41 into distances in the real space. Although known techniques can be applied as the conversion method, for example, the acquisition function 151 may acquire the correspondence relationship between the pixels in the medical image diagnostic apparatus 200 and the sizes in the real space from the medical image diagnostic apparatus 200.

[0046] Also, since the size of the brush 70 can be changed by the user as described above, the recording function 154 records the size of the brush 70 in association with the time. Thereby, the recording function 154 can acquire the chronological record of the size of the brush 70.

[0047] In addition, when the medical image data 41 is two-dimensional image data, since there is no distance (depth) in the Z direction, the recording function 154 records the distance in the X direction and the distance in the Y direction between the brush 70 and the anatomical landmark 60 in association with the time. Also, when the medical image data 41 is two-dimensional image data, the shape of the brush 70 also becomes a planar circle. Note that the image editing tool is not limited to the brush 70, and the shape of the image area that can be changed by one operation of the image editing tool is not limited to a spherical or circular shape either.

[0048] FIG. 4 is a diagram showing an example of the correction information 700 recorded by the recording function 154 according to the first embodiment. The recording function 154 records, in association with the time, information for specifying the anatomical landmark 60, the relative position of the brush 70 with respect to the anatomical landmark 60, and the size of the brush 70 as the correction information 700. The recording function 154 stores the recorded correction information 700, for example, in the storage circuit 120.

[0049] The information for specifying the anatomical landmark 60 is, for example, as shown in FIG. 4, the name of the anatomical landmark 60. Also, the size of the brush 70 is represented by, for example, as shown in FIG. 4, the length of the radius of the brush 70. The length of the radius of the brush 70 may be a value converted into a length in the real space or the number of pixels on the medical image data 41.

[0050] In FIG. 4, as an example, the recording function 154 records the trajectory of the brush 70 and the history of the size of the brush 70 at one-second intervals, but the recording frequency is not limited to this.

[0051] Also, in the example shown in FIG. 4, there is one anatomical landmark 60, but the recording function 154 may record, in association with each other, information for specifying a plurality of anatomical landmarks 60 and the relative position of the brush 70 with respect to each of the plurality of anatomical landmarks 60 as the correction information 700.

[0052] Returning to FIG. 1, the learning function 155 learns the difference between the mask 51 before correction and the mask 51 after correction by the user based on the relative positions of the mask 51 before correction and the mask 51 after correction with respect to the anatomical landmark 60. More specifically, the learning function 155 of the present embodiment learns the relative position of the mask 51 before correction with respect to the anatomical landmark 60 and the correction information 700 recorded by the recording function 154.

[0053] FIG. 5 is a diagram showing an example of the learning process according to the first embodiment. The learning function 155 executes a learning process by inputting learning data into the model 80. The learning data in the present embodiment includes the mask 51 before correction, the correction information 700 in which the operations of the user for correcting the mask 51 are recorded, and the mask 51 after correction. The mask 51 after correction does not include the second region 512 and includes only the first region 511. Also, the mask 51 after correction is the ground truth in the learning process by the learning function 155. Since the mask 51 after correction can be derived from the mask 51 before correction and the correction information 700, the mask 51 before correction and the correction information 700 may be the learning data.

[0054] The learning function 155 inputs, as learning data into the model 80, the shape of the mask 51 before correction, for example, the X, Y, and Z coordinates representing the relative positions of each of the plurality of pixels constituting the outer contour of the mask 51 with respect to the anatomical landmark 60. Alternatively, if the model 80 can learn image data, the learning function 155 may input the medical image data 41 including the mask 51 before correction and the anatomical landmark 60 as image data.

[0055] Also, since the correction information 700 includes a time-series record of the change in the position of the brush 70, the learning function 155 may further use the speed or acceleration of the movement of the brush 70 as learning data.

[0056] Model 80 is, for example, a model of reinforcement learning deep learning (deep learning) or other machine learning. More specifically, the model 80 of the present embodiment is a model that performs reinforcement learning based on learning data and rewards. Assume that the model 80 is stored, for example, in the memory circuit 120. The learning function 155 reads the model 80 from the memory circuit 120 and inputs the learning data. Alternatively, the model 80 may be incorporated into the learning function 155.

[0057] In the example shown in FIG. 5, the user performed a correction operation to erase a part of the mask 51 by moving the brush 70 as shown by the locus 71. As described above, since the user corrects the mask 51 based on anatomical findings, there may be no difference in pixel values between the region where the mask 51 was erased by the user and the region where it was not erased in the medical image data 41. In the example shown in FIG. 5, there is no difference in the pixel values of the pixels 501a to 501i at the boundary between the region where the mask 51 was erased and the region where it was not erased. However, according to the method of the present embodiment, the correction position can be learned based on the anatomical landmark 60 regardless of the presence or absence of a difference in pixel values.

[0058] Although FIG. 5 illustrates learning for the correction of one medical image data 41 as an example, the learning function 155 learns the mask 51 before correction of a plurality of medical image data 41, the mask 51 correction operation for each of the plurality of medical image data 41 by the user, and the mask 51 after correction.

[0059] In addition, the learning function 155 of the present embodiment learns a method for efficiently correcting the mask 51 by executing reinforcement learning based on learning data and rewards. The following formula (1) is an example of a formula that defines the reward in the reinforcement learning by the learning function 155.

[0060]

Equation

[0061] The recall used in Equation (1) is defined by the following Equation (2).

[0062] [Number]

[0063] FIG. 6 is a diagram for explaining Equation (1) and Equation (2) according to the first embodiment.

[0064] In Equation (1), r is the length of the radius of the sphere or circle of the brush 70, indicating the size of the brush 70. The larger the brush 70, the wider the editable range in one operation, thus improving the efficiency of the correction work. Since the reward increases as r increases, the learning function 155 learns the correction information such that the higher the reward, the larger the image area editable by the brush 70 in one operation.

[0065] In Equation (1), A represents the second region 512, B represents the first region 511, and C represents the brush 70. Also, S B represents the outline 5111 of the first region 511, and S C represents the outline 701 of the brush 70. S B ∩S C in Equation (1) represents the size of the overlapping range between the outline 5111 of the first region 511 and the outline 701 of the brush 70. In other words, S B ∩S C represents how much the outline 701 of the brush 70 follows the outline 5111 of the first region 511. When the medical image data 41 and the brush 70 are three-dimensional, S B represents the surface of the three-dimensional first region 511, and S C represents the surface of, for example, a spherical brush 70.

[0066] The recall A,C in Equation (1) represents the ratio of the range erased by the brush 70 in the entire second region 512. The recall B,C in Equation (1) represents the ratio of the range that remains un-erased out of the range that was erroneously erased by the brush 70 in the first region 511 and the range that remains un-erased.

[0067] The TP (True Positive) in Equation (2) is the specified area, that is, recall A,C In this case, the second area 512 that needs to be modified is the recall B,C In this case, the first area 511 which is the ground truth is shown.

[0068] The FN (False Negative) in Equation (2) indicates the range that is not processed as the specified area even though it is the specified area. Specifically, in recall A,C In this case, although FN is the second area 512, since the trajectory of the brush 70 does not pass through it, it indicates the size of the range where the correction is missed. Also, in recall B,C In this case, although FN is the first area 511, since the trajectory of the brush 70 passes through it, it indicates the size of the range that is erroneously erased.

[0069] “recall A,C -recall B,C ” The value becomes larger as the omission of the second area 512 that needs to be corrected is less and the erroneous erasure of the first area 511 is less. As the value of “recall A,C -recall B,C ” increases, the value of the reward in Equation (1) also increases, and the reward in reinforcement learning becomes higher.

[0070] The β 1 , β 2 , β 3 are hyperparameters that can be specified by the user. The user can adjust the size of the brush 70, recall 1 , and the weighting of S 2 , β 3 by changing the values of β B,C , and S B ∩ S C . For example, β 1The larger it becomes, the larger the size of the brush 70, that is, the learning result that emphasizes correction with a small number of operations. Also, β 2 The larger it becomes, the learning result that emphasizes reducing the misdeletion of the first region 511. Also, β 3 The larger it becomes, the learning result that emphasizes that the brush 70 follows the outline of the first region 511. Note that β 1 β 2 β 3 β may be automatically determined by learning by the model 80.

[0071] The learning function 155 performs reinforcement learning so as to maximize the reward, and then stores the learned model 80 in the memory circuit 120. Hereinafter, the model 80 learned by the learning function 155 is referred to as a learned model. The learned model learned by the learning function 155 is an example of the first learned model in the present embodiment.

[0072] Also, the learning function 155 may generate a plurality of learned models learned with different learning data for each target anatomical tissue, and store them in the memory circuit 120 in association with identification information capable of identifying the target anatomical tissue. The identification information capable of identifying the target anatomical tissue is, for example, the name of the anatomical tissue. Specifically, the learning function 155 may generate learned models for each anatomical tissue such as a learned model for the liver, a learned model for the pancreas, and a learned model for the heart. Alternatively, the learning function 155 may generate different learned models for each imaging range even for one anatomical tissue.

[0073] (Inference phase) Next, the processing of each function in the inference phase using the learned model will be described.

[0074] FIG. 7 is a diagram showing an example of an outline of processing in the inference phase according to the first embodiment.

[0075] The acquisition function 151 acquires medical image data 42 obtained by photographing a subject from the medical imaging device 200 via the network 300 and the NW interface 110, similar to the learning phase.

[0076] The medical image data 42 acquired by the acquisition function 151 in the inference phase is an example of the second medical image data in this embodiment. Also, the subject that is the imaging target of the medical image data 42 is an example of the second subject. The first subject and the second subject may be the same, but generally they are different. For example, a plurality of medical image data 41 obtained by imaging a plurality of first subjects are used for learning by the learning function 155 to generate a learned model. Then, the subject to be subjected to image diagnosis using the generated learned model is the second subject.

[0077] The image processing function 153 acquires, similar to the learning phase, for example, a mask 51 representing the region of the anatomical tissue of the subject depicted in the medical image data 42 for image diagnosis and the position of at least one anatomical landmark 60 in the subject in the medical image data 42 by segmentation or the like by rule-based image recognition processing. The mask 51 representing the region of the anatomical tissue of the subject depicted in the medical image data 42 for image diagnosis is an example of the third contour representing the outline of the anatomical tissue in this embodiment.

[0078] The estimation function 156 estimates the portion of the mask 51 to be corrected using the learned model learned by the learning function 155. Also, the estimation function 156 estimates the portion of the mask 51 to be corrected at a relative position based on the position of the anatomical landmark 60 in the medical image data 42 for image diagnosis.

[0079] More specifically, the estimation function 156 estimates, by means of the learned model 81, the position where the brush 70 is placed on the medical image data 42 for image diagnosis in order to correct the part of the mask 51 to be corrected. The position where the brush 70 is estimated to be placed corresponds to the part that requires correction of the mask 51. Also, the estimation function 156 estimates the position where the brush 70 is placed as a relative position based on the position of the anatomical landmark 60.

[0080] Since the estimation function 156 estimates the part to be corrected based on the anatomical structure according to the position of the anatomical landmark 60, it is possible to identify the correction part without depending on the pixel value. For example, in the example shown in FIG. 7, there is no difference in the pixel values of the pixels 502a to 502i at the boundary between the area that requires correction and the area that does not require correction in the mask 51. However, according to the method of this embodiment, the correction position can be estimated based on the anatomical landmark 60 regardless of the presence or absence of a difference in pixel values.

[0081] Also, the estimation function 156 selects, from among the plurality of learned models 81a to 81c stored in the memory circuit 120, the learned model 81a that has learned the medical image data 41 of the same anatomical tissue as the anatomical tissue to be imaged in the medical image data 42 for image diagnosis. In the example shown in FIG. 7, the medical image data 42 is image data of the liver, and among the plurality of learned models 81a to 81c, the learned model 81a is a model in which learning processing has been performed based on the medical image data 41 of the liver. Hereinafter, when the individual learned models 81a to 81c are not particularly distinguished, they are simply referred to as the learned model 81. Note that it is not essential to divide the learned model 81 into a plurality, and one learned model 81 may learn the medical image data 41 of a plurality of anatomical tissues.

[0082] In this embodiment, when the learned model 81 receives the input of the medical image data 42 for image diagnosis with the mask 51 attached and the positions of the anatomical landmarks 60 on the medical image data 42, it outputs the portions of the mask 51 to be corrected. The portions to be corrected are false positive regions that are included in the mask 51 as the target anatomical tissue even though they are not actually the target anatomical tissue and thus require correction. Also, the learned model 81 estimates not only "whether correction is required" but also the degree of necessity for correction for the portions to be corrected. For example, the learned model 81 estimates that the degree of necessity for correction is high for the portions where the reward increases when the brush 70 is placed during reinforcement learning. This is because the positions of the brush 70 where the reward increases in reinforcement learning are the positions of the brush 70 where the efficiency of the correction operation is high. In this embodiment, the estimation function 156 does not automatically correct the mask 51 but sends the estimation results of the portions to be corrected to the display control function 157.

[0083] The display control function 157 causes a reading viewer and a GUI for receiving various operations from the user to be displayed on the display 140.

[0084] Also, the display control function 157 causes the portions of the mask 51 of the medical image data 42 for image diagnosis that are estimated to have a high degree of necessity for correction and the portions that are estimated to have a low degree of necessity for correction to be displayed on the GUI of the display 140 in a distinguishable manner. The display control function 157 causes the portions that are estimated to have a higher degree of necessity for correction to be displayed as the degree of estimation that the brush 70 will be placed by the estimation function 156 is higher. This display can guide the user to the portions of the mask 51 to be corrected. For example, through this display, the user can grasp the operation positions of the brush 70 at which the portions of the mask 51 to be corrected can be efficiently corrected. In the example shown in FIG. 7, the user can erase the false positive regions that are included in the mask 51 as the liver even though they are not actually the liver by operating the brush 70 along the trajectory 71 according to this display.

[0085] FIG. 8 is a diagram showing an example of a display mode of a portion to be corrected according to the first embodiment. In the example shown in FIG. 8, the display control function 157 displays, on the medical image data 42, the mask 51 before correction and a heat map indicating the degree of correction required for the mask 51. Note that the display by the heat map is an example of the display mode of the portion to be corrected, and the display control function 157 may display an image display for guiding the brush 70 or a message. Note that the display control function 157 may display by filling the area of the mask 51, or may display as a contour line.

[0086] The reception function 152 receives various operations of the user via the input interface 130 in the same manner as in the learning phase. For example, the reception function 152 receives an operation of the user for correcting the mask 51 of the medical image data 42 for image diagnosis according to the heat map display. The diagnostic medical image data 42 in which the mask 51 is corrected by the user is stored in, for example, the storage circuit 120.

[0087] Note that the user in the learning phase and the user in the inference phase may be the same or different. When the user in the learning phase and the user in the inference phase are the same, the user can utilize the record of his / her own mask 51 correction operation in the past as learning data, and in the subsequent image diagnosis work, be assisted in the correction work by the heat map based on the inference result by the learned model 81. Also, when the user in the learning phase and the user in the inference phase are different, for example, the learned model 81 that has learned the learning data based on the correction operations by a plurality of users can be utilized to assist the correction work of other users.

[0088] Next, the flow of the process executed by the medical image processing apparatus 100 of the present embodiment configured as described above will be described.

[0089] FIG. 9 is a flowchart showing an example of the overall flow of the process executed by the medical image processing apparatus 100 according to the first embodiment.

[0090] Among the flowcharts of FIG. 9, the processes of steps S1 to S6 are processes in the learning phase. Also, the processes of steps S7 to S11 are processes in the inference phase. Note that in FIG. 9, for the sake of showing the entire flow of the process, the processes of the learning phase and the inference phase are described continuously, but the learning phase and the inference phase do not necessarily need to be carried out continuously.

[0091] First, the acquisition function 151 acquires medical image data 41 for learning (S1).

[0092] Then, the image processing function 153 acquires, by image processing, a mask 51 representing the region of the anatomical tissue of the subject depicted in the medical image data 41 for learning, that is, an initial contour (S2).

[0093] Also, the image processing function 153 acquires, by image processing, the position of the anatomical landmark 60 on the medical image data 41 (S3).

[0094] Then, the recording function 154 records the user's correction operation on the mask 51 of the medical image data 41 (S4). Specifically, the recording function 154 generates correction information 700 by recording the relative position of the brush 70 with respect to the anatomical landmark 60 and the size of the brush 70 in association with time.

[0095] Then, the learning function 155 executes reinforcement learning of the model 80 using, as learning data, the relative position of the mask 51 before correction with respect to the anatomical landmark 60 and the correction information 700 (S5). Through the learning by the learning function 155, a learned model 81 is generated.

[0096] If the learning is not completed (S6 “No”), the process returns to S1, and the processes of steps S1 to S5 in the learning phase based on new medical image data 41 for learning are executed. The completion of learning may be determined, for example, by the end condition of the reinforcement learning, or may be when the learning process based on a predetermined number of pieces of learning data is completed. Alternatively, the completion of learning may be determined by the user.

[0097] Note that the order of processing in the learning phase is not limited to the example shown in FIG. 9. For example, after the processes of S1 to S4 are executed multiple times to generate a plurality of pieces of correction information 700, the learning function 155 may learn the plurality of pieces of correction information 700 in the learning process of S5. Further, for each target anatomical tissue, the processes of S1 to S5 may be executed multiple times to generate a plurality of learned models 81 for each anatomical tissue.

[0098] After the learning is completed (S6 “Yes”), the acquisition function 151 acquires medical image data 42 for diagnosis (S7).

[0099] Then, the image processing function 153 acquires, by image processing, a mask 51 representing the region of the anatomical tissue of the subject depicted in the medical image data 42 for diagnosis, that is, an initial contour (S8).

[0100] Further, the image processing function 153 acquires the position of the anatomical landmark 60 on the medical image data 42 (S9).

[0101] Then, the estimation function 156 executes an estimation process for the correction target portion of the mask 51 (S10). Specifically, the estimation function 156 inputs the medical image data 42 for image diagnosis with the mask 51 applied to the learned model 81 and the position of the anatomical landmark 60 on the medical image data 42, and thereby acquires the correction target portion of the mask 51 output from the learned model 81 and the degree of necessity of correction within the correction portion.

[0102] The display control function 157 executes an editing assist process for the mask 51 based on the result of the estimation process (S11). More specifically, the display control function 157 supports the correction of the mask 51 by the user by displaying, on the medical image data 42, the degree of necessity of correction for the mask 51 as a heat map. Here, the processing of this flowchart ends.

[0103] As described above, the medical image processing apparatus 100 of the present embodiment acquires a pre-modification mask 51 representing a region of an anatomical tissue of a subject depicted in medical image data 41, a post-modification mask 51, and the positions of at least one anatomical landmark 60 in the subject in the medical image data 41, and learns the difference between the pre- and post-modification masks 51 based on the relative positions of each of the pre- and post-modification masks 51 with respect to the anatomical landmark 60. Therefore, according to the medical image processing apparatus 100 of the present embodiment, regardless of the difference in pixel values, the modified portions of the mask 51 can be learned, so that the workload when a user manually corrects portions that cannot be distinguished by automatic processing based on pixel values based on anatomical knowledge can be reduced.

[0104] Further, the medical image processing apparatus 100 of the present embodiment learns correction information 700 including a record of the temporal change in the relative position with respect to the anatomical landmark 60 of the portion to be changed in the operation of the user who corrects the mask 51, thereby learning a correction operation for efficiently correcting the mask 51.

[0105] Further, the medical image processing apparatus 100 of the present embodiment learns the correction information 700 such that the reward increases as the mask 51 is corrected so that there is less omission of deletion of the second region 512, which is a portion of the mask 51 that needs to be corrected, and less incorrect deletion of the first region 511 that should be included in the mask 51. Therefore, according to the medical image processing apparatus 100 of the present embodiment, it is possible to learn so as to improve the accuracy of correcting the mask 51.

[0106] Further, the correction information 700 of the present embodiment includes information regarding the movement trajectory of the brush 70 that changes the mask 51 and the size of the brush 70, and the medical image processing apparatus 100 of the present embodiment learns the correction information such that the reward increases as the size of the brush 70 increases. Therefore, according to the medical image processing apparatus 100 of the present embodiment, it is possible to learn the setting of the size of the brush 70 and the movement route that can correct the mask 51 with a small number of operations.

[0107] In addition, the medical image processing apparatus 100 of the present embodiment causes the display 140 to display in a distinguishable manner the portions of the mask 51 of the medical image data 42 for diagnosis where the degree of being estimated to require correction is high and low. Therefore, according to the medical image processing apparatus 100 of the present embodiment, the user can visually grasp where to place the brush 70, and the load of correcting the mask 51 can be reduced.

[0108] (Second Embodiment) In the above-described first embodiment, the medical image processing apparatus 100 assisted the user in correcting the mask 51 by presenting the portions to be corrected by the heat map. In this second embodiment, the medical image processing apparatus 100 assists the user in correcting the mask 51 by force feedback.

[0109] FIG. 10 is a diagram showing an example of the overall configuration of a medical image processing system S according to the second embodiment. Similar to the first embodiment, the medical image processing system S includes, as an example, a medical image processing apparatus 100 and a medical image diagnostic apparatus 200. Further, the medical image processing apparatus 100 includes, as in the first embodiment, an NW interface 110, a storage circuit 120, an input interface 130, a display 140, and a processing circuit 150.

[0110] The processing circuit 150 of the present embodiment includes an acquisition function 151, a reception function 152, an image processing function 153, a recording function 154, a learning function 155, an estimation function 156, a display control function 157, and a force feedback function 158. The force feedback function 158 is an example of an operation control unit.

[0111] The acquisition function 151, the reception function 152, the image processing function 153, the recording function 154, the learning function 155, the estimation function 156, and the display control function 157 have the same functions as those in the first embodiment.

[0112] Note that the display control function 157 of this embodiment may perform heat map display as in the first embodiment, or may simply display the medical image data 42 for image diagnosis and the mask 51 on the GUI without performing heat map display.

[0113] The force feedback function 158 of this embodiment controls the operation of the brush 70 on the medical image data 42 for image diagnosis so as to facilitate the approach of the brush 70 to the position estimated by the estimation function 156 to place the brush 70 for correcting the mask 51 in the inference phase.

[0114] More specifically, the force feedback function 158 makes it easy for the user to move the brush 70 toward the position estimated by the estimation function 156 to place the brush 70 for correcting the mask 51, and makes it difficult for the user to move the brush 70 away from the estimated position, thereby controlling the movement of the brush 70 on the GUI.

[0115] In the absence of control by the force feedback function 158, the brush 70 on the GUI moves in conjunction with the operation of the input interface 130 such as a tablet or a mouse by the user. In contrast, when the user moves the brush 70 away from the position estimated by the estimation function 156 to place the brush 70 for correcting the mask 51, the force feedback function 158 reduces the amount of movement of the position of the brush 70 on the GUI with respect to the amount of movement of the input interface 130 by the user. Also, when the user moves the brush 70 closer to the position estimated by the estimation function 156 to place the brush 70 for correcting the mask 51, the force feedback function 158 increases the amount of movement of the position of the brush 70 on the GUI with respect to the amount of movement of the input interface 130 by the user. By such control, the force feedback function 158 controls the moving direction of the brush 70 by the user.

[0116] FIG. 11 is a diagram for explaining an example of force feedback according to the second embodiment. The arrows 75a to 75c shown in FIG. 11 indicate the directions for prompting the movement of the brush 70 on the GUI by the force feedback function 158. For example, when the user operates the brush 70a, the force feedback function 158 suppresses the movement of the brush 70a in the direction away from the mask 51, and prompts the brush 70a to move in the direction along the locus 71a, that is, in the direction in which the second region 512 can be erased from the efficiency history.

[0117] Note that the arrows 75a to 75c shown in FIG. 11 are illustrated to indicate the direction of the force, and may not actually be displayed on the screen. Alternatively, the display control function 157 may present the position where the brush 70 should be placed to the user by displaying the arrows 75a to 75c on the GUI. Note that the image for presenting the position where the brush 70 should be placed to the user is not limited to the arrow. For example, the display control function 157 may display a figure having the same shape as the shape of the icon of the brush 70 in a broken line or a lighter color than the icon of the brush 70.

[0118] FIG. 12 is a diagram showing an example of the magnitude of the force (F) in the force feedback according to the second embodiment. Further, the following formula (3) is an example of a mathematical formula representing the magnitude of the force (F) in the force feedback according to the second embodiment.

[0119]

Number

[0120] The point a in formula (3) and FIG. 12 is the local maximum point where the degree of placement of the brush 70 is estimated to be the highest in the range estimated by the estimation function 156 for correcting the mask 51. Further, the point b in formula (3) and FIG. 12 is the position of the center of the brush 70 placed on the medical image data 42. The estimation function 156 specifies the positions of the local maximum point a and the point b as relative positions based on the anatomical landmark 60.

[0121] As shown in Equation (3), the force (F) in force feedback increases as the distance from the maximum point a increases. Also, the force (F) in force feedback increases in the direction toward the maximum point a. For this reason, the user feels a force that pulls the brush 70 toward the maximum point a, in other words, a force that repels from the portion where the brush 70 should not be placed. Note that the force feedback in the present embodiment is feedback that virtually makes the user feel a force by controlling the movement of the brush 70 on the GUI, and does not physically apply pressure or the like to the user. For this reason, the force feedback in the present embodiment can be applied to a general input interface 130 such as a tablet or a mouse.

[0122] Note that in FIG. 12, the medical image data 42 is displayed two-dimensionally. However, when the medical image data 42 is three-dimensional image data, the force of the force feedback is also controlled three-dimensionally.

[0123] FIG. 13 is a diagram showing an example of a GUI 141 in which a mask 51 according to the second embodiment can be modified. The GUI 141 shown in FIG. 13 is displayed on the display 140 by the display control function 157.

[0124] In the right half of the GUI 141, medical image data 42 provided with the mask 51 is displayed. Also, an icon indicating a brush 70 for modifying the mask 51 is displayed on the medical image data 42.

[0125] In the left half of the GUI 141, various settings regarding annotation assist are displayed so that they can be changed. Specifically, the GUI 141 includes a check box 141a indicating on / off of force feedback, a display column 141b for the value of tunable parameters, a display column 141c for the calculation formula of reward, an UNDO button 141d, and a REDO button 141e.

[0126] By operating the checkbox 141a, the user can select whether to use force feedback. Also, since the checkbox 141a is displayed on the GUI 141, it is easy for the user to grasp whether the force feedback is effective when performing an operation.

[0127] The display column 141b for the values of adjustable parameters and the display column 141c for the reward calculation formula are columns in which the hyperparameters and the reward calculation formula used in the reinforcement learning of the learned model 81 can be displayed or changed. Also, the display column 141b may be provided with checkboxes that allow the user to select the parameters to be adopted. Also, in the display column 141b, it may be possible to add new parameters by the user. The user can adjust the estimation result by the learned model 81 by operating the parameters and formulas with the GUI 141.

[0128] The UNDO button 141d and the REDO button 141e are image buttons for the user to perform UNDO operations and REDO operations in the correction of the mask 51. Note that the UNDO button 141d and the REDO button 141e not only simply cancel or revert the operations by the user, but may also have a function of performing UNDO or REDO while estimating and correcting the areas that would have been erroneously corrected by the user with the assistance of AI (Artificial Intelligence). For example, the learned model 81 may be used for the assistance. The AI assistance may also be used for operations other than the UNDO operation and the REDO operation. Also, the GUI 141 may have a dropdown or a scale bar for adjusting the strength of the AI assistance. Also, each time the user operates the UNDO button 141d or the REDO button 141e, the learning function 155 may perform additional learning of the learned model 81 to adjust the parameters.

[0129] The user can refer to and modify these settings on the GUI141. Additionally, if necessary, non-display or modification restrictions for some or all of the settings may be imposed by the administrator of the medical image processing system S or the like.

[0130] Further, the display control function 157 displays, in the area 141f below the medical image data 42 on the right side of the GUI141, the name of the target anatomical tissue, the magnitude and direction of the force feedback, etc. Note that the display control function 157 may display which anatomical location the assist corresponding to is in operation. The anatomical location is, for example, the anatomical landmark 60 used as a reference for the position of the correction site, the position of the abnormal region such as a disease (Patient position), etc. Also, the display control function 157 may display the correction purpose such as "modifying the outer edge of the S3 region of the liver".

[0131] Further, as shown in FIG. 13, the display control function 157 may display an arrow 76 indicating the moving direction of the brush 70. The arrow 76 indicates the vector from the current position of the brush 70 to the next position estimated by the learned model 81. Although the medical image data 42 is displayed two-dimensionally in FIG. 13, when the medical image data 42 is three-dimensional image data, the arrow 76 shall also indicate the direction three-dimensionally. Also, the display control function 157 is not limited to the example shown in FIG. 13, and may visually display the direction and magnitude of the force feedback by an image or a numerical value.

[0132] Note that the functions and screen layout of the GUI141 shown in FIG. 13 are an example and are not limited thereto. For example, the GUI141 may further have a setting field for changing the size and shape of the brush 70. Also, by the AI assist, the size and shape of the brush 70 may be proposed or automatically changed according to the shape of the correction site.

[0133] As described above, the medical image processing apparatus 100 according to the present embodiment controls the operation of the brush 70 on the medical image data 42 for image diagnosis so that the brush 70 approaches a position estimated by the estimation function 156 to be a position where the brush 70 is placed for correcting the mask 51. Therefore, according to the medical image processing apparatus 100 in the present embodiment, in addition to the effects of the first embodiment, it is further possible to assist the user in correcting the mask 51.

[0134] (Modification Example 1) In each of the above-described embodiments, the learning phase process and the estimation phase process are executed by one medical image processing apparatus 100, but these processes may be performed by different medical image processing apparatuses 100. Also, the individual processes included in the learning phase and the estimation phase may be executed by different medical image processing apparatuses 100, respectively. For example, when a user performs a correction operation on the mask 51 on a certain information processing apparatus, the information processing apparatus may perform a recording process of the correction operation, and the medical image processing apparatus 100 may execute learning using the correction information 700 in which the correction operation is recorded. In this case, the acquisition function 151 of the medical image processing apparatus 100 acquires, from the above information processing apparatus, the medical image data 41 in which the mask 51 is set by the information processing apparatus, the correction information 700, and the mask 51 corrected by the user.

[0135] (Modification Example 2) In each of the above-described embodiments, the initial contour is the mask 51 generated by image segmentation or the like by the image processing function 153, but the initial contour may be generated by a learned model. The learned model is an example of the second learned model in the present modification example.

[0136] Also, in each of the above-described embodiments, the recording of the correction operation in which the user corrects the initial contour is used as learning data, but the acquisition of the recording of the correction operation is not essential.

[0137] For example, when adopting medical image data, which is public data, as the training medical image data 41, the learning function 155 may learn the difference between the initial contour of the anatomical tissue inferred by the trained model trained with the public data and the contour of the anatomical tissue that is the ground truth of the public data. In this case, the initial contour is the estimation result of the outer contour of the region of the anatomical tissue depicted in the public data by the trained model.

[0138] The public data is an example of the first medical image data in this modification. Also, the initial contour of the anatomical tissue inferred by the trained model trained with the public data is an example of the first contour in this modification. The contour of the anatomical tissue that is the ground truth of the public data is an example of the second contour in this modification. In this modification example, the acquisition function 151 of the medical image processing apparatus 100 may acquire, from an external device, the initial contour of the anatomical tissue inferred by the trained model trained with the public data and the contour of the anatomical tissue that is the ground truth of the public data. Alternatively, the learning function 155 of the medical image processing apparatus 100 may have a function of performing learning processing using the public data. Also, the estimation function 156 may have a function of estimating the initial contour of the anatomical tissue by the trained model trained with the public data.

[0139] (Modification Example 3) In each of the above-described embodiments, the pre-modification mask 51, the modification information 700 in which the operations of the user who modifies the mask 51 are recorded, and the post-modification mask 51 are taken as an example of the training data, but the training data is not limited to these.

[0140] For example, the learning function 155 may use the eye movement of the user when modifying the mask 51 as the training data. The eye movement can be acquired, for example, by tracking the direction of the line of sight with a camera that photographs the user's face.

[0141] In addition, the learning function 155 may learn the individual differences in the correction operations. In this case, the learning data includes, for example, identification information such as an ID (identification) that can identify the user. Further, the learning function 155 may learn the differences in the correction operations for groups such as hospitals to which the user belongs or for each country. By learning the information of the individual user or the user's affiliation in this way, the learning function 155 can learn how to move the brush 70 for each individual user or the user's affiliation, and the criteria for determining the range in which the anatomical tissue is depicted on the medical image data 41, etc.

[0142] In addition, the learning function 155 may use the detection results by a plurality of modalities as learning data. For example, when the target medical image data 41 is an image taken by an X-ray CT device, the learning function 155 may further use the inspection results such as the hardness of the anatomical tissue detected by an ultrasonic diagnostic device as learning data.

[0143] In addition, in each of the above-described embodiments, reinforcement learning is exemplified as the learning method, but the learning function 155 may execute the learning process by other methods such as supervised learning.

[0144] (Modification Example 4) As another example of the initial contour, a contour manually created by the user on the GUI may be used as the initial contour.

[0145] (Modification Example 5) In each of the above-described embodiments, it is assumed that the image processing function 153 acquires the anatomical landmark 60 by image processing or the like, but the anatomical landmark 60 does not have to be automatically extracted and may be specified by the user. Alternatively, the acquisition function 151 may acquire information indicating the position of the anatomical landmark 60 from the medical image diagnostic apparatus 200 or other external devices together with the medical image data 41, 42.

[0146] (Modification Example 6) In each of the above embodiments, the estimation function 156 assisted the user in the correction work instead of automatically correcting the mask 51, but automatic correction may be performed. In this case, in the inference phase, as post-processing of the segmentation by the image processing function 153, automatic correction of the mask 51 by the estimation function 156 is executed. Then, the display control function 157 causes the corrected mask 51 to be displayed on the display 140. Note that the display control function 157 may cause the display 140 to display the mask 51 before and after correction so that they can be compared.

[0147] (Modification Example 7) Also, in the above embodiment, the recording function 154 records values obtained by converting the distances in the X, Y, and Z directions on the medical image data 41 into distances in the real space. However, instead of the converted values, the length or the number of pixels of the distances on the medical image data 41 may be recorded.

[0148] (Modification Example 8) In the above embodiment, the medical image processing apparatus 100 acquired the medical image data 41 and 42 from the medical image diagnostic apparatus 200, but the acquisition sources of the medical image data 41 and 42 are not limited to this. The medical image processing apparatus 100 may acquire the medical images taken by the medical image diagnostic apparatus 200 from a medical image storage apparatus that stores the medical images. The medical image storage apparatus is, for example, a server apparatus of a PACS (Picture Archiving and Communication System) and stores the medical image data in a format compliant with DICOM (Digital Imaging and Communications in Medicine).

[0149] (Modification Example 9) In each of the above embodiments, the case where the medical image data 41 and 42 are the targets has been described as an example, but the functions of the medical image processing apparatus 100 may be applied to other two-dimensional or three-dimensional masks other than the medical image data 41 and 42. Specifically, the functions of the medical image processing apparatus 100 can be applied to the correction of masks indicating the outlines of objects such as buildings in aerial photographs or point cloud data depicting landscapes.

[0150] In addition, various data handled in this specification are typically digital data.

[0151] Also, the learned model 81 in each of the above-described embodiments includes a "self-learning model" that further updates the internal algorithm of the learned model 81 when the user gives feedback on the product output by the learned model 81.

[0152] According to at least one of the embodiments described above, the load of the editing process of medical image data can be reduced.

[0153] Although several embodiments have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, replacements, changes, and combinations of the embodiments can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, as well as in the invention described in the claims and the equivalent scope thereof.

Explanation of Reference Numerals

[0154] 41, 42 Medical image data 51 Mask 60 Anatomical landmark 70, 70a~70c Brush 71, 71a~71c Locus 75a~75c, 76 Arrow 80 Model 81, 81a~81c Learned model 100 Medical image processing apparatus 110 NW interface 120 Memory circuit 130 Input interface 140 Display 141 GUI 141a Check box 141b, 141c Display field 141d UNDO Button 141e REDO Button 141f Area 150 Processing Circuit 151 Acquisition Function 152 Reception Function 153 Image Processing Function 154 Recording Function 155 Learning Function 156 Estimation Function 157 Display Control Function 158 Force Feedback Function 200 Medical Imaging Diagnosis Device 300 Network 501a~501i Pixel 502a~502i Pixel 511 First Area 512 Second Area 700 Correction Information a Maximum Point b Point S Medical Image Processing System

Claims

1. an acquisition unit that acquires a first contour representing an anatomical tissue region of a first subject depicted in first medical image data, a second contour different from the first contour representing the anatomical tissue region, and a position of at least one anatomical landmark in the first subject in the first medical image data; a learning unit that learns a difference between the first contour and the second contour based on a relative position of each of the first contour and the second contour with respect to the anatomical landmark; A medical image processing device comprising:

2. the second contour is a contour obtained by modifying the first contour by a user, the acquisition unit acquires correction information including a record of a time series change in a relative position of a change target portion with respect to the anatomical landmark in an operation by the user to correct the first contour to the second contour, The learning unit learns the correction information. The medical image processing device according to claim 1 .

3. the first contour includes a first region that overlaps with the second contour and a second region that does not overlap with the second contour; the learning unit learns the correction information so that a reward becomes higher as the first contour is corrected so that an omission of erasure in the second region is reduced and an erroneous erasure in the first region is reduced. The medical image processing device according to claim 2 .

4. the modification information includes information regarding a path of movement of an image editing tool that modifies the first contour and a size of an image area that can be modified by a single operation of the image editing tool; the learning unit learns the modification information so that a reward becomes higher as an image area that can be edited by a single operation of the image editing tool becomes larger. The medical image processing device according to claim 2 .

5. an estimation unit that estimates a correction target portion of a third contour representing an outline of an anatomical tissue of the second subject depicted in the second medical image data as a relative position based on a position of an anatomical landmark of the second subject, using a first trained model trained by the training unit; The medical image processing device according to claim 1 .

6. a display control unit that causes a display unit to distinguishably display a portion of the third contour of the second medical image data that is estimated to require a high degree of correction from a portion of the third contour that is estimated to require a low degree of correction. The medical image processing device according to claim 5 .

7. The estimation unit estimates, using the first trained model, a position where an image editing tool is placed on the second medical image data to correct the correction target portion of the third contour; and a manipulation control unit configured to control operation of the image editing tool on the second medical image data to facilitate approach of the image editing tool to the estimated location. The medical image processing device according to claim 5 .

8. The first contour is an estimation result of an outer perimeter of the region of the anatomical tissue depicted in the first medical image data by a second trained model. The medical image processing device according to claim 1 .

9. an acquiring step of acquiring a first contour representing an area of ​​an anatomical tissue of a first subject depicted in first medical image data, a second contour different from the first contour representing the area of ​​the anatomical tissue, and a position of at least one anatomical landmark in the first subject in the first medical image data; a learning step of learning a difference between the first contour and the second contour based on a relative position of each of the first contour and the second contour with respect to the anatomical landmark; A program for causing a computer to execute the following.

10. an acquiring step of acquiring a first contour representing an area of ​​an anatomical tissue of a first subject depicted in first medical image data, a second contour different from the first contour representing the area of ​​the anatomical tissue, and a position of at least one anatomical landmark in the first subject in the first medical image data; a learning step of learning a difference between the first contour and the second contour based on a relative position of each of the first contour and the second contour with respect to the anatomical landmark; The method includes:

Images