Image processing apparatus, image processing method, and program

Abstract

To improve the inference performance that acquires region information of lesion candidates by combining multiple models based on machine learning.SOLUTION: An image processing apparatus acquires first region information representing a first region from acquired medical image data, acquires a first inference result by applying a first inference model to the first medical image data, and a second inference result by applying a second inference model to second medical image data based on the medical image data, and acquires a region that satisfies a predetermined condition as second region information in at least one of a region based on the first inference result and a region based on the second inference result. The second region information acquires a region that spans the first region and outside of the first region in the regions based on the first inference result or a region that spans the first region and outside of the first region in the regions based on the second inference result as a region that satisfies the predetermined condition.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The embodiments disclosed in this specification and the drawings relate to an image processing device, an image processing method, and a program. [Background technology]

[0002] In the medical field, diagnosis is performed using images acquired by various medical image diagnostic devices (also referred to as medical image acquisition devices or modalities). In addition, a technology is known that supports diagnosis by detecting lesion candidates such as tumors from images using machine learning techniques. Specifically, area information of lesion candidates can be acquired from an image by inference processing using a trained model (also referred to as an inference model), and the acquired area information can be provided to a user such as a doctor as reference information for diagnosis.

[0003] In inference processing using an inference model, inference performance may vary for each region in the image. For example, when an image contains multiple body parts, the tumor characteristics may differ for each part, and it is known that inference does not work well if a single inference model is used because it cannot handle the different characteristics. On the other hand, when a method is adopted in which an image is divided into multiple regions and inference is performed using a different inference model for each region and then integrated (i.e., when multiple models are combined), the accuracy of the inference results at the boundaries of the regions may decrease. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-069793 Summary of the Invention [Problem to be solved by the invention]

[0005] One of the problems to be solved by the embodiments disclosed in this specification and the drawings is to improve inference performance when multiple models are combined in an inference process based on machine learning for acquiring region information of a lesion candidate. However, the problems to be solved by the embodiments disclosed in this specification and the drawings are not limited to the above problem. Problems corresponding to the effects of each configuration shown in the embodiments described later can also be positioned as other problems. [Means for solving the problem]

[0006] The image processing device according to the embodiment includes an image data acquisition unit, a first region information acquisition unit, an inference result acquisition unit, and a second region information acquisition unit. The image data acquisition unit acquires medical image data. The first region information acquisition unit acquires first region information representing a first region with respect to the medical image data. The inference result acquisition unit acquires a first inference result obtained by performing a first inference process that applies a first inference model to first medical image data based on the medical image data, and a second inference result obtained by performing a second inference process that applies a second inference model to second medical image data based on the medical image data. The second region information acquisition unit acquires, based on the first region information, a region that satisfies a predetermined condition from at least one of a region based on the first inference result and a region based on the second inference result as second region information. Furthermore, the second region information acquisition unit acquires the second region information by considering an area based on the first inference result that straddles the first region and an area outside the first region, or an area based on the second inference result that straddles the first region and an area outside the first region, as an area that satisfies the specified condition. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram illustrating an example of the overall configuration of an image processing apparatus according to the first embodiment. [Diagram 2] FIG. 2 is a flowchart showing an example of a process performed by the image processing device according to the first embodiment. [Diagram 3]FIG. 3 is a flowchart showing an example of acquisition of the first region information according to the first embodiment. [Figure 4] FIG. 4 is a flowchart showing an example of updating the second area information according to the first embodiment. [Diagram 5] FIG. 5 is an example of the first region and the second region according to the first embodiment. [Figure 6] FIG. 6 is a special example of the first region and the second region according to the first embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of the overall configuration of an image processing apparatus according to the second embodiment. [Figure 8] FIG. 8 is a flowchart showing an example of a process performed by the image processing device according to the second embodiment. [Figure 9] FIG. 9 is a flowchart showing an example of updating the second area information according to the second embodiment. [Figure 10] FIG. 10 is a flowchart showing an example of additional learning of an inference model according to the second embodiment. [Figure 11] FIG. 11 is a flowchart showing an example of a process performed by the image processing device according to the third embodiment. [Figure 12] FIG. 12 is a flowchart showing an example of additional learning of an inference model according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, preferred embodiments of an image processing device, an image processing method, and a program will be described in detail with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the configurations shown in the drawings.

[0009] <First embodiment> The image processing device according to the embodiment is a device that receives medical image data of a subject (e.g., a patient) as input, and acquires area information (lesion candidate area information) of lesion candidates such as tumors and polyps by inference processing based on machine learning. The image processing device executes a first inference processing using an inference model (first inference model) specialized for a specific part (partial area) such as the head, liver, or large intestine, and a second inference processing using an inference model (second inference model) mainly for the whole body, and acquires lesion candidate area information by combining these results. In this embodiment, a medical image will be described as an example of a processing target, but the processing target is not limited to a medical image.

[0010] Fig. 1 is a diagram showing an example of the configuration of an image processing device 10 according to the first embodiment. As shown in Fig. 1, the image processing device 10 has a processing circuitry 11, a communication interface 12, a storage circuitry 13, a display 14, an input interface 15, and a connection unit 16. In addition, the image processing device 10 is communicably connected to a medical image diagnostic device (modality), a medical image storage device, each department system, and the like via a network (not shown).

[0011] Examples of medical imaging diagnostic devices include PET (Positron Emission Tomography) devices, X-ray CT (Computed Tomography) devices, Magnetic Resonance Imaging (MRI) devices, X-ray diagnostic devices, etc. Medical image storage devices are realized by PACS (Picture Archiving and Communication System) or the like, and store medical images captured by medical imaging diagnostic devices in a format conforming to DICOM (Digital Imaging and Communications in Medicine). Examples of departmental systems include various systems such as a Hospital Information System (HIS), a Radiology Information System (RIS), a diagnostic report system, and a Laboratory Information System (LIS).

[0012] The processing circuit 11 executes a control function 11a, an image data acquisition function 11b, a first area information acquisition function 11c, an inference result acquisition function 11d, and a second area information acquisition function 11e in response to an input operation received from a user via an input interface 15 to control the image processing device 10. Here, the image data acquisition function 11b is an example of an image data acquisition unit. The first area information acquisition function 11c is an example of a first area information acquisition unit. The inference result acquisition function 11d is an example of an inference result acquisition unit. The second area information acquisition function 11e is an example of a second area information acquisition unit.

[0013] The control function 11a generates various GUIs (Graphical User Interfaces) and various display information in response to operations via the input interface 15, and controls the display 14 to display them. The control function 11a also controls the transmission and reception of information to and from devices and systems on a network (not shown) via the communication interface 12. For example, the control function 11a acquires information on the subject from an external system connected to the network. The control function 11a also outputs processing results to devices and systems on the network.

[0014] The image data acquiring function 11b acquires medical image data, obtained by capturing an image of a subject using a medical image diagnostic device or the like, from a medical image diagnostic device or a medical image storage device, based on reference address information of the image (such as a uniform resource locator (URL), various unique identifiers (UIDs), or a path character string). For example, the image data acquiring function 11b acquires a three-dimensional medical image (volume data) as the medical image data. The processing performed by the image data acquiring function 11b will be described in detail later.

[0015] The first region information acquiring function 11c acquires, as the first region information, region information representing a region of a specific part such as the head, the liver, the large intestine, etc. Note that the processing by the first region information acquiring function 11c will be described in detail later.

[0016] The inference result acquisition function 11d performs inference on the medical image data using each of the first inference model and the second inference model, and acquires the inference results by each of them. Note that the processing by the inference result acquisition function 11d will be described in detail later.

[0017] The second region information acquiring function 11e acquires region information of lesion candidates such as tumors and polyps (lesion candidate region information) as second region information based on the inference result acquired by the inference result acquiring function 11d. Note that the processing by the second region information acquiring function 11e will be described in detail later.

[0018] The above-mentioned processing circuitry 11 is realized by, for example, a processor. In this case, each of the above-mentioned processing functions is stored in the storage circuitry 13 in the form of a program executable by a computer. Then, the processing circuitry 11 realizes a function corresponding to each program by reading and executing each program stored in the storage circuitry 13. In other words, the processing circuitry 11 has each of the processing functions shown in FIG. 1 in a state where each program is read.

[0019] The processing circuit 11 may be configured by combining a plurality of independent processors, and each processor may execute a program to realize each processing function. Each processing function of the processing circuit 11 may be appropriately distributed or integrated and realized in a single or a plurality of processing circuits. Each processing function of the processing circuit 11 may be realized by a mixture of hardware such as a circuit and software. Here, an example in which a program corresponding to each processing function is stored in a single storage circuit 13 has been described, but the embodiment is not limited to this. For example, a configuration in which a program corresponding to each processing function is distributed and stored in a plurality of storage circuits, and the processing circuit 11 reads out and executes each program from each storage circuit.

[0020] The communication interface 12 controls the transmission and communication of various data transmitted and received between the image processing device 10 and other devices or systems connected via a network. Specifically, the communication interface 12 is connected to the processing circuitry 11, and outputs data received from other devices or systems to the processing circuitry 11, or transmits data output from the processing circuitry 11 to other devices or systems. For example, the communication interface 12 is realized by a network card, a network adapter, a NIC (Network Interface Controller), or the like.

[0021] The memory circuitry 13 stores various data and various programs. Specifically, the memory circuitry 13 is connected to the processing circuitry 11 and stores data input from the processing circuitry 11, or reads out stored data and outputs it to the processing circuitry 11. For example, the memory circuitry 13 is realized by a semiconductor memory element such as a random access memory (RAM) or a flash memory, a hard disk, an optical disk, or the like.

[0022] The display 14 displays various information and data. Specifically, the display 14 is connected to the processing circuit 11, and displays various information and data output from the processing circuit 11. For example, the display 14 is realized by a liquid crystal display, a CRT (Cathode Ray Tube) display, an organic EL display, a plasma display, a touch panel, or the like.

[0023] The input interface 15 accepts input operations of various instructions and various information from a user. Specifically, the input interface 15 is connected to the processing circuit 11, converts the input operation received from the user into an electrical signal, and outputs the electrical signal to the processing circuit 11. For example, the input interface 15 is realized by a trackball, a switch button, a mouse, a keyboard, a touch pad that performs an input operation by touching the operation surface, a touch screen in which a display screen and a touch pad are integrated, a non-contact input interface using an optical sensor, and a voice input interface. Note that in this specification, the input interface 15 is not limited to only those that have physical operation parts such as a mouse and a keyboard. For example, an electrical signal processing circuit that receives an electrical signal corresponding to an input operation from an external input device provided separately from the device and outputs the electrical signal to a control circuit is also included as an example of the input interface 15.

[0024] The connection unit 16 is a bus or the like that connects the processing circuit 11, the communication interface 12, the memory circuit 13, the display 14, and the input interface 15.

[0025] FIG. 2 is a flowchart showing an example of a process performed by the image processing device 10 according to the first embodiment. The image processing device 10 receives medical image data of an image of a subject, performs various inferences, and outputs second region information (lesion candidate region information) indicating the position of a lesion candidate. Here, the medical image data is three-dimensional volume data. The x-axis of the image coordinate system is set to increase toward the left hand side of the subject, the y-axis is set to increase toward the back side of the subject, and the z-axis is set to increase toward the head side of the subject. The pixel value of the coordinate "(x,y,z)" of the medical image a can be referenced by "a[z,y,x]". Furthermore, the partial image of the rectangular range with the coordinates "(x1,y1,z1)" and "(x2,y2,z2)" as diagonal points can be referenced by "a[z1:z2,y1:y2,x1:x2]". Here, if "x1,y1,z1" is omitted, 0 is specified. If "x2, y2, z2" is omitted, the image width in each axis direction is assumed to be specified.

[0026] In this embodiment, an example of processing for medical image data acquired by a PET-CT examination in which a PET examination and an X-ray CT examination are performed simultaneously will be mainly described. However, the embodiment is not limited to this, and can be similarly applied to image data of other modalities such as a PET image, an X-ray CT image, an MRI (Magnetic Resonance Imaging) image, and an ultrasound image.

[0027] 2 is realized, for example, by the processing circuitry 11 calling a program corresponding to the first area information acquisition function 11c from the storage circuitry 13 and executing the program. Moreover, the processing of steps S202 to S205 is realized, for example, by the processing circuitry 11 calling a program corresponding to the inference result acquisition function 11d from the storage circuitry 13 and executing the program. Moreover, the processing of steps S206 to S208 is realized, for example, by the processing circuitry 11 calling a program corresponding to the second area information acquisition function 11e from the storage circuitry 13 and executing the program.

[0028] (Step S200) In this embodiment, as shown in step S200, the processing circuitry 11 starts image processing by receiving medical image data as an input. For example, the processing circuitry 11 receives medical image data of a subject determined by an instruction operation or the like via the input interface 15 and starts image processing. The medical image data is loaded into the processing circuitry 11 by the image data acquisition function 11b in FIG.

[0029] (Step S201) In step S201, the first region information acquisition function 11c analyzes the medical image data acquired in step S200 to acquire first region information representing the first region. For example, the first region information acquisition function 11c acquires information representing a region of a specific part such as the head, liver, or large intestine as the first region information. The specific processing of step S201 can be performed in different configurations depending on what kind of region the first region is. The processing of step S201 in this embodiment will be described in detail below with reference to the flowchart in FIG. 3.

[0030] (If the first area is the head area) The processing procedure of step S201 will be described with reference to FIG. 3(a) using an example in which the first region is a head region. In the case of FIG. 3(a), the first region information acquisition function 11c inputs medical image data and starts acquiring the first region information as shown in step S300. Also, the first region information acquisition function 11c outputs the first region information as shown in step S305, and ends the acquisition of the first region information. Specifically, the first region information output in step S305 is stored in, for example, the memory circuitry 13, and is read out and used as appropriate in subsequent processing.

[0031] (Step S301) In step S301, the first region information acquisition function 11c acquires anatomical landmark information (e.g., landmark positions) of the subject from the medical image data acquired in step S200 using a known method such as random forest. At this time, if the medical image data is image data obtained by a PET-CT examination, the anatomical landmark information is acquired using data from a CT image in which the structural characteristics of body tissues are more clearly seen.

[0032] (Step S302) In step S302, the first region information acquisition function 11c executes a process of identifying the region in which the head is depicted in the medical image data. Specifically, the boundary position between the head and the region below it is identified using the position of a landmark (e.g., the atlas) located at the bottom end of the head among the landmarks acquired in step S301. More specifically, the z coordinate of the bottom end of the head is set as "iz1", and the z coordinate (image coordinate system) of the atlas in the landmark information is set.

[0033] (Step S303) In step S303, the first region information acquisition function 11c prepares a zero-initialized three-dimensional image of the same size as the medical image data as a mask image. Here, the mask image is an image in which the pixel values ​​of pixels included in the mask target are set to "1" and the other pixel values ​​are set to "0". The pixel values ​​of this mask image are changed by the process of step S304 described later.

[0034] (Step S304) In step S304, the first region information acquisition function 11c completes the identification of the head region in the medical image data, i.e., the creation of the first region information, by setting all pixel values ​​of the region of the medical image data that is closer to the head than the z coordinate value "iz1" to "1."

[0035] The process of step S201 is executed when the first region is the head by a series of processes from step S300 to step S305 shown in Fig. 3(a). The first region acquired in this embodiment corresponds specifically to the region indicated by 601 in Fig. 5, and is a region surrounding the head.

[0036] (If the first area is the large intestine area) The processing procedure of step S201 will be described with reference to FIG. 3(b) using an example in which the first region is the large intestine region. In the case of FIG. 3(b), the first region information acquisition function 11c inputs medical image data and starts acquiring the first region information, as shown in step S310. Also, the first region information acquisition function 11c outputs the first region information, as shown in step S319, and ends the acquisition of the first region information. Specifically, the first region information output in step S319 is stored in, for example, the memory circuitry 13, and is read out and used as appropriate in subsequent processing.

[0037] (Step S311) In step S311, the first region information acquisition function 11c acquires anatomical landmark information of the subject from the medical image data acquired in step S200 using a known method such as random forest. The specific process is similar to the method described in step S301, so a detailed description is omitted here.

[0038] (Step S312) In step S312, the first region information acquisition function 11c executes a process of identifying the position of the lower end of the region in which the large intestine is depicted in the medical image data. Since the position of the large intestine on the body axis is anatomically located between the 12th thoracic vertebra and the 5th sacrum, in step S312, the z coordinate of the lower end of the large intestine is set as "iz1" and the z coordinate (image coordinate system) of the 5th sacrum in the landmark information is set.

[0039] (Step S313) In step S313, the first region information acquisition function 11c executes a process of identifying the position of the upper end of the region in which the large intestine is depicted in the medical image data. Specifically, the z coordinate of the upper end of the large intestine is set as "iz2", and the z coordinate (image coordinate system) of the 12th thoracic vertebra in the landmark information is set.

[0040] (Step S314) In step S314, the first region information acquisition function 11c prepares a zero-initialized three-dimensional image having the same size as the medical image data as a mask image. This processing step is similar to the above-mentioned step S303, so detailed description will be omitted.

[0041] (Step S315) In step S315, the first region information obtaining function 11c cuts out an image in the range of "iz1:iz2" from the medical image data, and creates medical image data including the large intestine.

[0042] (Step S316) In step S316, the first region information acquisition function 11c acquires a body region mask image by analyzing the medical image data including the large intestine using known algorithms such as threshold processing, morphological calculation, labeling, etc. In the body region mask image, a region with a pixel value of 1 represents the inside of the body region, and a region with a pixel value of "0" represents the outside of the body region (air or bed).

[0043] (Step S317) In step S317, the first region information acquisition function 11c acquires the region with a CT value below about "-900" from the medical image data including the large intestine as the large intestine lumen region (including the air region outside the body region). In general, in a large intestine CT examination, air or carbon dioxide is injected to inflate the large intestine, so that the large intestine lumen region can be acquired by this process.

[0044] (Step S318) In step S318, the first region information acquisition function 11c multiplies each pixel value of the large intestine lumen region by each pixel value of the corresponding body region mask image (element-by-element multiplication) to exclude the air region outside the body region. The result is expanded by the morphological operation dilation to acquire a region including the large intestine wall, and the creation of the first region information is completed by substituting it for a predetermined slice of the first region.

[0045] Through a series of processes from step S310 to step S319 shown in FIG. 3(b), the process of step S201 is executed when the first region is the large intestine region.

[0046] (If the first area is the liver) The processing procedure of step S201 will be described with reference to FIG. 3(c) taking the case where the first region is the liver as an example. In the case of FIG. 3(c), the first region information acquisition function 11c inputs medical image data and starts acquiring the first region information as shown in step S320. Also, the first region information acquisition function 11c outputs the first region information as shown in step S324, and ends the acquisition of the first region information. Specifically, the first region information output in step S324 is stored in, for example, the memory circuitry 13, and is read out and used as appropriate in subsequent processing.

[0047] (Step S321) In step S321, the first region information acquisition function 11c loads a liver region acquisition model from the storage circuitry 13 into the processing circuitry 11 as a first region information acquisition model.

[0048] (Step S322) In step S322, the first region information acquisition function 11c performs inference on the medical image data using the above model to obtain an inference result for the first region. Here, the model for acquiring first region information is a set of parameters used in the inference process, and the inference process itself can be realized by a known inference technique such as U-Net. In addition, in this embodiment, the inference result is expressed as a three-dimensional image, and each pixel value of the image represents the probability that the pixel is included in the first region, but the present invention is not limited to this as a method of expressing the inference result, and it may be expressed as the same mask image as the first region.

[0049] (Step S323) In step S323, the first region information acquisition function 11c divides each pixel of the inference result into "0" and "1" using the probability threshold as a boundary value, and obtains the first region. Here, the probability threshold value is set to a value of about "0.5".

[0050] Through the series of processes from step S320 to step S324 shown in FIG. 3(c), the process of step S201 is executed when the first region is the liver.

[0051] In these examples of acquiring the first region information shown in FIG. 3, the first region is expressed as a mask image in which the pixel values ​​of the image corresponding to the parts included in the region are set to "1" and the other values ​​are set to "0". However, the embodiment is not limited to this expression method, and the first region information may be expressed by the coordinate values ​​of the bounding box of the region or a set thereof. Note that the method of acquiring the first region information is not limited to the above, and any organ recognition method or organ segmentation method based on image processing or machine learning may be used. Also, the first region information may be acquired from an external device, or the first region information may be input by a user via a UI (not shown). Also, the specific part may be an organ other than the head, large intestine, or liver, or may be a combination of multiple parts (for example, the entire abdomen or the entire chest and abdomen).

[0052] (Step S202) In step S202, the inference result acquisition function 11d acquires first medical image data, which is a target region of the first inference process, from the medical image data. Specifically, a process is executed to cut out a region wider than the first region acquired in step S201 from the medical image data. Here, the degree of expansion is a degree to which tumors and polyps that may cross the boundary of the first region are contained within the expanded range, and can be determined, for example, from teacher data of the first inference model. Specifically, a method is considered in which a typical size of a tumor or polyp is identified from teacher data, and the region is expanded by an amount approximately equal to the size. As a method of expanding the range of the region, for example, dilation of morphological operation can be used. Note that, although an example of expanding the region by morphological operation has been described, the embodiment is not limited thereto, and the region may be expanded by other image processing. For example, when the first region is a region of the large intestine, image data within a bounding box surrounding the first region may be the first medical image data. The first medical image data acquired in this embodiment specifically corresponds to an image of the region indicated by 602 in FIG. 5, and is an image of a region slightly protruding from the first region 601.

[0053] In addition, the first medical image data may be acquired by other methods as long as the first region includes a range with a margin. For example, a slice image group in which a predetermined number of adjacent slice images are added as margins to all slice images including the first region may be configured as the first medical image data. Also, the entire medical image data may be configured as the first medical image data. In this case, processing such as excluding unnecessary regions such as extracorporeal regions from the first medical image data may be performed.

[0054] (Step S203) In step S203, the inference result acquisition function 11d performs a first inference process on the first medical image data. Specifically, the inference result acquisition function 11d first loads a first inference model into the processing circuit 11, performs inference on the first medical image data using the first inference model, and obtains a first inference result.

[0055] Here, the inference model is a set of parameters used in the inference process. For example, the first inference model is a model that has been trained to match the characteristics of an image in a first region that represents a specific part. In this embodiment, for example, the first inference model is a model that has been constructed to extract a lesion candidate region (e.g., a tumor region) by inference from a PET-CT image of the head, large intestine, or liver. The inference process itself can be realized by a known inference technology such as U-Net, and inference is performed to calculate a discrimination (probability) of whether each position in the image is a lesion candidate region or not. Here, the range inferred by the first inference model is the entire region of the first medical image data, including the outside of the first region. In this embodiment, the inference result is expressed as a three-dimensional image, and each pixel value of the image represents the probability that the pixel is included in the lesion candidate region of the inference result. However, the method of expressing the inference result is not limited to this, and for example, the lesion candidate region may be expressed as a mask image.

[0056] (Step S204) In step 204, the inference result acquisition function 11d executes a process of cutting out the second medical image data including the target region of the second inference process from the medical image data acquired in step S200. Here, the specific process of step 204 can be performed in different configurations depending on what kind of region is the target region of the second inference process.

[0057] When the target region of the second inference process is the "whole body including the first region," the medical image data acquired in step S200 is treated as the second medical image data as is. In other words, the image including the first medical image data that includes the first region is treated as the second medical image data.

[0058] When the main target region of the second inference process is "the whole body except the first region," the cut-out range of the second medical image data is set so that the range of the second medical image data overlaps with the first region. Specifically, a first contracted region is obtained by contracting the first region. Next, the second medical image data is obtained by cutting out the medical image data other than the region corresponding to the first contracted region. Here, the degree of overlap of the regions is the degree to which a tumor that may cross the boundary of the first region falls within the overlapping range, and can be determined, for example, from the expected size of the tumor.

[0059] The second medical image data acquired in this embodiment specifically corresponds to an image of the region indicated by 603 in FIG. 5, and is an image of a region that partially overlaps with the first region 601.

[0060] The second medical image data may be acquired by other methods as long as it includes a range in which a margin is added to an area other than the first area. For example, the second medical image data may be a slice image group in which a predetermined number of adjacent slice images are added as margins to all slice images that do not include the first area. The second medical image data may be the entire medical image data. The second medical image data does not need to include all areas other than the first area, and may be processed to exclude unnecessary areas such as extracorporeal areas.

[0061] (Step S205) In step S205, the inference result acquisition function 11d performs a second inference process on the second medical image data using a second inference model. The specific processing method is the same as in step S203, but the image to be processed and the inference model are different in this processing step. The second inference model used in this processing step is different from the first inference model used in step S203, and is a model that has been trained to match the characteristics of images outside the first region. Specifically, the first inference model is a model constructed to extract lesion candidates from medical image data of a specific region (head, large intestine, or liver), while the second inference model is a model constructed to extract lesion candidates from medical images of regions other than the specific region. As a result, in the second inference process, an inference process with characteristics different from those of the first inference process is performed.

[0062] (Step S206) In step S206, the second region information acquisition function 11e prepares the second region information in an empty state, as indicated by "φ" in Fig. 2. In this embodiment, the second region information is a mask image that sets the pixel values ​​of the image corresponding to the locations included in the region it indicates to "1" and sets the rest to "0". However, the second region information may be expressed by other methods besides this, such as the coordinate values ​​of the bounding box of the region or a set of coordinates.

[0063] The second area prepared in the above process is updated in steps S207 and S208, which will be described later, whereby the first inference result and the second inference result are integrated.

[0064] (Step S207) In step 207, the second area information acquisition function 11e updates the second area information using the first inference result. A specific processing procedure of this processing step will be described in detail with reference to FIG. 4(a). In the case of FIG. 4(a), the second area information acquisition function 11e starts update 1 of the second area information with the first inference result, the first area information, and the second area information as inputs, as shown in step S600. Also, the first area information acquisition function 11c outputs the updated second area information, as shown in step S606, and ends update 1 of the second area information. Specifically, the second area information output in step S606 is stored in, for example, the memory circuit 13, and is read out and used as appropriate in subsequent processing.

[0065] (Step S601) In step S601, the second region information acquisition function 11e acquires a list of regions that are candidates for the second region from the first inference result. More specifically, first, each pixel of the first inference result acquired in step S203 is binarized with a predetermined threshold value (e.g., "0.5"). Next, the binarized result is labeled by a known method, and each region assigned to the same label is recorded. In other words, each cluster of continuous regions resulting from the binarization of the first inference result is recorded, and this is used as a region list.

[0066] (Step S602) Thereafter, steps S603 to S605 are repeated for all areas in the area list acquired in step S601, with each area being treated as a target area.

[0067] (Step S603) In step S603, the second region information acquisition function 11e determines whether the target region straddles the boundary of the first region. This determination can be made, for example, by the following conditional expression (1).

[0068]

number

[0069] If the above conditional expression is true, the target area crosses the boundary of the first area, so the process proceeds to the Yes branch and to step S605. If not, the process proceeds to the No branch and to step S604. It should be noted that the determination of Yes in this processing step is made when the target area corresponds to the area indicated by 605 in FIG. 5 and is an area that protrudes outside the first area 601.

[0070] (Step S604) In step S604, the second region information acquisition function 11e judges whether the first region completely contains the target region. If the judgment is Yes, the process proceeds to step S605. If the judgment is No, the process of the inner loop of the loop process in step S602 is terminated. Note that, in this processing step, it is judged as Yes when the target region corresponds to the region indicated by 604 in FIG. 5 and is a region completely contained in the first region 601. In addition, in this processing step, it is judged as No when the target region corresponds to the region indicated by 606 in FIG. 5 and is a region outside the first region 601.

[0071] (Step S605) In step S605, the second region information acquisition function 11e executes a process of adding the target region to the second region. For example, in the example shown in Fig. 5, a region (605) that straddles the boundary of the first region and a region (604) that is completely contained in the first region are added to the second region.

[0072] The process of step S207 is executed by a series of processes from step S600 to step S606 shown in Fig. 4(a). As a result, among the regions of the first inference result acquired in step S203, the regions at least partially included in the first region are acquired as the second region. That is, the region obtained by excluding the regions not even partially included in the first region from the first inference result is acquired as the second region.

[0073] If the first inference result is sufficiently reliable (few false positives) even outside the first region, all regions in the region list may be set as second regions without excluding regions that are not part of the first region. In this case, in step S602, the second region information acquisition function 11e does not perform the processes from step S603 to step S605, but executes a process of adding all regions in the region list to the second region.

[0074] (Step S208) In step S208, the second area information acquisition function 11e updates the second area information using the second inference result. A specific processing procedure of this processing step will be described in detail with reference to FIG. 4(b). In the case of FIG. 4(b), the second area information acquisition function 11e starts update 2 of the second area information with the second inference result, the first area information, and the second area information as inputs, as shown in step S610. In addition, the second area information acquisition function 11e outputs the updated second area information, as shown in step S616, and ends update 2 of the second area information. Specifically, the second area information output in step S616 is stored in, for example, the memory circuit 13, and is read out and used as appropriate in subsequent processing.

[0075] (Step S611) In step S611, the second region information acquisition function 11e acquires a list of regions that are candidates for the second region from the second inference result. More specifically, first, each pixel of the second inference result acquired in step S205 is binarized with a predetermined threshold value (for example, "0.5"). Next, the binarized result is labeled by a known method, and each region assigned to the same label is recorded, and this is made into a region list.

[0076] (Step S612) Thereafter, steps S613 to S615 are repeated for all areas in the area list acquired in step S611, with each area being treated as a target area.

[0077] (Step S613) In step S613, the second region information acquisition function 11e determines whether the target region straddles the boundary of the first region. This determination can be made, for example, by the conditional expression (1) in the same manner as in step S603.

[0078] If the above conditional expression is true, the process proceeds to the Yes branch and proceeds to step S615. If not, the process proceeds to the No branch and proceeds to step S614. It should be noted that the determination of Yes in this processing step is made when the target region corresponds to the region indicated by 608 in FIG. 5 and is a region that has entered the first region 601.

[0079] (Step S614) In step S614, the second area information acquisition function 11e judges whether the target area is completely contained in the "area outside the first area". If the judgement is Yes, the process proceeds to step S615. If the judgement is No, the process of the inner loop of the loop process in step S612 is terminated. Note that, in this processing step, it is judged Yes when the target area corresponds to the area indicated by 609 in FIG. 5 and is an area outside the first area 601. Also, in this processing step, it is judged No when the target area corresponds to the area indicated by 607 in FIG. 5 and is an area completely contained in the first area 601.

[0080] (Step S615) In step S615, the second region information acquisition function 11e executes a process of adding the target region to the second region. For example, in the example shown in Fig. 5, a region (608) that straddles the boundary of the first region and a region (609) that is completely included in the "region outside the first region" are added to the second region.

[0081] The process of step S208 is executed by a series of processes from step S610 to step S616 shown in Fig. 4(b). This allows acquisition of second region information based on the second inference result including the region that has entered the first region. That is, the region excluding the region that is not partly included outside the first region from the second inference result is acquired as the second region.

[0082] In addition, if the second inference result is sufficiently reliable even within the first region (few false positives), it is possible to configure all regions in the region list as second regions without excluding regions that are not partly included outside the first region. For example, in the example shown in Fig. 5, a region (607) that is not partly included in the "region outside the first region" may be added to the second region. In this case, in step S612, the second region information acquisition function 11e does not perform the processes of steps S613 to S615, but executes a process of adding all regions in the region list to the second region.

[0083] Furthermore, through the processing of steps S207 and S208, second region information is acquired as follows. That is, among the lesion candidate regions obtained by the first inference processing, at least a part of which is included in the first region, and among the lesion candidate regions obtained by the second inference processing, at least a part of which is included outside the first region are acquired. At this time, among the lesion candidate regions obtained by the first inference processing, a region not included in any part in the first region and / or among the lesion candidate regions obtained by the second inference processing, not included in any part outside the first region can be configured to be excluded from the lesion candidate regions. This is expected to have the effect of reducing false positives caused by expanding the processing range to parts not assumed by each inference model.

[0084] Furthermore, among the lesion candidate regions obtained by the first inference process and / or the second inference process, a region that straddles the first region and the outside of the first region can be included in the lesion candidate region. This is expected to reduce false negatives caused by tumor candidates at seams being divided.

[0085] In the process of steps S605 and S615 in FIG. 4, whether or not to add the "straddling region" for which the judgment in step S603 or S613 is Yes to the second region may be determined based on whether or not the size of the region (specifically, volume or area) is equal to or larger than a predetermined size. Since each pixel value of the image of the inference result represents the probability that the pixel is included in the lesion candidate region of the inference result, whether or not to add the "straddling region" to the second region may be determined based on whether or not the probability of the region is equal to or larger than a predetermined value. That is, whether or not to add the "straddling region" to the second region may be determined based on whether or not the sum of the probabilities that each pixel in the region is included in the second region (or the maximum value, median value, average value, minimum value, etc. of the probability) is equal to or larger than a predetermined value. Here, the thresholds of the size and probability need to be determined so as to minimize the occurrence of false positives, and can be determined, for example, from teacher data of each inference model. In addition, the second region information acquisition function 11e may determine whether or not to add the "straddling region" to the second region depending on whether or not the region is a continuous region.

[0086] Furthermore, in the processing of steps S603 and S613 in Fig. 4, whether the region crosses the boundary of the first region may be determined based on the contact area with the boundary of the first region. That is, for example, it may be determined based on whether the area intersecting with the end face of the first region is large. This determination method is also effective when the first region (601) and the first medical image data (614) are in the same range as shown in Fig. 6. That is, when the surface (613) intersecting with the end face of the first region is large, the region (612) can be regarded as a region that crosses the first region and outside the first region even if there is no region that actually extends beyond the first region.

[0087] Furthermore, as shown in step S209, the processing circuitry 11 outputs the second region information and ends the image processing. For example, in step S209, the control function 11a stores (saves) the second region information acquired by the processing of steps S200 to S208 in the memory circuitry 13, or displays it on the display 14. For example, a user such as a doctor can refer to the second region information displayed on the display 14 as reference information for diagnosis.

[0088] As described above, by using the image processing device of this embodiment, it is possible to improve the inference performance when multiple models are combined in inference processing based on machine learning to obtain area information of lesion candidates.

[0089] It is known that when inference processing is performed using a single inference model, inference does not work well due to differences in the characteristics of each region. For example, in a PET device, tumors are imaged brightly, but in the brain, not only tumors but also non-tumorous parts are imaged brightly. For this reason, when performing tumor detection using machine learning on PET images, it is known that inference does not work well if a single inference model is used for the entire body because it cannot handle different characteristics.

[0090] In response to this, for example, Patent Document 1 discloses a method of performing inference in an area where different sections are adjacent to each other using a trained model different from that used in other areas. However, when inferring and integrating each of a plurality of areas using a different model, the accuracy of the inference results at the boundaries between the areas is reduced.

[0091] In contrast, according to the image processing device of the first embodiment, while using multiple inference models in the machine learning-based inference process to obtain regional information of lesion candidates, it is possible to reduce false positives at the boundary between regions as described above and improve inference performance.

[0092] The flowchart shown in FIG. 2 is merely an example, and specific processing procedures can be changed as appropriate. For example, the processing of step S206 in which the second region information is prepared as an empty state may be executed before step S205. Also, in FIG. 2, the output processing is shown as step S209, but in each step of steps S201 to S208, the product of each step may be output as appropriate. For example, the control function 11a may store the first region information acquired in step S201 in the memory circuit 13 or display it on the display 14 as the product of step S201. Similarly, specific processing procedures can be changed as appropriate for other flowcharts other than FIG. 2.

[0093] <Second embodiment> As an example of the process of step S201 in the first embodiment, a process of acquiring first region information by inference using an inference model based on machine learning has been described. The model used for this inference may be additionally learned using the result of acquiring second region information. An example of this case will be described as the second embodiment. In this embodiment, a medical image will be described as an example of a processing target, but the processing target is not limited to a medical image.

[0094] 7 shows the overall configuration of an image processing device 20 of this embodiment. The processing circuit 21 of this embodiment is different from the processing circuit 11 of the first embodiment in the processing content by the second area information acquisition function 11e. Also, compared to the processing circuit 11 of the first embodiment, the processing circuit 21 of this embodiment has an additional learning function 11f. Here, the additional learning function 11f is an example of an additional learning unit. The following mainly describes these.

[0095] Fig. 8 is a flowchart showing an example of a process performed by the image processing device 20 according to this embodiment. Here, the process shown in the dotted line frame (step S802) in Fig. 8 is the same process as steps S202 to S206 in Fig. 2. Moreover, the processes of steps S803 to S805 shown in Fig. 8 are realized, for example, by the processing circuitry 11 calling from the storage circuitry 13 a program corresponding to the second area information acquisition function 11e and executing the program. Moreover, the process of step S806 shown in Fig. 8 is realized, for example, by the processing circuitry 11 calling from the storage circuitry 13 a program corresponding to the additional learning function 11f and executing the program.

[0096] As shown in step S800, the processing circuitry 11 starts image processing by inputting medical image data. Also, as shown in step S807, the processing circuitry 11 outputs second region information and a model for acquiring first region information, which will be described later, and ends the image processing. For example, in step S807, the control function 11a outputs the second region information, similar to step S209. Also, for example, in step S807, the control function 11a stores the model for acquiring first region information in the memory circuitry 13 so that it can be read out and used later.

[0097] (Step S801) In step S801, first area information is acquired in the same manner as in step S201 in the first embodiment. However, in this processing step, the first area is acquired by inference using an inference model based on machine learning. The inference model used here is a model constructed to acquire the first area, but can be updated by the additional learning function 11f. Here, this model is referred to as a model for acquiring first area information. The model is updated in a later processing step, which will be described in detail in the explanation of the processing in this step.

[0098] (Step S802) In step S802, the same processes as those in steps S202 to S206 in the first embodiment are performed.

[0099] (Step S803) In step S803, the first improvement region is initialized with the first region information. In steps S804 and S805, this first improvement region is updated, and in step S806, it is used for additional learning of the model for acquiring first region information.

[0100] (Step S804) In step S804, the second area information acquisition function 11e updates the second area information using the first inference result, and also updates the first improvement area at the same time. A specific processing procedure of this processing step will be described in detail with reference to FIG. 9(a). In the case of FIG. 9(a), the second area information acquisition function 11e starts update 1 of the second area information by inputting the first inference result, the first area information, the second area information, and the first improvement area information indicating the first improvement area, as shown in step S1000. In addition, the second area information acquisition function 11e outputs the updated second area information and the first improvement area information, as shown in step S1007, and ends update 1 of the second area information. Specifically, the second area information and the first improvement area information output in step S1007 are stored in, for example, the storage circuit 13, and are appropriately read and used in the subsequent processing.

[0101] Specifically, as shown in FIG. 9(a), the second region information acquisition function 11e acquires a list of regions that are candidates for the second region from the first inference result in step S1001. Next, steps S1003 to S1006 are repeated for all regions in the region list. Specifically, in step S1003, whether or not the region crosses the boundary of the first region is determined by the conditional expression of formula (1), similarly to step S603 in FIG. 4. If the determination is Yes, the region is a region that protrudes outside the first region, so in step S1006, the region is added to the first improvement region. Also, in step S1005, the region is added to the second region. If the determination is No in step S1003, the process proceeds to step S1004, where it is determined whether or not the region is completely contained in the first region. If the determination is Yes, the region is added to the second region in step S1005. If the determination is No, the region is not added to the second region.

[0102] (Step S805) Next, as shown in step S805 of FIG. 8, the second area information acquisition function 11e updates the second area information using the second inference result, and also updates the first improvement area at the same time. A specific processing procedure of this processing step will be described in detail with reference to FIG. 9(b). In the case of FIG. 9(b), the second area information acquisition function 11e starts update 2 of the second area information by inputting the second inference result, the first area information, the second area information, and the first improvement area information, as shown in step S1100. In addition, the second area information acquisition function 11e outputs the updated second area information and the first improvement area information, as shown in step S1107, and ends update 1 of the second area information. Specifically, the second area information and the first improvement area information output in step S1107 are stored in, for example, the storage circuit 13, and are appropriately read and used in the subsequent processing.

[0103] Specifically, as shown in FIG. 9(b), the second region information acquisition function 11e acquires a list of regions that are candidates for the second region from the second inference result in step S1101. Next, steps S1103 to S1106 are repeated for all regions in the region list. Specifically, in step S1103, it is determined whether the region crosses the boundary of the first region using a conditional expression similar to that in step S603 in FIG. 4. If the determination is Yes, the region is an area that has entered the first region from the outside, so in step S1106, the region is deleted from the first improvement region. In addition, in step S1105, the region is added to the second region. If the determination is No in step S1103, the process proceeds to step S1104, where it is determined whether the region is completely contained in the "region outside the first region". If the determination is Yes, the region is added to the second region in step S1105. If the determination is No, the region is not added to the second region.

[0104] By the above process, it is possible to obtain a "first improvement region" by adding the region that was insufficiently extracted as the first region and deleting the region that was excessively extracted.

[0105] (Step S806) Next, as shown in step S806 of FIG. 8, the additional learning function 11f performs additional learning of the first region information acquisition model using the medical image data and the first improvement region as additional teacher data. A specific processing procedure of this processing step will be described in detail with reference to FIG. 10. In the case of FIG. 10, as shown in step S910, the additional learning function 11f starts additional learning of the first region information acquisition model using the medical image data and the first improvement region information as input. In addition, as shown in step S915, the additional learning function 11f outputs the additionally learned first region information acquisition model and ends the additional learning. Step S915 of FIG. 10 may be integrated into step S807 of FIG. 8.

[0106] Specifically, as shown in FIG. 10, the additional learning function 11f loads the first region information acquisition model from the storage circuitry 13 to the processing circuitry 21 in step S911. Next, in step S912, medical image data is input, and the first improvement region is used as the correct answer to calculate the gradient "∇E" of the loss function "E" of the first region information acquisition model. Next, in step S913, only "ε∇E" is subtracted from the first region information acquisition model to update the first region information acquisition model. Here, "ε" is a learning rate and is set to a sufficiently small value so that the loss function "E" does not diverge. The model update method of this embodiment is an example, and the present invention is not limited to this, and the momentum method, AdaGrad method, etc. may be used. Finally, in step S914, the first region information acquisition model is saved in the storage circuitry 13. As a result, the first region information acquisition model used in step S801 is updated. From then on, in step S801, inference is performed using this updated model.

[0107] In this way, by performing additional learning in the "first improvement area", a more accurate "first area" can be obtained, and ultimately, the accuracy of the first inference process can be improved.

[0108] As described above, by using the image processing device according to this embodiment, it is possible to improve the accuracy of region segmentation when combining multiple models, and to improve the overall inference performance.

[0109] <Third embodiment> As the processing of step S203 and step S205 in the first embodiment, the processing of acquiring the first inference result and the second inference result using an inference model based on machine learning has been described. The model used for this inference may be additionally learned using the result of the second region acquisition. An example of this case will be described as the third embodiment. In this embodiment, a medical image will be described as an example of the processing target, but the processing target is not limited to a medical image.

[0110] The overall configuration of this embodiment is the same as that of the second embodiment, that is, the image processing device 20 in Fig. 7. The processing circuit 21 of this embodiment is different from the processing circuit 11 of the second embodiment in the processing contents by the additional learning function 11f. The following mainly describes these.

[0111] Fig. 11 is a flowchart showing an example of steps performed by the image processing device 20 according to this embodiment. Here, the processes shown in the dotted lines in Fig. 11 (steps S1201, S1203, and S1205) are the same as steps S201 to S202, step S204, and steps S206 to S208 in Fig. 2, respectively. Furthermore, the processes in steps S1206 and S1207 shown in Fig. 11 are realized, for example, by the processing circuitry 11 calling up a program corresponding to the additional learning function 11f from the storage circuitry 13 and executing it.

[0112] As shown in step S1200, the processing circuitry 11 starts image processing by inputting medical image data. Also, as shown in step S1208, the processing circuitry 11 outputs the second region information and the first inference model and the second inference model after additional learning, and ends the image processing. For example, in step S1208, the control function 11a outputs the second region information, similar to step S209. Also, for example, in step S1208, the control function 11a stores the first inference model and the second inference model after additional learning in the memory circuitry 13 so that they can be read out and used later.

[0113] (Step S1201) In step S1201 in FIG. 11, the same processes as those in steps S201 and S202 in the first embodiment are performed.

[0114] (Step S1202) In step S1202, the first inference process is performed in the same manner as in step S203 in the first embodiment. The inference model used here is a model constructed to obtain the first inference result, but is updateable by the additional learning function 11f. Here, this model is referred to as the first inference model. The model is updated in a later processing step, which will be described in detail in the explanation of the processing in this step.

[0115] (Step S1203) In step S1203, the same process as in step S204 in the first embodiment is performed.

[0116] (Step S1204) In step S1204, the second inference process is performed similarly to step S205 in the first embodiment. The inference model used here is a model constructed to obtain the second inference result, but is updateable by the additional learning function 11f. Here, this model is referred to as the second inference model. The model is updated in a later processing step, which will be described in detail in the explanation of the processing in this step.

[0117] (Step S1205) In step S1205, the same processes as those in steps S206 to S208 in the first embodiment are performed.

[0118] (Step S1206) Next, as shown in step S1206 of FIG. 11, the additional learning function 11f performs additional learning of the first inference model using the first medical image data, the second region, and the first inference result.

[0119] The specific processing procedure of step S1206 will be described in detail with reference to Fig. 12. Note that Fig. 12 collectively describes the additional learning of the first inference model in step S1206 and the additional learning of the second inference model in step S1207 described below. That is, Fig. 12 is a diagram for explaining the additional learning of the Nth inference model, and shows the processing procedure of step S1206 when "N=1" and the processing procedure of step S1207 when "N=2".

[0120] In the case of "N=1" in FIG. 12, the additional learning function 11f starts additional learning of the first inference model using the first medical image data, the second region information, and the first inference result as inputs, as shown in step S1300. In addition, the additional learning function 11f outputs the additionally learned first inference model as shown in step S1307, and ends the additional learning. For example, in step S1307, the additional learning function 11f stores the additionally learned first inference model in the memory circuitry 13 so that it can be read out and used later.

[0121] Specifically, as shown in FIG. 12 (when N is 1), the additional learning function 11f extracts a portion of the second region that is included in the region of the first medical image data and obtains a first additional teacher label in step S1301. Next, in step S1302, it is determined whether the first additional teacher label is equal to the result of binarizing the first inference result. If the determination is Yes, it is determined that there is no region information worth additional learning, and the process proceeds to step S1303 to end the additional learning process. If the determination is No, the process proceeds to step S1304 to load the first inference model from the storage circuitry 13 to the processing circuitry 21. Next, in step S1305, the first medical image data is input, and the gradient "∇E" of the loss function "E" of the first inference model is calculated using the first additional teacher label as the correct answer. Next, in step S1306, only "ε∇E" is subtracted from the first inference model to update the first inference model. Here, "ε" is a learning rate and is set to a sufficiently small value so that the loss function "E" does not diverge. Note that the above-mentioned model update method is an example, and the present embodiment is not limited to this. The momentum method, AdaGrad method, etc. may also be used.

[0122] (Step S1207) Next, as shown in step S1207 of Fig. 11, the additional learning function 11f performs additional learning of the second inference model using the second medical image data, the second region, and the second inference result. Specifically, it is as shown in Fig. 12 (when N is 2), and a detailed explanation is omitted because it can be inferred from the case where N is 1.

[0123] Through the above processing, it is possible to strengthen the extraction of areas that have failed to be extracted by the inference of each model through additional learning using additional teacher labels.

[0124] As described above, by using the image processing device according to this embodiment, it is possible to improve the inference accuracy of individual lesion candidate regions when combining multiple models, and to improve the overall inference performance.

[0125] <Other embodiments> Although the embodiment has been described in detail above, the present invention is not limited to this. For example, the present invention can be embodied as a system, an apparatus, a method, a program, or a recording medium (storage medium). Specifically, the present invention may be applied to a system composed of multiple devices (for example, a host computer, an interface device, an imaging device, a Web application, etc.), or may be applied to an apparatus composed of a single device.

[0126] The term "processor" used in the above description means a circuit such as a CPU, a GPU (Graphics Processing Unit), an ASIC, a programmable logic device (e.g., a Simple Programmable Logic Device (SPLD), a Complex Programmable Logic Device (CPLD), and a Field Programmable Gate Array (FPGA)). When the processor is, for example, a CPU, the processor realizes a function by reading and executing a program stored in a memory circuit. On the other hand, when the processor is, for example, an ASIC, instead of storing a program in a memory circuit, the function is directly incorporated as a logic circuit in the circuit of the processor. Note that each processor in the embodiment is not limited to being configured as a single circuit for each processor, but may be configured as one processor by combining multiple independent circuits to realize the function. Furthermore, multiple components in each figure may be integrated into one processor to realize the function.

[0127] Each component of each device according to the above-mentioned embodiment is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or a part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc. Furthermore, each processing function performed by each device can be realized in whole or in any part by a CPU and a program analyzed and executed by the CPU, or can be realized as hardware using wired logic.

[0128] The image processing method described in the above embodiment can be realized by executing a prepared program on a computer such as a personal computer or a workstation. This program can be distributed via a network such as the Internet. This program can also be recorded on a non-transitory recording medium readable by a computer such as a hard disk, a flexible disk (FD), a CD-ROM, an MO, or a DVD, and executed by being read from the recording medium by a computer.

[0129] According to at least one of the embodiments described above, inference performance can be improved when multiple models are combined in inference processing based on machine learning for acquiring region information of lesion candidates.

[0130] Although some embodiments have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, and combinations of embodiments can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope of the invention and its equivalents as described in the claims, as well as in the scope and spirit of the invention. [Explanation of symbols]

[0131] 10: Image processing device 11: Processing circuit 11a: Control functions 11b: Image data acquisition function 11c: First area information acquisition function 11d: Inference result acquisition function 11e: Second area information acquisition function 11f: Additional learning function 12: Communication interface 13:Memory circuit 14: Display 15: Input interface 16: Connection 20: Image processing device 21: Processing circuit

Claims

1. an image data acquisition unit for acquiring medical image data; a first region information acquisition unit that acquires first region information representing a first region with respect to the medical image data; an inference result acquisition unit that acquires a first inference result obtained by performing a first inference process that applies a first inference model to first medical image data based on the medical image data, and a second inference result obtained by performing a second inference process that applies a second inference model to second medical image data based on the medical image data; and a second region information acquisition unit that acquires, based on the first region information, a region that satisfies a predetermined condition from at least one of a region based on the first inference result and a region based on the second inference result as second region information, The image processing device, wherein the second region information acquisition unit acquires the second region information by determining, as a region that satisfies the specified condition, an area based on the first inference result that straddles the first region and an area outside the first region, or an area based on the second inference result that straddles the first region and an area outside the first region.

2. 2. The image processing device according to claim 1, wherein the second region information acquisition unit acquires the second region information by considering an area based on the first inference result and an area based on the second inference result that exists between the first region and outside the first region as an area that satisfies the specified condition.

3. 3. The image processing device of claim 1, wherein the second region information acquisition unit does not acquire as the second region information at least one of an area based on the first inference result that exists only outside the first region and an area based on the second inference result that exists only within the first region, as an area that does not satisfy the specified condition.

4. 3. The image processing device according to claim 1, wherein the second region information acquisition unit acquires the second region information by determining that an area that exists between the first region and an area outside the first region is equal to or larger than a predetermined size as an area that satisfies the predetermined condition.

5. 3. The image processing device according to claim 1, wherein the second region information acquisition unit acquires the second region information by determining that a region that exists between the first region and an area outside the first region satisfies the specified condition when the probability of inference for the region is greater than or equal to a specified value.

6. The second region information acquisition unit is configured to acquire information on a total probability of the inference for pixels constituting a region that exists across the first region and outside the first region, The image processing device according to claim 5 , wherein if the value is equal to or greater than a predetermined value, the region is determined to satisfy the predetermined condition, and the second region information is acquired.

7. 3. The image processing device according to claim 1, wherein the second region information acquisition unit determines, based on the first region information, whether at least one of the region based on the first inference result and the region based on the second inference result spans the first region and outside the first region, and acquires the second region information based on the determination result.

8. 8. The image processing device according to claim 7, wherein the second region information acquisition unit makes the determination based on a contact area between at least one of the region based on the first inference result and the region based on the second inference result and a boundary of the first region.

9. the first region information acquisition unit acquires the first region information by inference using a first region information acquisition model; The image processing device according to claim 1 or 2, further comprising an additional learning unit that performs additional learning of the first region information acquisition model using an area based on the first inference result and an area based on the second inference result that exists between the first region and outside the first region.

10. An image processing device as described in claim 1 or 2, further comprising an additional learning unit that additionally learns the first inference model based on difference information between the first inference result and the result acquired by the second area information acquisition unit.

11. An image processing device as described in claim 1 or 2, further comprising an additional learning unit that additionally learns the second inference model based on difference information between the second inference result and the result acquired by the second area information acquisition unit.

12. 3. The image processing device according to claim 1, wherein the second region information acquisition unit acquires the second region information by determining that an area that exists between the first region and an area outside the first region is a continuous area as an area that satisfies the specified condition when the area is a continuous area.

13. An image data acquisition step of acquiring medical image data; A first region information acquisition step of acquiring first region information representing a first region with respect to the medical image data; an inference result acquisition step for acquiring a first inference result obtained by performing a first inference process that applies a first inference model to first medical image data based on the medical image data, and a second inference result obtained by performing a second inference process that applies a second inference model to second medical image data based on the medical image data; An image processing method including a second region information acquisition unit that acquires, based on the first region information, a region that satisfies a predetermined condition from at least one of a region based on the first inference result and a region based on the second inference result as second region information, An image processing method in which the second region information acquisition step acquires the second region information by defining an area based on the first inference result that straddles the first region and an area outside the first region, or an area based on the second inference result that straddles the first region and an area outside the first region, as an area that satisfies the specified condition.

14. A program for causing a computer to execute the image processing method according to claim 13.

15. an image data acquisition unit for acquiring medical image data; a first region information acquisition unit that acquires first region information representing a first region of a subject from the medical image data; a first inference result acquisition unit that acquires a first inference result, which is a result of performing a first inference process using a first inference model trained using teacher data corresponding to a head region of a subject on first medical image data based on the medical image data; a second inference result acquisition unit that acquires a second inference result, which is a result of performing a second inference process on second medical image data based on the medical image data, using a second inference model learned using teacher data including at least one of teacher data corresponding to a whole-body region including the head of the subject and teacher data corresponding to a whole-body region other than the head of the subject; and a second region information acquisition unit that integrates the first inference result and the second inference result based on the first region information to acquire an inference result for the medical image data; The image processing device wherein the second region information acquisition unit, when a region based on the first inference result is a region that exists both within and outside the region corresponding to the first region information, or a region based on the second inference result is a region that exists both within and outside the region corresponding to the first region information, integrates the region into the inference result for the medical image data as a region that satisfies a predetermined condition.

16. 16. The image processing device of claim 15, wherein the second region information acquisition unit does not integrate into the inference result at least one of a region based on the first inference result that exists only outside the first region and a region based on the second inference result that exists only within the first region as a region that does not satisfy the specified condition.

Images