Image processing method, image processing device and program

Abstract

To provide an image processing method, an image processing apparatus, a program, and a trained model which can reduce computing resources required for alignment between a plurality of images.SOLUTION: The image processing method is performed by one or more processors and includes acquiring respective feature maps of a plurality of images by the one or more processors and calculating a deformation vector field from combinations of the feature maps of the individual images by the one or more processors. The one or more processors can calculate the deformation vector by generating, using a first neural network, the feature maps of the plurality of images from the individual images and inputting combinations of the generated feature maps to a second neural network.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to an image processing method, an image processing device, a program, and a trained model, and in particular to an image processing technique for aligning multiple images. [Background technology]

[0002] Dynamic contrast imaging of the liver, performed using a CT (Computed Tomography) or MRI (Magnetic Resonance Imaging) device, involves injecting a contrast agent and taking multiple images at different time phases to observe changes in the density of the lesion. Because this type of examination involves taking images at three or four time phases over the course of two to three minutes, body movement can occur between each phase due to changes in breathing, etc. Body movement causes misalignment between images, making it difficult to compare the images at each time phase.

[0003] Various methods for aligning images are known, and in recent years, methods using deep learning have also been widely researched (Non-Patent Documents 1 and 2). Non-Patent Document 1 proposes a method for aligning images by using deep learning to generate a prediction model (alignment model) that outputs a deformation vector field between two images in response to the input of two images. Non-Patent Document 1 uses one image and an image artificially generated from the deformation vector field during learning, thereby eliminating the need to define a correct answer from two images. The method described in Non-Patent Document 1 employs a 3D U-net architecture as the network structure, and is structured so that the two images to be aligned are input as two channels. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] KAJ Eppenhof and JPW Pluim. “Pulmonary CT Registration through Supervised Learning with Convolutional Neural Networks.” IEEE Transactions on Medical Imaging, 38(5):1097-1105, 2019.ISSN 0278-0062. doi: 10.1109 / Tmi.2018.2878316. [Non-patent document 2] Yabo Fu, Tonghe Wang, Walte J.Curran, Tian Liu, Xiaofen Yang, “Deep Learning in Medical Image Registration: A Review”<https: / / arxiv.org / pdf / 1912.12318.pdf> Summary of the Invention [Problem to be solved by the invention]

[0005] Dynamic contrast imaging is a method in which iodine contrast is injected intravenously into the arm, followed by repeated imaging of the same area to observe changes over time. The contrast phase refers to the state a specific number of seconds after the contrast injection, and in dynamic contrast imaging of the liver, there are the arterial phase, portal venous phase (liver parenchyma phase), and equilibrium phase. For example, the arterial phase is when a large amount of contrast is flowing through the arteries. The appearance of each phase varies depending on the type of tumor. The state before the contrast injection is called non-contrast.

[0006] Generally, dynamic contrast imaging of the liver requires four imaging phases: non-contrast, arterial phase, portal venous phase, and equilibrium phase, and the image changes between these multiple time phases must be compared. Because there is a time difference between each time phase, misalignment occurs between images from different time phases. Therefore, when interpreting the images, it is necessary to align the images with different contrast conditions so that a common region of interest can be observed in the images from each time phase. This requires rapid image processing, including the alignment process.

[0007] However, 3D images such as CT or MRI images have a large amount of data, and image registration requires a large amount of computational resources. In particular, when there are a combination of images from multiple time phases, such as in dynamic contrast imaging, the more combinations of images to be registered, the greater the computational effort.

[0008] In order to realize quick response in the registration process and the subsequent property analysis process, for example, the following two approaches can be considered.

[0009] [The first approach and its challenges] As a first approach, it is possible to reduce the amount of calculation by limiting the input to a region of interest, such as the vicinity of a lesion area, within the captured image.

[0010] However, when using one of N images as a reference to align other images, if the method described in Non-Patent Document 1 is adopted, it is necessary to perform 3D U-net calculations (N-1) times for the combination of two-channel input images. Therefore, further improvement in the efficiency of the processing is required.

[0011] [The second approach and its challenges] The second approach is to align the entire image or the entire organ when saving the images taken during the examination, and then save the deformation vector field that represents the correspondence between each pixel on the image as a result of the alignment. In this case, when interpreting the images, the saved results are used to correct the misalignment.

[0012] However, this method requires that the calculation results for each combination of images be stored in advance, which poses a problem in that it requires a large amount of memory capacity to store the calculation results.

[0013] The present disclosure has been made in consideration of these circumstances, and aims to provide an image processing method, an image processing device, a program, and a trained model that can reduce the computational resources required when aligning multiple images. [Means for solving the problem]

[0014] An image processing method according to one embodiment of the present disclosure is an image processing method executed by one or more processors, and includes the one or more processors acquiring feature maps for each of a plurality of images and calculating a deformation vector field from a combination of the feature maps for each image.

[0015] The phrase "acquiring a feature map" is not limited to the case where one or more processors acquire a feature map from outside, but also includes the concept of one or more processors generating and acquiring a feature map.

[0016] According to this aspect, since a feature map is obtained for each image, even if there are multiple combinations of images to be aligned, the computational resources required to calculate the deformation vector field between images can be reduced.

[0017] In an image processing method according to another aspect of the present disclosure, one or more processors may be configured to use a first neural network to generate a feature map for each of a plurality of images, and input the combination of the feature maps generated for each image using the first neural network into a second neural network, thereby calculating a deformation vector field using the second neural network.

[0018] In an image processing method according to another aspect of the present disclosure, the first neural network may be a network that accepts an input of a single image and outputs one or more feature maps by processing the input image, and the second neural network may be a network that accepts an input of a pair of feature maps generated from each of two different images and outputs a deformation vector field between the two different images by processing the input feature map pair.

[0019] In an image processing method according to another aspect of the present disclosure, the first neural network and the second neural network may be trained models that have been trained in advance using a training image set, and the machine learning process may be performed by inputting two images into the first neural network, and then inputting the combination of feature maps of the two images into the second neural network to output a deformation vector field.

[0020] In an image processing method according to another aspect of the present disclosure, the training image set includes a plurality of different images, and one of the two images input into the first neural network during machine learning may be an image generated by modifying the other image.

[0021] In an image processing method according to another aspect of the present disclosure, a deformation field that defines the deformation may be randomly generated within a predetermined constraint range, and the deformation field applied to the deformation processing may be taken as the correct answer, and learning may be performed so that the output of the second neural network approaches the correct answer.

[0022] In the image processing method according to another aspect of the present disclosure, each of the multiple images may be a medical image.

[0023] In the image processing method according to another aspect of the present disclosure, the plurality of images may be images with different enhancement states, which include the presence or absence of contrast and time phases.

[0024] In an image processing method according to another aspect of the present disclosure, one or more processors may further analyze the aligned images using the deformation vector field and output characteristic findings representing the contrast enhancement effect of the region of interest.

[0025] In the image processing method according to another aspect of the present disclosure, the multiple images may be images taken on different days.

[0026] In the image processing method according to another aspect of the present disclosure, the multiple images may be images of different modalities.

[0027] In an image processing method according to another aspect of the present disclosure, the multiple images may be three or more images, and one or more processors may calculate a deformation vector field for each combination of a reference image and an image other than the reference image from a combination of feature maps of two images, one of the multiple images being a reference image and the other image being other than the reference image.

[0028] In an image processing method according to another aspect of the present disclosure, the one or more processors may further include accepting a designation of a point of interest in one of the plurality of images, calculating a corresponding point corresponding to the point of interest in another of the plurality of images based on the calculated deformation vector field, and displaying the image with the positions of the point of interest and the corresponding point aligned.

[0029] An image processing device according to another aspect of the present disclosure includes one or more processors and one or more memories storing programs to be executed by the one or more processors, and the one or more processors execute instructions of the programs to obtain feature maps for each of a plurality of images and calculate a deformation vector field from a combination of the feature maps for each image.

[0030] In another aspect of the image processing device of the present disclosure, one or more processors may be configured to use a first neural network to generate a feature map for each of a plurality of images, and to input a combination of the feature maps generated for each image using the first neural network into a second neural network, thereby calculating a deformation vector field using the second neural network.

[0031] A program according to another aspect of the present disclosure causes a computer to realize a function of acquiring feature maps for each of a plurality of images and a function of calculating a deformation vector field from a combination of the feature maps for each image.

[0032] In another aspect of the program of the present disclosure, the program may be configured to cause a computer to perform the following functions: generating a feature map for each image from each of a plurality of images using a first neural network; and calculating a deformation vector field using a second neural network by inputting a combination of the feature maps generated for each image using the first neural network into a second neural network.

[0033] A trained model according to another aspect of the present disclosure is a trained model that enables a computer to perform the function of calculating a deformation vector field from multiple images, the trained model including a first neural network and a second neural network, wherein the first neural network accepts input of a single image and outputs one or more feature maps by processing the input image, and the second neural network is trained to accept input of a pair of feature maps for each of two different images generated using the first neural network, and output a deformation vector field between the two different images by processing the input feature map pair. [Effects of the Invention]

[0034] According to the present disclosure, it is possible to reduce the computational resources required when aligning multiple images. [Brief explanation of the drawings]

[0035] [Figure 1] FIG. 1 is a conceptual diagram illustrating the operation of the registration model to determine the deformation vector field between two images. [Figure 2] FIG. 2 is a network structure diagram that schematically shows the network structure of the registration model used in the image processing method according to the first embodiment. [Figure 3] FIG. 3 is an explanatory diagram of a process for aligning image B and image C with image A. In FIG. [Figure 4] FIG. 4 is a network structure diagram of the registration model according to the second embodiment. [Figure 5] FIG. 5 is a diagram showing the network structure of the registration model according to the third embodiment. [Figure 6] FIG. 6 is a block diagram showing a configuration example of a medical information system to which an image processing device according to an embodiment of the present disclosure is applied. [Figure 7] FIG. 7 is a block diagram schematically illustrating an example of the hardware configuration of an image processing device. [Figure 8] FIG. 8 is an explanatory diagram showing an overview of an application example 1 of image processing using an image processing device. [Figure 9] FIG. 9 is a flowchart of the process of aligning the regions of interest in the dynamic contrast-enhanced CT examination of the liver shown in FIG. [Figure 10] FIG. 10 is a flowchart showing an example of a subroutine applied to step S103 in FIG. [Figure 11] FIG. 11 is a diagram showing an overview of a learning method by a machine learning device for generating a registration model, and shows the configuration of a processing unit that generates training data. [Figure 12]FIG. 12 is a diagram showing an overview of a learning method by a machine learning device for generating a registration model, and shows the configuration of a processing unit that trains a learning model using training data. [Figure 13] FIG. 13 is an explanatory diagram schematically illustrating the learning phase of the registration model shown in FIG. [Figure 14] FIG. 14 is an explanatory diagram showing an outline of an application example 2 of image processing using an image processing device. [Figure 15] FIG. 15 is a flowchart of the alignment process applied to the temporal comparison shown in FIG. 14, and shows an example of the process when the image is saved. [Figure 16] FIG. 16 is a flowchart of the registration process applied to the temporal comparison shown in FIG. 14, and shows an example of the process during image interpretation. [Figure 17] FIG. 17 is an explanatory diagram that schematically shows the learning phase of the registration model that is applied to the temporal comparison shown in FIG. [Figure 18] FIG. 18 is an explanatory diagram showing an outline of an application example 3 of image processing using an image processing device. [Figure 19] FIG. 19 is an explanatory diagram that schematically shows the learning phase of the registration model that is applied to the inter-modality image comparison shown in FIG. DETAILED DESCRIPTION OF THE INVENTION

[0036] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings.

[0037] Overview of image processing method according to first embodiment The alignment of two images is achieved by calculating the deformation vector field between the two images, which is a space of deformation vectors that match any point on the deformed image with the corresponding point on the target image.

[0038] 1 is a conceptual diagram showing the operation of a registration model 10 that calculates a deformation vector field between two images. The registration model 10 is a machine learning model configured as computer software (program). The registration model 10 is configured using, for example, a convolutional neural network, and is a trained model that has been trained to output a deformation vector field in response to input of two images to be registered.

[0039] In this embodiment, a neural network having a network structure as shown in FIG. 2 is employed as the registration model 10. FIG. 2 is a network structure diagram that schematically illustrates the network structure of the registration model 101 used in the image processing method according to the first embodiment. Here, an example is shown in which a deformation vector field between two images, image A and image B, is calculated. Images A and B are three-dimensional images captured using, for example, a CT device. The three-dimensional image here includes the concept of a collection of two-dimensional slice images captured consecutively. Images A and B may be three-dimensional images reconstructed from three-dimensional data obtained by capturing two-dimensional slice tomographic images consecutively.

[0040] For comparison, the following description will be made in comparison with the structure of the neural network described in FIG. 2 of Non-Patent Document 1. The neural network described in FIG. 2 of Non-Patent Document 1 employs a 3D U-net architecture that accepts two images as inputs as two channels.

[0041] In contrast, in the image processing method according to this embodiment, a neural network that calculates a deformation vector field from two images is configured by dividing it into a portion common to each image and an individual portion. That is, as shown in Fig. 2, the registration model 101 according to this embodiment includes a first neural network NN1 that is applied in common to each image to be registered, and a second neural network NN2 that receives a combination of outputs from the first neural network NN1.

[0042] The first neural network NN1 is a network that accepts an input of one image and outputs a feature map of the input image. The first neural network NN1 functions as a feature extractor that extracts features from the input image. The second neural network NN2 is a network that accepts an input of a combination of feature maps for two images generated using the first neural network NN1 and outputs a deformation vector field between the two images in response to these inputs. The second neural network NN2 functions as a deformation vector field calculator that calculates a deformation vector field from the input combination of feature maps.

[0043] The first neural network NN1 shown in Figure 2 has a 3D U-net type architecture. The numbers in the boxes in the figure indicate the number of channels. The first neural network NN1 has one input channel, and one image is input as one channel, which differs from the two-channel input configuration described in Non-Patent Document 1.

[0044] The right-pointing solid arrows between the squares labeled with the channel numbers in the figure represent a process that includes 3D convolution using a 3x3x3 filter and computation using the Leaky Rectified Linear Unit (LReLU) activation function. The downward arrows in the figure represent max pooling using a 2x2x2 filter. The two squares at the tip of the right-pointing dashed arrow in the figure represent channel combinations. The upward arrows in the figure represent upscaling using a 2x2x2 filter, convolution using a 3x3x3 filter, and computation using LReLU. The right-pointing dashed arrow in the final stage of the second neural network NN2 (processing that reduces 32 channels to 3 channels) represents convolution using a 1x1x1 filter. The three channels obtained as the output of the second neural network NN2 correspond to the x, y, and z components of the deformation vector field.

[0045] 2 illustrates two networks: a first neural network NN1 that receives an input of image A and outputs a feature map A for image A, and a first neural network NN1 that receives an input of image B and outputs a feature map B for image B. These two first neural networks NN1 are the same (common) network that shares weights (network parameters). The processing for each image using the first neural network NN1 may be performed in parallel or concurrently, or sequentially.

[0046] In Figure 2, a pair of feature map A output from the first neural network NN1 by inputting image A into the first neural network NN1 and feature map B output from the first neural network NN1 by inputting image B into the first neural network NN1 are input into a second neural network NN2, and a deformation vector field between images A and B is output from the second neural network NN2.

[0047] The data representation of the image input to the first neural network NN1 may be three-dimensional data in the space W×H×D, where W is the number of pixels in the X-axis direction, H is the number of pixels in the Y-axis direction, and D is the Z-axis direction. shaft represents the number of pixels in the direction. W, H, and D can each be set to any value. W×H×D may be, for example, 128×128×128, or 512×512×512. The representation of the deformation vector field output from the second neural network NN2 may be in the same space W×H×D as image A and image B.

[0048] As shown in Figure 2, the network of the alignment model 101 has a network structure divided into a first neural network NN1 that accepts each of the two images to be aligned as a single channel input and performs feature extraction on a per-image basis, and a second neural network NN2 that accepts input of a combination of feature maps extracted from each image using the first neural network NN1 and calculates a deformation vector field between the images, and the first neural network NN1 and the second neural network NN2 can be calculated separately.

[0049] [Regarding alignment between three images including image C] While Fig. 1 illustrates the case of aligning images A and B, when aligning images A and C, similarly to image B, image C is input to the first neural network NN1, and the first neural network NN1 outputs a feature map C corresponding to image C. Then, the combination of feature map A and feature map C is input to the second neural network NN2, and a deformation vector field is output from the second neural network NN2 in response to the input combination of feature map A and feature map C.

[0050] 3 is an explanatory diagram showing an overview of processing when aligning image B and image C with image A using the image processing method according to this embodiment. Alignment processing unit 110 shown in FIG. 3 is an image processing unit to which alignment model 101 described in FIG. 2 is applied. Alignment processing unit 110 includes feature extraction unit 111 configured using a first neural network NN1 and deformation vector field calculation unit 112 configured using a second neural network NN2.

[0051] In the image processing method of this embodiment, as shown in Fig. 3, feature extraction processing is performed using a first neural network NN1 for each of images A, B, and C, and feature maps A, B, and C are generated for each image. That is, images A, B, and C are each input to the first neural network NN1, and calculations are performed for each image using the first neural network NN1. Thereafter, a combination of feature map A and feature map B and a combination of feature map A and feature map C are input to a second neural network NN2, and calculations are performed using the second neural network NN2 using the combinations of feature maps.

[0052] As a result, a deformation vector field BA is output from the second neural network NN2 to which the combination of feature map A and feature map B is input, and a deformation vector field CA is output from the second neural network NN2 to which the combination of feature map A and feature map C is input.

[0053] When aligning image B and image C with image A as a reference image, the method described in Non-Patent Document 1 requires calculations of the entire network for each image pair: the combination of image A and image B, and the combination of image A and image C.

[0054] In contrast, according to this embodiment, by calculating feature map A for image A, which serves as the reference for alignment, once, the calculation result (feature map A) can be combined with feature map B and feature map C, respectively, and used as input to the second neural network NN2, thereby determining deformation vector fields BA and CA. This makes it possible to reduce the amount of calculation required for a pair of two images compared to the method described in Non-Patent Document 1.

[0055] The same applies to the case where four or more images are to be aligned, and according to this embodiment, the amount of calculation required for a pair of two images to be aligned can be reduced.

[0056] Second Embodiment Fig. 4 is a network structure diagram of the alignment model 102 according to the second embodiment. The network structure shown in Fig. 4 may be adopted instead of the configuration described in Fig. 2. The drawing description rules in Fig. 4 are the same as those in Fig. 2. Differences between the alignment model 102 shown in Fig. 4 and Fig. 2 will be described below.

[0057] The alignment model 102 includes a first neural network NN1 and a second neural network NN2 having the network structure shown in FIG. 4, instead of the first neural network NN1 and the second neural network NN2 having the network structure described in FIG. 2.

[0058] The first neural network NN1 shown in FIG. 4 has a network structure corresponding to the first half of the encoder section (downsampling section) in the 3D U-net network described in FIG. 2. The first neural network NN1 shown in FIG. 4 accepts one input image and outputs multiple feature maps from the input image. The feature maps output from the first neural network NN1 shown in FIG. 4 include a first feature map with 32 channels, a second feature map with 64 channels, a third feature map with 128 channels, a fourth feature map with 256 channels, and a fifth feature map with 512 channels. That is, the first neural network NN1 in the registration model 102 accepts an input image A and outputs a set of feature maps including these multiple types of feature maps. Similarly, the first neural network NN1 accepts an input image B and outputs a set of feature maps corresponding to image B.

[0059] The second neural network NN2 in the registration model 102 has a network structure equivalent to the latter half of the decoder part (upsampling part) in the 3D U-net type network shown in FIG. 2 of Non-Patent Document 1. This second neural network NN2 accepts input of a combination of sets of feature maps generated for each image using the first neural network NN1 shown in FIG. 4, and calculates a deformation vector field between the two images from the input combination of sets of feature maps.

[0060] The second neural network NN2 shown in FIG. 4 receives a combination of a set of feature maps for image A and a set of feature maps for image B, and outputs a deformation vector field between images A and B.

[0061] Although not shown, the same applies to the case where image B and image C are aligned with image A, where image C is input to a first neural network NN1, and a set of feature maps corresponding to image C is output from the first neural network NN1. Then, a combination of the set of feature maps for image A and the set of feature maps for image C is input to a second neural network NN2, and a deformation vector field between the two images, image A and image C, is output from the second neural network NN2. The same applies to the case where four or more images are aligned, and according to this embodiment, the amount of calculation required to find the deformation vector field between images for a combination of multiple images to be aligned can be reduced.

[0062] Third Embodiment Fig. 5 is a network structure diagram of the alignment model 103 according to the third embodiment. The network structure shown in Fig. 5 may be adopted instead of the configuration described in Fig. 2. The drawing description rules in Fig. 5 are the same as those in Fig. 2. Differences between the alignment model 103 shown in Fig. 5 and the configurations shown in Figs. 2 and 4 will be described below.

[0063] The alignment model 103 includes a first neural network NN1 and a second neural network NN2 having the network structure shown in FIG. 5, instead of the first neural network NN1 and the second neural network NN2 having the network structure described in FIG. 2.

[0064] The first neural network NN1 shown in Fig. 5 may have a network structure similar to that of the first neural network NN1 shown in Fig. 4. The first neural network NN1 shown in Fig. 5 accepts an input of one image and outputs a 512-channel feature map from the input image. The representation of the feature map output by this first neural network NN1 is in a 1x1x1 space.

[0065] The first neural network NN1 of the registration model 103 outputs a feature map A in response to an input of image A. Furthermore, this first neural network NN1 outputs a feature map B in response to an input of image B. FIG. 5 shows an example in which a pair of feature map A output from the first neural network NN1 by inputting image A to the first neural network NN1 and feature map B output from the first neural network NN1 by inputting image B to the first neural network NN1 are input to the second neural network NN2.

[0066] The second neural network NN2 in the registration model 103 accepts as input a combination of 512-channel feature maps in a 1x1x1 space and calculates a deformation vector field between two images based on these inputs. The representation of the deformation vector field output from this second neural network NN2 is in the same 1x1x1 space as the input. In this case, the deformation vector field corresponds to a deformation vector. In other words, the representation of the feature maps and deformation vector field includes the case of a 1x1x1 space.

[0067] 5, a combination of feature map A and feature map B is input to the second neural network NN2, and the second neural network NN2 outputs a deformation vector field between images A and B. Although not shown, the same applies when aligning three or more images including image C. According to this embodiment, the amount of calculation required to find a deformation vector field between images for a combination of multiple images to be aligned can be reduced.

[0068] <<Example of medical information system configuration>> 6 is a block diagram showing an example configuration of a medical information system 200 to which an image processing device 220 according to an embodiment of the present disclosure is applied. The registration model 101, 102, or 103 described as each of the first to third embodiments is incorporated into the image processing device 220.

[0069] The medical information system 200 is realized as a computer network established in a medical institution such as a hospital. The medical information system 200 includes an electronic medical record system 202, a CT apparatus 204, an MRI apparatus 206, an image storage server 210, an image processing apparatus 220, and a viewer terminal 230, and these elements are connected via a communication line 240. The communication line 240 may be an in-house communication line within the medical institution. Furthermore, part of the communication line 240 may include a wide-area communication line. Some of the elements of the medical information system 200 may be configured using cloud computing.

[0070] 6 illustrates a CT apparatus 204 and an MRI apparatus 206 as examples of modalities, but the apparatuses for capturing medical images are not limited to the CT apparatus 204 and the MRI apparatus 206, and various other examination apparatuses may be used, such as an ultrasound diagnostic apparatus, a PET (Positron Emission Tomography) apparatus, a mammography apparatus, an X-ray diagnostic apparatus, an X-ray fluoroscopic diagnostic apparatus, and an endoscope apparatus, all of which are not shown. The types and numbers of modalities connected to the communication line 240 may be combined in various ways for each medical institution.

[0071] The image storage server 210 may be, for example, a DICOM (Digital Imaging and Communications in Medicine) server that operates in accordance with the DICOM specifications. The image storage server 210 is a computer that stores and manages various data, including images captured using various modalities such as the CT device 204 and the MRI device 206, and is equipped with a large-capacity external storage device and a database management program. The image storage server 210 communicates with other devices via a communication line 240 to send and receive various data, including image data. The image storage server 210 receives various data, including images generated by modalities such as the CT device 204, via the communication line 240, and stores and manages the data on a recording medium such as a large-capacity external storage device. The storage format of the image data and communication between devices via the communication line 240 are based on the DICOM protocol.

[0072] For example, when a dynamic contrast imaging examination of the liver is performed on a patient using the CT device 204, multiple images obtained by the imaging, including non-contrast images, arterial phase images, portal venous phase images, and equilibrium phase images, are stored in the image database 212 of the image storage server 210.

[0073] The image processing device 220 can acquire data from the image storage server 210 or the like via a communication line 240. The image processing device 220 can be realized using computer hardware and software. The form of the image processing device 220 is not particularly limited, and may be a server computer, a workstation, a personal computer, a tablet terminal, or the like. The image processing device 220 may include an input device 222 and a display device 224.

[0074] The input device 222 may be, for example, a keyboard, a mouse, a multi-touch panel, or other pointing device, or a voice input device, or an appropriate combination of these. The display device 224 is an output interface on which various information is displayed. The display device 224 may be, for example, a liquid crystal display, an organic electro-luminescence (OEL) display, a projector, or an appropriate combination of these. The input device 222 and the display device 224 may be integrated into one unit, such as a touch panel. The input device 222 and the display device 224 may be included in the image processing device 220, or the image processing device 220, the input device 222, and the display device 224 may be integrated into one unit.

[0075] The image processing device 220 performs image analysis and various other processes on medical images captured by a modality. In addition to the processing function of aligning images, the image processing device 220 may be configured to perform various analytical processes such as computer-aided diagnosis (CAD), for example, a process for recognizing a lesion area from an image, a process for identifying a disease classification, or a segmentation process for recognizing an organ area. The image processing device 220 may also include a processing module that supports the creation of an interpretation report. The image processing device 220 can send the results of the image processing to the image storage server 210 and the viewer terminal 230. Some or all of the processing functions of the image processing device 220 may be incorporated into the image storage server 210 or the viewer terminal 230.

[0076] Various types of data stored in the image database 212 of the image storage server 210 and various information including the processing results generated by the image processing device 220 can be displayed on the display device 234 of the viewer terminal 230.

[0077] The viewer terminal 230 may be a terminal for viewing images called a PACS (Picture Archiving and Communication Systems) viewer or a DICOM viewer. While one viewer terminal 230 is illustrated in FIG. 6 , multiple viewer terminals 230 may be connected to the communication line 240. The form of the viewer terminal 230 is not particularly limited, and may be a personal computer, a workstation, a tablet terminal, or the like. The viewer terminal 230 includes an input device 232 and a display device 234. The input device 232 and the display device 234 may have the same configuration as the input device 222 and the display device 224 of the image processing device 220.

[0078] <<Example of Hardware Configuration of Image Processing Device 220>> 7 is a block diagram showing an example of the hardware configuration of the image processing device 220. The image processing device 220 can be realized by a computer system configured using one or more computers. Here, an example will be described in which various functions of the image processing device 220 are realized by one computer executing a program.

[0079] The image processing device 220 includes a processor 302 , a non-transitory tangible computer-readable medium 304 , a communication interface 306 , an input / output interface 308 , and a bus 310 .

[0080] The processor 302 includes a CPU (Central Processing Unit). The processor 302 may include a GPU (Graphics Processing Unit). The processor 302 is connected to a computer-readable medium 304, a communication interface 306, and an input / output interface 308 via a bus 310. The processor 302 reads various programs, data, etc. stored in the computer-readable medium 304 and executes various processes. The term "program" includes the concept of a program module and includes instructions equivalent to a program.

[0081] The computer-readable medium 304 is, for example, a storage device including a memory 322 serving as a main storage device and a storage 324 serving as an auxiliary storage device. The storage 324 is configured using, for example, a hard disk drive (HDD) device, a solid state drive (SSD) device, an optical disk, a magneto-optical disk, or a semiconductor memory, or an appropriate combination of these. The storage 324 stores various programs, data, and the like.

[0082] The memory 322 is used as a working area for the processor 302, and as a storage unit that temporarily stores programs and various data read from the storage 324. The programs stored in the storage 324 are loaded into the memory 322, and the processor 302 executes the instructions of the programs, causing the processor 302 to function as a means for performing various processes defined by the programs. The memory 322 stores programs such as a registration processing program 330, a corresponding point calculation program 340, a property analysis program 350, and a display control program 360 that are executed by the processor 302, as well as various data.

[0083] The registration processing program 330 includes the registration model 101, 102, or 103 described with reference to Figures 2 to 5. When the processor 302 executes the instructions of the registration processing program 330, the processor 302 functions as a feature extraction unit 332 and a deformation vector field calculation unit 334. The corresponding point calculation program 340 is a program that executes a process for finding corresponding points in the images to be compared, using the deformation vector field calculated by the deformation vector field calculation unit 334.

[0084] The characterization analysis program 350 is an example of a CAD module that detects regions such as lesions within an image and performs lesion characterization. The characterization analysis program 350 may be, for example, a program that performs liver tumor characterization analysis from dynamic contrast-enhanced CT images of the liver. The characterization analysis program 350 may be configured using a trained model trained by machine learning to output the desired characterization analysis processing results from input images. The characterization analysis program 350 analyzes aligned images of multiple time phases using the deformation vector field calculated by the deformation vector field calculation unit 334 and outputs characterization findings that represent the contrast enhancement effect of the region of interest. The image processing device 220 is not limited to the characterization analysis program 350 and may also include other CAD modules, such as an organ recognition program and a lesion detection program (not shown).

[0085] The display control program 360 generates a display signal required for display output to the display device 224 and controls the display of the display device 224 .

[0086] The communication interface 306 performs communication processing with external devices via wired or wireless connections, and exchanges information with the external devices. The image processing device 220 is connected to the communication line 240 via the communication interface 306, and is capable of exchanging data with devices such as the image storage server 210 and the viewer terminal 230. The communication interface 306 can serve as a data acquisition unit that accepts input of data such as images.

[0087] The input device 222 and the display device 224 are connected to the bus 310 via the input / output interface 308 .

[0088] <Application Example 1> Fig. 8 is an explanatory diagram showing an overview of application example 1 of image processing using the image processing device 220. Fig. 8 shows an example of registration processing of a region of interest (ROI) in a dynamic contrast-enhanced CT examination of the liver. Here, an example using a registration model 130 having the network structure (see Fig. 5) explained in the third embodiment will be described.

[0089] When a dynamic contrast-enhanced CT examination of the liver is performed on a patient, images of multiple time phases taken using the CT device 204 are stored in the image storage server 210. A doctor in charge of interpreting the images can observe the images of each time phase using the viewer terminal 230. Three images A, B, and C shown on the far left of Figure 8 are examples of CT images with different contrast enhancement states. Images A, B, and C are examples of "medical images" in this disclosure. Although three images are shown in Figure 8, four or more images may be present. Below, the processing procedure by the image processing device 220 will be described with specific examples.

[0090] In step 0, a point of interest is designated on an image of one of the time phases. A doctor observes the images while one or more of the images of the multiple time phases are displayed on the display device 234 of the viewer terminal 230, and if a region suspected of being a lesion such as a liver tumor is found, the doctor can input to designate the point of interest. This input operation to designate the point of interest can be performed using the input device 222. Of the images of the multiple time phases, the image on which the point of interest is designated serves as the reference image for alignment. FIG. 8 shows an example in which a point of interest is designated on image A, and image A serves as the reference image. For example, image A may be an image of the arterial phase, image B may be an image of the portal vein phase, and image C may be an image of the equilibrium phase. Although not shown in FIG. 8, image D (e.g., a non-contrast image) may also be included.

[0091] When a point of interest is specified, the image processing device 220 performs a process of setting a temporary corresponding point in each image and cutting out the surrounding area as an ROI image in step 1. For image A, which is a reference image in which a point of interest is specified, the surrounding area including the point of interest is cut out as an ROI image based on the point of interest. For example, the image processing device 220 cuts out an image area of ​​a predetermined size around the point of interest as the ROI image. The image size cut out as the ROI image may be a predetermined size or may be an arbitrarily specified or selected size. The ROI image cut out from image A is denoted as ROI(A).

[0092] For images other than the reference image, such as image B and image C, the image processing device 220 uses the DICOM coordinates of the point of interest to set a tentative corresponding point corresponding to the point of interest, and based on the tentative corresponding point, extracts the area around the tentative corresponding point as an ROI image. Here, the DICOM coordinates refer to position information obtained from, for example, "Image Position (Patient)" of tag number (0020,0032) included in the DICOM header information. The ROI image extracted from image B is referred to as ROI(B), and the ROI image extracted from image C is referred to as ROI(C).

[0093] Next, in step 2, the image processing device 220 performs a process of calculating the amount of misalignment between the images from the combination of ROI images generated in step 1. The process of step 2 is performed using the registration model 130. By inputting ROI(A) into the first neural network NN1, a feature map FM(A) of ROI(A) is generated. Similarly, by inputting ROI(B) and ROI(C) into the first neural network NN1, a feature map FM(B) of ROI(B) and a feature map FM(C) of ROI(C) are generated.

[0094] By inputting the combination of feature map FM(A) and feature map FM(B) generated by the first neural network NN1 into the second neural network NN2, the second neural network NN2 calculates a deformation vector field between the images of ROI(A) and ROI(B), in this case a deformation vector (dxB, dyB, dzB) indicating the amount of deviation.

[0095] Similarly, by inputting the combination of feature maps FM(A) and FM(C) generated by the first neural network NN1 into the second neural network NN2, the second neural network NN2 calculates a deformation vector (dxC, dyC, dzC) that indicates the amount of displacement between the images of ROI(A) and ROI(C). In this way, the amount of displacement between multiple ROI images can be calculated.

[0096] The image processing device 220 can perform various optional processes using the amount of deviation calculated using the registration model 103. For example, in step 3 shown in Fig. 8, the amount of deviation is used to find a corresponding point corresponding to the point of interest, and the image is displayed with the positions of the point of interest and the corresponding point aligned. As a display mode, for example, the images are displayed so that the point of interest or the corresponding point of each image coincides with the center of the window displaying each image.

[0097] The image processing device 220 calculates a corresponding point CP(B) of the point of interest in image B based on a deformation vector indicating the amount of displacement between the images ROI(A) and ROI(B), displays image A so that the point of interest coincides with the center of the display window of image A, and displays image B so that the corresponding point CP(B) coincides with the center of the display window of image B. Similarly, the image processing device 220 calculates a corresponding point CP(C) of the point of interest in image C based on a deformation vector indicating the amount of displacement between the images ROI(A) and ROI(C), and displays image C so that the corresponding point CP(C) coincides with the center of the display window of image C.

[0098] In addition to the process of displaying images with the positions of the attention point and the corresponding point aligned, the image processing device 220 may also perform a process of displaying annotations indicating the positions of the corresponding points on each image, as shown in Figure 8.

[0099] For example, the image processing device 220 can calculate a corresponding point CP(B) of the point of interest in image B based on a deformation vector indicating the amount of displacement between the images of ROI(A) and ROI(B), and superimpose and display information indicating the position of the corresponding point CP(B) or a mark indicating the corresponding point CP(B) on the image of image B. Furthermore, the image processing device 220 can calculate a corresponding point CP(C) of the point of interest in image B based on a deformation vector indicating the amount of displacement between the images of ROI(A) and ROI(C), and superimpose and display information indicating the position of the corresponding point CP(C) or a mark indicating the corresponding point CP(C) on the image of image C. Such calculation and display processing of corresponding points is performed using the corresponding point calculation program 340.

[0100] Furthermore, instead of or in addition to the processing of step 3, the image processing device 220 may perform, as step 4, image analysis of regions of interest (ROIs) of multiple images aligned using the shift amount, and output characteristic findings representing contrast enhancement. The characteristic findings may include, for example, classification of contrast enhancement related to multiple time phases, such as early enhancement and washout. The image processing device 220 may be configured to perform image analysis using a trained model trained by machine learning to output a classification of characteristic findings from input images of multiple time phases. Such characteristic analysis processing is performed using the characteristic analysis program 350.

[0101] Fig. 9 is a flowchart of the ROI alignment process in the dynamic contrast-enhanced CT examination of the liver shown in Fig. 8. In step S101, the processor 302 of the image processing device 220 accepts designation of a point of interest in an image of one of the time phases among a group of images of multiple time phases.

[0102] When the point of interest is designated, in step S102, the processor 302 sets a tentative corresponding point in an image other than the reference image in which the point of interest is designated, and cuts out the area around the point of interest or the tentative corresponding point from each image as an ROI image.

[0103] Next, in step S103, the processor 302 uses the registration model 103 to calculate the amount of deviation from the combination of ROI images.

[0104] Then, in step S104, processor 302 uses the calculated amount of deviation to find a corresponding point corresponding to the point of interest in an image other than the reference image, and displays the image with the positions of the point of interest and the corresponding point aligned. Processor 302 may also display information indicating the positions of the corresponding points together with the image. After step S104, processor 302 ends the flowchart of Fig. 9. Note that processor 302 may return to step S101 after step S104, and repeat steps S101 to S104 in response to an input specifying a point of interest.

[0105] Fig. 10 is a flowchart showing an example of a subroutine applied to step S103 in Fig. 9. In step S111, the processor 302 inputs each of the ROI images extracted from each of the images in the multiple time phases into the first neural network NN1, and generates a feature map for each ROI image.

[0106] In step S112, the processor 302 inputs a pair of feature maps FM(A) generated from ROI(A) and FM(B) generated from ROI(B) into a second neural network NN2, and calculates the amount of deviation dfB between the images of ROI(A) and ROI(B).

[0107] Similarly, in step S113, the processor 302 inputs a pair of the feature map FM(A) generated from ROI(A) and the feature map FM(C) generated from ROI(C) into the second neural network NN2, and calculates the amount of displacement dfC between the images of ROI(A) and ROI(C). Although not shown in Fig. 10, when an image D is included, the processor 302 similarly inputs a pair of the feature map FM(A) generated from ROI(A) and the feature map FM(D) generated from ROI(D) into the second neural network NN2, and calculates the amount of displacement dfD between the images of ROI(A) and ROI(D).

[0108] After step S113, the processor 302 ends the flowchart in FIG. 10 and returns to the flowchart in FIG.

[0109] [Characteristic analysis of liver tumors and generation of findings] As a further optional process (step 4) that can be performed by the image processing device 220, the image processing device 220 may perform a process of analyzing the characteristics of the contrast effect by comparing multiple images of the region of interest, and generating and presenting a finding statement to be written in the radiology report based on the analysis results. The technology for generating a finding statement from multiple findings that represent the characteristics (features) of the region of interest can be, for example, the technology described in International Publication WO2020 / 209382.

[0110] When the position of a tumor is specified (clicked) on an image of the arterial phase among images of multiple time phases, the image processing device 220 extracts an ROI based on the specified tumor position and aligns the images of each time phase, and performs a characteristic analysis of the specified tumor based on the ROI images of multiple time phases. As a result of the characteristic analysis based on image analysis, analysis results such as "Boundary: clear," "Margin: smooth," "Early enhancement: +," "Washout: +," "Contrast effect: heterogeneous," "Late enhancement: -," "Margin enhancement: -," "Ring-shaped: -," "Capsule formation: +," "Fatty degeneration: +," "Location: S8," "Size: 42 mm," etc. are obtained.

[0111] The finding generation program selects and selects information to be included in the radiology report from the analysis results obtained by the characteristic analysis, and automatically generates candidate findings. For example, based on the analysis results shown above, the image processing device 220 incorporating the finding generation program may generate a finding such as, "A 42mm mass with a smooth margin and clear contours is observed in S8. Heterogeneous early staining is observed, accompanied by washout. A capsule-like structure is also observed. Fat components are also present." The process of generating such findings is realized, for example, using a machine learning model that employs a neural network architecture, such as Transformer.

[0112] [Example of a training method for generating a registration model] Here, an example of a learning method for generating the alignment model 132 will be described. Fig. 11 and Fig. 12 show an overview of the learning method by the machine learning device 400 applied to this embodiment. Fig. 11 shows the configuration of a processing unit that generates training data (hereinafter referred to as a training data generation unit), and Fig. 12 shows the configuration of a processing unit that trains a learning model using the generated training data (hereinafter referred to as a learning processing unit). "Training" is synonymous with learning.

[0113] Normally, in the case of images of multiple time phases actually taken using a modality such as the CT scanner 204, the correct deformation vector field between two images with different contrast conditions is not specified, and it is difficult to obtain the correct deformation vector field between two images to be compared. For this reason, it is difficult to prepare the large amount of training data required for machine learning using only actual images.

[0114] Therefore, in the learning method of the registration model 130 of this embodiment, pairs of training images are artificially generated based on actually captured images, and a deformation vector field that defines the deformation transformation used in generating the pairs is used as a teacher signal of the correct answer. As a method of such data augmentation, a method similar to the method described in Non-Patent Document 1 can be applied.

[0115] 11, the training data generation unit in the machine learning device 400 includes a cropping unit 402, data augmentation and conversion units 404 and 405, and a random transformation unit 406. The machine learning device 400 can be realized by a combination of computer hardware and software.

[0116] The cropping unit 402 performs a process of cropping a partial image area from an original training image TI, which is an actually captured 3D image, and resizing it to a predetermined size. The cropping position by the cropping unit 402 may be changed randomly. The data augmentation conversion unit 404 applies a known deformation transformation to the cropped image TI(x) cropped by the cropping unit 402 to perform image conversion for data augmentation, thereby generating an artificial augmented training image TIa(x).

[0117] The data augmentation conversion unit 405 performs image conversion by applying the same transformation function as the data augmentation conversion unit 404. In Fig. 11, the data augmentation conversion unit 404 and the data augmentation conversion unit 405 are illustrated as separate processing units, but they may be the same unit, and the augmented training image TIa(x) generated by the data augmentation conversion unit 404 may be input to the random transformation processing unit 406.

[0118] The random deformation processor 406 performs image deformation using a deformation vector field U(x) randomly generated within a predetermined constraint range. The "constraint range" here includes, for example, the type of algorithm applied to the deformation, the amount of deformation, the range of the deformation area, and other numerical ranges of various deformation parameters. The random deformation processor 406 uses the deformation vector field U(x) on the augmented training image TIa(x) generated by the data augmentation converter 404 to generate an artificially deformed augmented deformation training image TId(x). The three-dimensional random deformation performed by the random deformation processor 406 may be a combination of rigid and non-rigid deformations. The deformation vector field U(x) that defines the deformation performed by the random deformation processor 406 is an example of a "deformation field" in this disclosure. Note that while FIG. 11 illustrates the data augmentation converter 405 and the random deformation processor 406 separately, these processes may be combined into a transformation processor that performs both data augmentation and random deformation simultaneously.

[0119] In this way, training data including a pair of an augmented training image TIa(x) and an augmented transformed training image TId(x), as well as a ground-truth transformation vector field U(x) between these images, can be generated from one training image TI. By varying the combination of the cropping position by the cropping unit 402, the transformation functions applied to the data augmentation conversion units 404 and 405, and the transformation vector field U(x) applied to the random transformation unit 406, multiple training data can be generated from one training image TI. By preparing a learning image set including multiple training images TI and applying the process shown in FIG. 11 to each training image TI, a dataset including a large amount of training data required for machine learning can be obtained.

[0120] It is also possible to omit the crop processing unit 402 shown in Figure 11, or to omit the data extension conversion units 404 and 405, or to omit the crop processing unit 402 and the data extension conversion units 404 and 405.In any case, by applying the processing of the random transformation processing unit 406 to the training image TI, a pair of an image before transformation and an image after transformation can be obtained.

[0121] The machine learning device 400 may generate training data on the fly during the learning process, or may generate training data in advance prior to the learning process to prepare a data set required for training.

[0122] 12, the machine learning device 400 includes a learning model 410 and an optimizer 420. When generating the alignment model 130, the network structure of the learning model 410 has the same configuration as the network structure described in FIG.

[0123] The augmented training image TIa(x) and the augmented deformed training image TId(x) are each input to a first neural network NN1 of the learning model 410, and their respective feature maps are input to a second neural network NN2, which outputs a deformed vector field u(x). In the case of the learning model 410 having the network structure described in Fig. 5, the deformed vector field u(x) is expressed in the space 1x1x1.

[0124] The optimizer 420 determines the amount of update for the parameters of the learning model 410 based on the calculation result of the loss indicating the error between the output of the learning model 410 and a teacher signal, so that the deformation vector field u(x) output by the learning model 410 approaches the correct deformation vector field U(x), and performs parameter update processing for the learning model 410. The optimizer 420 updates the parameters based on an algorithm such as gradient descent. Note that the parameters of the learning model 410 include filter coefficients (weights of connections between nodes) of filters used in processing each layer of the neural network and node biases. The machine learning device 400 may acquire data and update parameters in units of mini-batches, which are groups of multiple training data.

[0125] In this way, a learning process is performed using a large amount of training data, whereby the parameters of the learning model 410 are optimized and a registration model 130 with the desired performance is generated.

[0126] Fig. 13 is an explanatory diagram schematically illustrating the learning phase of the registration model 130. Images IM1c and IM1a shown in the upper left of Fig. 13 represent cross sections of image TI1, which is a three-dimensional image for training, with image IM1c being a coronal image and image IM1a being an image (axial image) of a cross section taken along line AA of image IM1c. Rectangular frames BB1 shown in images IM1a and IM1c represent ROIs randomly extracted from training image TI1. The "x" marks shown in images IM1a and IM1c represent positions corresponding to points of interest.

[0127] A training image TI2 is generated by applying three-dimensional random deformation to this training image TI1. Images IM2c and IM2a shown in the lower left of FIG. 13 represent the training image TI2, with image IM2c being a coronal image and image IM2a being an axial image. Image IM2a is a cross-sectional image of image IM2c taken along line AA. Rectangular frames BB2 shown in images IM2a and IM2c represent ROIs extracted from image TI2. The position of rectangular frame BB2 corresponds to the position of rectangular frame BB1.

[0128] The ROIs randomly extracted from each of the images TI1 and TI2 are input to the first neural network NN1 of the learning model 410, and processing of the first neural network NN1 is executed for each ROI. The output of the first neural network NN1 that processes each ROI is connected to the input of the second neural network NN2, and the combination of the feature maps FM1 and FM2 of each ROI is input to the second neural network NN2, which outputs a vector (dx, dy, dz) indicating the amount of three-dimensional deformation (amount of deviation) between the ROIs.

[0129] The parameters of the learning model 410 are updated based on the difference between the deformation amount output from the learning model 410 and the correct deformation amount (gt_dx, gt_dy, gt_dy) which is the teacher signal. The correct deformation amount (gt_dx, gt_dy, gt_dy) can be calculated from a deformation vector field corresponding to the transformation function applied to the three-dimensional random deformation process.

[0130] <Application Example 2> The image registration technique of the present disclosure is not limited to registration between images at multiple time phases in a dynamic contrast imaging examination, but can be applied to a variety of uses.

[0131] Fig. 14 is an explanatory diagram showing an overview of a second application example of image processing using the image processing device 220. Fig. 14 shows an example of alignment processing for temporal comparison of liver test images. Here, an example is described in which an alignment model 132 having a network structure similar to that of the alignment model 101 (see Fig. 2) described in the first embodiment is used, but the alignment model 132 may also have a network structure similar to that of the alignment model 102 (see Fig. 4) described in the second embodiment.

[0132] When a CT examination of the liver of a patient is performed, images taken using the CT device 104 are stored in the image storage server 210. Multiple examinations may be performed on the same patient on different days (times), and changes in the patient's condition may be observed by comparing the multiple examination images taken on different days. One useful method for such a time-series comparison is to generate a feature map of the examination image using the first neural network NN1 of the registration model 132 when storing the examination image, and store the feature map together with the examination image in the image storage server 210. The processing procedure performed by the image processing device 220 will now be described with specific examples.

[0133] In step 0, the image processing device 220 detects organs such as the liver and other landmarks in the images obtained by the examination, and roughly aligns the positions of the images. Image A shown on the far left of FIG. 14 is the latest image obtained by the current examination and represents the patient's current condition. In this example, this latest image A serves as the reference image for alignment. Images B and C shown below image A represent images taken in the past of the same patient, but were taken at different times (examination dates). Images A, B, and C shown in FIG. 14 are examples of "images taken on different days" in the present disclosure. Although not shown in FIG. 14, one or more previous images, such as image D, may also be included.

[0134] The processing of step 0 is preferably performed as pre-processing for the next step 1, but is not a required process and is an optional process that can be selected whether or not to perform.

[0135] In step 1, the image processing device 220 applies the first neural network NN1 to each image obtained by performing an inspection, generates a feature map as a processing result of the first neural network NN1, associates the feature map with the image, and stores the respective feature maps in the image storage server 210. In Fig. 14, the process of applying the first neural network NN1 to each of images A, B, and C is illustrated in parallel, but these processes are performed at different times, i.e., when each image is obtained by inspection.

[0136] Thereafter, at the time of interpretation, the image processing device 220 reads out from the image storage server 210 the images to be compared and the features resulting from processing of the images by the first neural network NN1, and applies the second neural network NN2 to the pair of feature maps of the two images to be compared. The pair of feature map FM(A) of image A and feature map FM(B) of image B is input to the second neural network NN2, and a deformation vector field DVf(B) corresponding to the displacement amount map B between images A and B is output from the second neural network NN2.

[0137] In addition, when a pair of feature map FM(A) of image A and feature map FM(C) of image C is input to the second neural network NN2, the second neural network NN2 outputs a deformation vector field DVf(C) corresponding to the displacement map C between images A and C.

[0138] The image processing device 220 can perform various optional processes using the displacement map calculated using the registration model 132. For example, as shown in FIG. 14, in step 3, the image processing device 220 accepts the designation of a point of interest during image interpretation. When the point of interest is designated, the image processing device 220 refers to the displacement map to find a corresponding point corresponding to the point of interest for each past image, and performs processing to display the images with the positions of the point of interest and the corresponding point aligned. For example, the image processing device 220 displays the current image and the past image so that the point of interest or the corresponding point of each image is aligned at the center of the window displaying each image. Furthermore, as shown in FIG. 14, information (annotation) indicating the positions of the corresponding points may be displayed together with the past image.

[0139] Figures 15 and 16 are flowcharts of the registration process applied to the temporal comparison shown in Figure 14. Figure 15 is a flowchart showing an example of the process when saving an image, and Figure 16 is a flowchart showing an example of the process when interpreting an image.

[0140] 15, the processor 302 of the image processing device 220 acquires an examination image. The processor 302 may acquire the latest examination image from a modality such as the CT device 104, or may acquire the examination image from the image storage server 210.

[0141] In step S202, the processor 302 detects organs such as the liver and other landmarks from the acquired image, and specifies the approximate position of a region of interest including the region to be observed based on information about the detected landmarks.

[0142] Next, in step S203, the processor 302 inputs the acquired image into the first neural network NN1 to generate a feature map. Then, in step S204, the processor 302 associates the acquired image with the feature map, which is the processing result of the first neural network NN1, and stores them in the image storage server 210. After step S204, the processor 302 ends the flowchart in FIG. 15.

[0143] Each time a new inspection image is captured during inspection, the flowchart of FIG. 15 is executed, and the processing results of the first neural network NN1 for each inspection image are stored in advance in association with the inspection image.

[0144] During image interpretation, the flowchart in Fig. 16 is executed. In step S211, the processor 302 of the image processing device 220 reads out the target image and its feature map from the image storage server 210 in accordance with an instruction from the viewer terminal 230. Then, in step S212, the processor 302 inputs each pair of feature maps of the multiple images to be compared into the second neural network NN2.

[0145] In step S213, the processor 302 executes processing using the second neural network NN2 to generate a displacement map between images (i.e., a deformation vector field). The generated displacement map between each image may be stored in the image processing device 220 or the image storage server 210.

[0146] In step S214, the processor 302 accepts the designation of the point of interest. When the designation of the point of interest is input from the viewer terminal 230, the designation information is sent to the processor 302.

[0147] When the point of interest is designated, in step S215, the processor 302 refers to the displacement map to find a corresponding point in the previous image that corresponds to the point of interest, and displays the image with the positions of the point of interest and the corresponding point aligned. The processor 302 may also display information indicating the positions of the corresponding points together with the previous image.

[0148] After step S215, the processor 302 ends the flowchart of Fig. 16. After step S215, the processor 302 may return to step S211 and repeat steps S211 to S215 in response to an input specifying a point of interest.

[0149] 14 to 16, the processing using the first neural network NN1 and the processing using the second neural network NN2 in the registration model 132 may be performed at different times. The first neural network NN1 and the second neural network NN2 may be configured as separate processing modules capable of performing separate calculations. A system configuration in which a device for performing processing using the first neural network NN1 and a device for performing processing using the second neural network NN2 are configured as separate devices is also possible.

[0150] [Example of a training method for generating alignment model 132] Fig. 17 is an explanatory diagram that schematically shows the learning phase of the alignment model 132 that is applied to the temporal comparison shown in Fig. 16. In Fig. 17, elements that are common to Fig. 13 are given the same reference numerals, and duplicated explanations will be omitted. The network structure of the learning model 412 when generating the alignment model 132 is the network structure shown in Fig. 2 or Fig. 4.

[0151] A training image TI2 is generated by applying a three-dimensional random deformation to a training image TI1. Each of the training images TI1 and TI2 is input to a first neural network NN1 of the learning model 412, and processing of the first neural network NN1 is performed for each image. The output of the first neural network NN1 that processes each image is connected to the input of a second neural network NN2, and a combination of feature maps FM1 and FM2 generated from each image is input to the second neural network NN2, causing the second neural network NN2 to output a deformation vector field between the images.

[0152] Learning Model 41 2 The parameters of the learning model 412 are updated based on the difference between the deformation vector field output from and the correct deformation vector field, which is the teacher signal. The correct deformation vector field is a deformation vector field that corresponds to the deformation transformation function applied to the three-dimensional random deformation process.

[0153] <Application Example 3> FIG. 18 is an explanatory diagram showing an overview of an application example 3 of image processing using the image processing device 220. The image registration technique of the present disclosure can be applied to comparison between images of different modalities. FIG. 18 shows an example of registration processing for image comparison between different modalities. Here, an example is described in which a registration model 133 having a network structure similar to that of the registration model 101 (see FIG. 2) described in the first embodiment is used, but the registration model 133 may also have a network structure similar to that of the registration model 102 (see FIG. 4) described in the second embodiment. FIG. 18 shows the application of the mechanism of the time-lapse comparison processing described in FIG. 17 to comparison between images of different modalities.

[0154] Image A to be processed shown in FIG. 18 may be, for example, a CT image, image B may be a T1-weighted image captured using the MRI device 206, and image C may be a T2-weighted image. Images A, B, and C of different modalities are images of the same patient, and the images may be captured on the same or different examination dates. Images A, B, and C shown in FIG. 18 are examples of "images of different modalities" in the present disclosure. Although not shown in FIG. 18, one or more other modality images, such as image D, may also be included.

[0155] In step 0, the image processing device 220 detects organs such as the liver and other landmarks in the images obtained by performing the examination, and roughly aligns the positions of the images.

[0156] In step 1, the image processing device 220 applies the first neural network NN1 of the registration model 133 to each image, and generates a feature map as a processing result of the first neural network NN1.

[0157] The processing of steps 0 and 1 may be performed when the image is saved, or when the image is interpreted, as in the example of FIG.

[0158] After step 1, the image processing device 220 applies a second neural network NN2 to each pair of feature maps of the two images to be compared. A pair of feature map FM(A) of image A and feature map FM(B) of image B is input to the second neural network NN2, and a deformation vector field DVf(B) corresponding to the displacement map B between images A and B is output from the second neural network NN2.

[0159] In addition, when a pair of feature map FM(A) of image A and feature map FM(C) of image C is input to the second neural network NN2, the second neural network NN2 outputs a deformation vector field DVf(C) corresponding to the displacement map C between images A and C.

[0160] The image processing device 220 can perform various optional processes using the displacement map calculated using the registration model 133. For example, as shown in Fig. 18, in step 3, the image processing device 220 receives designation of a point of interest during interpretation, and when the point of interest is designated, it refers to the displacement map to find corresponding points corresponding to the point of interest for each image of a different modality, and performs processing to display the images with the positions of the point of interest and the corresponding points aligned. The image processing device 220 may also perform processing to display annotations indicating the positions of the corresponding points on each image.

[0161] [Example of a training method for generating alignment model 133] Fig. 19 is an explanatory diagram schematically showing the learning phase of the registration model 133 applied to the inter-modality image comparison shown in Fig. 18. The network structure of the learning model 413 when generating the registration model 133 is the network structure shown in Fig. 2 or 4.

[0162] For training, we use a training image set that contains a mixture of images from multiple modalities, such as CT images, MRI (T1 weighted) images, and MRI (T2 weighted) images. Image IM1 shown in Figure 19 is an image selected from the training image set and is the image before three-dimensional random deformation is applied.

[0163] Three-dimensional random deformation is applied to this image IM1 to generate a deformed image IM2. The three-dimensional random deformation may be a deformation process that combines rigid deformation and non-rigid deformation.

[0164] The images IM1 and IM2 thus obtained are each input to the first neural network NN1 of the learning model 413, and processing of the first neural network NN1 is executed for each image. The combination of feature maps FM1 and FM2 generated from each image by the first neural network NN1 is input to the second neural network NN2, and a deformation vector field between the images is output from the second neural network NN2.

[0165] Based on the difference between the deformation vector field output from the learning model 413 and the correct deformation vector field that is the teacher signal, the learning model 41 3 The parameters of the first neural network NN1 are updated. As a result, the first neural network NN1 is trained to extract features suitable for alignment from the input images, regardless of the image type.

[0166] About the programs that run computers A program that causes a computer to realize some or all of the processing functions of the image processing device 220 can be recorded on a computer-readable medium such as an optical disk, a magnetic disk, a semiconductor memory, or other tangible non-transitory information storage medium, and the program can be provided through this information storage medium.

[0167] In addition, instead of providing the program by storing it on such a tangible, non-transitory computer-readable medium, it is also possible to provide the program signal as a download service using a telecommunications line such as the Internet.

[0168] Furthermore, some or all of the processing functions of the image processing device 220 may be realized by cloud computing. a It can also be provided as a Software as a Service (S) service.

[0169] <<Hardware configuration of each processing unit>> The hardware structure of the processing units that perform various processes, such as the alignment processing unit 110, feature extraction units 111, 332, and deformation vector field calculation units 112, 334 in the image processing device 220, and the crop processing unit 402, data extension conversion units 404, 405, and random deformation processing unit 406 in the machine learning device 400, is, for example, various processors as shown below.

[0170] Various types of processors include CPUs, which are general-purpose processors that execute programs and function as various processing units, GPUs, which are processors specialized for image processing, programmable logic devices (PLDs), such as FPGAs (Field Programmable Gate Arrays), which are processors whose circuit configuration can be changed after manufacturing, and dedicated electrical circuits, such as ASICs (Application Specific Integrated Circuits), which are processors with circuit configurations designed specifically to execute specific processes.

[0171] A single processing unit may be configured with one of these various processors, or may be configured with two or more processors of the same or different types. For example, a single processing unit may be configured with multiple FPGAs, or a combination of a CPU and an FPGA, or a combination of a CPU and a GPU. Alternatively, multiple processing units may be configured with a single processor. A first example of multiple processing units configured with a single processor is a configuration in which one or more CPUs and software are combined to form a single processor, as typified by client or server computers, and this processor functions as multiple processing units. A second example is a configuration in which a processor is used to realize the functions of an entire system including multiple processing units on a single IC (Integrated Circuit) chip, as typified by a system-on-chip (SoC). In this way, the various processing units are configured with one or more of the above-mentioned various processors as a hardware structure.

[0172] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit made up of a combination of circuit elements such as semiconductor elements.

[0173] Advantages of the embodiments of the present disclosure According to the embodiments described in the first to third embodiments and the application examples of application examples 1 to 3, a feature map is generated for each image using the first neural network NN1, and a combination of feature maps of different images is input to the second neural network NN2 to calculate a deformation vector field between the images, thereby reducing the computational resources (computational amount and / or storage capacity) required when aligning images. In particular, when three or more images are used, one of which is used as a reference image and the other images are aligned, the feature map of the reference image can be commonly used for the combination with the other images, which significantly reduces the computational amount.

[0174] Other application examples In the above-described embodiment, medical images have been described as an example, but the scope of application of the present disclosure is not limited to medical images and can be applied to various types of images regardless of their intended use. Furthermore, in the above-described embodiment, an example of handling three-dimensional images has been described, but the technology of the present disclosure can also be applied to two-dimensional images. When the images to be handled are two-dimensional images, a network structure for processing two-dimensional images can be adopted for the first neural network NN1 and the second neural network NN2.

[0175] "others" The present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the technical idea of ​​the present disclosure. [Explanation of symbols]

[0176] 10,101,102,103 Alignment Model 110 Alignment processing unit 111 Feature Extraction Unit 112 Deformation vector field calculation unit 130,132,133 Alignment Model 200 Medical Information Systems 202 Electronic Medical Record System 204 CT device 206 MRI machine 210 Image storage server 212 Image Database 220 Image Processing Device 222 Input Device 224 Display device 230 viewer terminals 232 Input Device 234 Display device 240 communication lines 302 processor 304 Computer-Readable Medium 306 Communication Interface 308 Input / Output Interface 310 Bus 322 memory 324 Storage 330 Alignment Processing Program 332 Feature Extraction Unit 334 Deformation Vector Field Calculation Unit 340 Corresponding point calculation program 350 Property Analysis Program 360 Display Control Program 400 Machine Learning Device 402 Crop processing section 404,405 Data Expansion and Conversion Unit 406 Random transformation processing unit 410,412,413 Learning Model 420 Optimizer NN1 The first neural network NN2 Second Neural Network BB1, BB2 rectangular frame ROI(A),ROI(B),ROI(C) ROI image FM(A),FM(B),FM(C) feature maps CP(B),CP(C) corresponding points TI training images TI(x) B Top image TIa(x) extended training images TId(x) Extended deformation training image TI1, TI2 images IM1, IM1a, IM1c images IM2, IM2a, IM2c images FM1,FM2 feature maps DVf(B),DVf(C) Deformation vector field S101 to S104: Steps for aligning the region of interest S111 to S113: Steps for calculating the amount of deviation between ROI images S201~S204 Image saving process steps S211~S215 Processing steps during image interpretation

Claims

1. 1. An image processing method executed by one or more processors, comprising: the one or more processors: obtaining a feature map for each of a plurality of images; calculating a deformation vector field from the combination of the feature maps for each image; Including, the plurality of images are images with different contrast conditions; the one or more processors: analyzing the aligned images using the deformation vector field and outputting a characteristic finding indicative of a contrast enhancement effect in the region of interest. Image processing methods.

2. the one or more processors: generating the feature map for each image from each of the plurality of images using a first neural network; calculating the deformation vector field using the second neural network by inputting the combination of the feature maps generated for each of the images using the first neural network into the second neural network; The image processing method according to claim 1 .

3. the first neural network is a network that receives an input of one image and outputs one or more feature maps by processing the input image, the second neural network is a network that receives an input of a pair of feature maps of two different images generated from each of the two different images, and outputs the deformation vector field between the two different images by processing the input pair of feature maps. The image processing method according to claim 2 .

4. The first neural network and the second neural network are trained models that have been trained in advance using a training image set; The machine learning step is performed by inputting two images into the first neural network, respectively, and inputting a combination of feature maps of the two images into the second neural network to output the deformation vector field.

4. The image processing method according to claim 2 or 3.

5. the training image set includes a plurality of different images; One of the two images input to the first neural network during the machine learning is an image generated by transforming the other image. The image processing method according to claim 4.

6. a deformation field that defines the deformation is randomly generated within a predetermined constraint range, and the deformation field applied to the deformation process is taken as a correct answer, and learning is performed so that the output of the second neural network approaches the correct answer. The image processing method according to claim 5 .

7. the plurality of images is three or more images, the one or more processors: a reference image of the plurality of images; calculating the deformation vector field for each combination of the reference image and an image other than the reference image from the combination of the feature maps of two images and an image other than the reference image; The image processing method according to any one of claims 1 to 6.

8. the one or more processors further comprising: Accepting designation of a point of interest in one of the plurality of images; calculating a corresponding point corresponding to the point of interest in another image among the plurality of images based on the calculated deformation vector field; displaying the image with the positions of the attention point and the corresponding point aligned; The image processing method according to claim 1 , further comprising:

9. one or more processors; one or more memories in which programs to be executed by the one or more processors are stored; Equipped with The one or more processors execute the instructions of the program to: Obtain feature maps for each of the multiple images, An image processing device that calculates a deformation vector field from a combination of the feature maps for each image, the plurality of images are images with different contrast conditions; The one or more processors: analyzing the aligned images using the deformation vector field and outputting characteristic findings indicative of the enhancement effect of the region of interest; Image processing device.

10. The one or more processors: generating the feature map for each image from each of the plurality of images using a first neural network; calculating the deformation vector field using the second neural network by inputting the combination of the feature maps generated for each of the images using the first neural network into the second neural network; The image processing device according to claim 9 .

11. On the computer, The ability to obtain feature maps for each of multiple images, a function of calculating a deformation vector field from a combination of the feature maps for each of the images; To achieve this, the plurality of images are images with different contrast conditions; The computer, and analyzing the plurality of images aligned using the deformation vector field, and outputting characteristic findings representing the enhancement effect of the region of interest. program.

12. generating the feature map for each image from each of the plurality of images using a first neural network; a function of calculating the deformation vector field using the second neural network by inputting the combination of the feature maps generated for each of the images using the first neural network into the second neural network; The program according to claim 11, which causes the computer to execute the program.

Images