Medical information processing device, medical information processing method, and medical information processing program

Abstract

To realize efficient processing and reduced burden on a user, e.g., in diagnostic imaging employing supervised machine learning.SOLUTION: A medical information processing apparatus comprises an acquisition unit and a processing unit. The acquisition unit acquires medical image data in respect of at least one subject, and non-image data. The processing unit generates a filter to limit a region of the medical image data based on the non-image data, and uses the filter to process the medical image data.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The embodiments disclosed in this specification relate to a medical information processing device, a medical information processing method, and a medical information processing program. [Background technology]

[0002] It is known to train machine learning algorithms to process data, such as medical data.

[0003] Training of machine learning models can be done either supervised, unsupervised, or a combination of supervised and unsupervised.

[0004] Supervised machine learning techniques require large amounts of annotated training data to achieve good performance. However, annotated data can be difficult and expensive to obtain, especially in the medical domain where only time-poor domain experts can provide reliable labels. It is desirable to improve the output quality of trained machine learning models and / or reduce the amount of annotated training data required to train such models and / or reduce the annotation burden on experts. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] US Patent Application Publication No. 2016 / 0217368 Summary of the Invention [Problem to be solved by the invention]

[0006] One of the problems that the embodiments disclosed in this specification and the like aim to solve is to improve the efficiency of processing and reduce the burden on users in image diagnosis using supervised machine learning, for example. However, the problems that the embodiments disclosed in this specification and the drawings aim to solve are not limited to the above problems. Problems corresponding to the effects of each configuration shown in the embodiments described later can also be positioned as other problems. [Means for solving the problem]

[0007] The medical image processing apparatus according to the present embodiment includes an acquisition unit and a processing unit. The acquisition unit acquires medical image data and non-image data for at least one subject. The processing unit generates a filter that limits an area of ​​the medical image data based on the non-image data, and processes the medical image data using the filter. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic diagram of an apparatus according to an embodiment. [Diagram 2] FIG. 2 is a schematic diagram of the processing performed by the embodiment of FIG. [Diagram 3] FIG. 3 shows images used when performing a test process using the process according to FIG. [Figure 4] FIG. 4 is a schematic diagram of a network according to an embodiment. [Diagram 5] 5a and 5b are images with and without an attention-attention map overlay. [Figure 6] FIG. 6 is a schematic diagram of the processing performed by the embodiment of FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, a medical information processing apparatus, a medical information processing method, and a medical information processing program according to an embodiment will be described with reference to the accompanying drawings.

[0010] 1 shows a schematic diagram of a data processing device 20 according to an embodiment. In this embodiment, the data processing device 20 is configured to process medical imaging data. In other embodiments, the data processing device 20 may be configured to process any suitable data, such as, for example, imaging data, text data, structured data, or a combination of heterogeneous data.

[0011] The data processing device 20 comprises a computing device 22, which in this example is a personal computer (PC) or a workstation. The computing device 22 is an example of a medical information processing device. The computing device 22 is connected to a display screen 26 or other display device and one or more input devices 28, such as a computer keyboard and a mouse. The display screen 26 is an example of a display.

[0012] The computing device 22 is configured to retrieve an image data set from a data store 30. The image data set is generated by processing data acquired by the scanner 24 and stored in the data store 30.

[0013] The scanner 24 is configured to generate medical imaging data, which may comprise two-dimensional, three-dimensional, or four-dimensional data of any imaging modality. For example, the scanner 24 may comprise a magnetic resonance (MR or MRI) scanner, a CT (computed tomography) scanner, a cone-beam CT scanner, an X-ray scanner, an ultrasound scanner, a PET (positron emission tomography) scanner, or a SPECT (single photon emission computed tomography) scanner. The medical imaging data may comprise or be associated with additional conditioning data, which may include, for example, non-imaging data.

[0014] Computing device 22 may receive medical image data and / or further conditioning data from one or more further data stores (not shown) instead of or in addition to data store 30. For example, computing device 22 may receive medical image data from one or more remote data stores (not shown) that may form part of a Picture Archiving and Communication System (PACS) or other information system.

[0015] The computing device 22 provides processing resources for automatically or semi-automatically processing medical image data. The computing device 22 includes a processing device 32. The processing device 32 is an example of a processing unit. The processing device 32 generates a filter for limiting a region of the medical image data based on non-image data, and processes the medical image data using the filter. More specifically, the processing device 32 includes a model training circuit 34, a data processing / labeling circuit 36, and an interface circuit 38. The model training circuit 34 is configured to train one or more models. The data processing / labeling circuit 36 ​​applies the trained model to obtain an output, such as to obtain a label, a segmentation, or other processing result, for example, for output to a user or for providing to the model training circuit 34 for further model training processing, and / or to obtain a label. The interface circuit 38 receives a user input or other input, data stored in the data storage unit 30. The interface circuit 38 is also configured to output a result of the data processing.

[0016] In this embodiment, the model training circuitry 34, data processing / labeling circuitry 36, and interface circuitry 38 are each implemented in computing device 22 by a computer program having computer readable instructions executable to perform the method of the embodiment, however, in other embodiments, the various circuits may be implemented as one or more ASICs (application specific integrated circuits) or FPGAs (field programmable gate arrays).

[0017] Computing device 22 also has a hard drive and other components of a PC, including RAM, ROM, a data bus, an operating system including various device drivers, and hardware devices including a graphics card, although such components are not shown in FIG.

[0018] The data processing apparatus 20 of FIG. 1 is configured to carry out the methods illustrated and / or described below.

[0019] In a feature of the embodiment, conditioning data in the form of non-image data can be used to generate spatially-varying functions, sometimes referred to as filters, that can provide an attention mechanism that allows the incorporation of non-imaging information (also referred to as "non-image information") or non-imaging data (also referred to as "non-image data") into the imaging pipeline for spatial localization. Such information can be converted into regions of interest (ROIs) that can be applied to focus attention on specific regions of image data based on the non-image information, for example, during training of neural networks or other models.

[0020] The use of the generated spatially dependent filter based attention mechanism in model training can be beneficial for models used for annotation of medical images, for example, but not limited to, and can improve the quality of predictions without requiring additional training set annotation by an expert. In such applications, many images are accompanied by non-imaging information, for example, electronic health records (EHR) and (DICOM) metadata. Such non-image data is used in some embodiments to generate a spatially-varying function, for example, a filter, that is used as a spatially dependent attention mechanism in the training process. That is, the non-image data includes at least one of the following: age, weight, sex, presence or absence of a particular condition, at least one property of an imaging process or imaging device, output of another method, electronic health record, text data, structured data, metadata, DICOM data, DICOM metadata of at least one subject. The non-image data may also be information generated using image data acquired by an optical imaging device, for example, a digital camera. The non-image data may also be generated using image data acquired by other modalities, for example.

[0021] The process of applying non-image information in the embodiment of FIG. 1 in providing a spatially dependent attention mechanism in training a neural network or other model will now be described in detail with reference to FIG.

[0022] In a first stage 40 of the process, the interface circuit 38 receives (acquires) an annotated training set of medical image data of a plurality of subjects, representing the anatomical region to be considered. Although one image is shown in stage 40, this is for illustrative purposes only, and it is considered that an annotated training set of medical image data is used in training the model. In this embodiment, the medical image data acquired in stage 40 is, for example, a post-reconstruction image such as a CT image, an MR image, etc. In contrast, the medical image data acquired in stage 40 can also be, for example, raw data (sinograms, etc.) before reconstruction, k-space data before reconstruction, etc.

[0023] Each annotated training set of medical image data comprises medical image data representative of an anatomical region to be considered, and in each data set, the annotation identifies a particular anatomical feature or pathology of interest that is the subject of a model training process intended to train the model so that it can be applied, for example by the data processing / labeling circuitry 36, to new data sets to obtain annotations of the anatomical features or pathologies of interest in each of the new data sets. Such new data sets may, for example, be data sets acquired by the scanner 24.

[0024] The annotations included in the annotated training set may comprise, for example, labels used to represent segmentation of image data (e.g., which pixels or voxels, or regions of pixels or voxels, correspond to anatomical features or pathologies of interest).

[0025] After receiving the annotated training set of medical image data, the training set is passed to the model training circuit 34 by the interface circuit 38. In the embodiment of Figure 2, the model training circuit 34 provides a convolutional neural network (CNN) and trains the CNN with the annotated training set of medical image data. Note that the CNN in the embodiment of Figure 2 is an example of a first model that inputs medical image data and outputs output information.

[0026] The CNN of the embodiment of Figure 2 typically has a structure or architecture based on any suitable and known CNN structure or architecture, and includes multiple layers including convolutional layers, fully connected layers, and pooling layers, along with input and output layers. Any other suitable layers may be included in accordance with known CNN methods.

[0027] According to known techniques, for a particular layer, a feature map is created as an output from the processing performed by the previous layer, and the feature map is provided as an input to the layer. Then, the output of the layer, e.g., in the form of a feature map, is provided as an input to the next layer of the CNN. Each layer can have any suitable number of input and output channels. In each layer, any suitable and desired processing may be performed, such as, for example, filtering / convolution, pooling, ReLU processing, or any other desired processing. In an embodiment, any suitable number of layers may be provided. Also, any suitable number and arrangement of layers may be used, e.g., constrained only by the requirements of a particular CNN method and architecture.

[0028] Note that for clarity, Figure 2 does not depict the layers of the example CNN or the ordering and connections between layers. Instead, Figure 2 illustrates generally the conditioning processes 42, 44, and 46 that are performed in the layers of the CNN in the embodiment of Figure 2.

[0029] Although processes 42, 44, and 46 are shown adjacent to one another in Figure 2, this is not intended to imply that these processes must occur in successive layers of a CNN, although in some embodiments they may do so if desired. In an embodiment according to Figure 2, processes 42, 44, and 46 may occur in any desired layer of a CNN, and the layers may be separated by any desired number of other layers of the CNN.

[0030] In the embodiment of Fig. 2, the processing unit 32 trains a second model, a neural network different from the first model, to obtain at least one parameter value of the filter to generate the filter. That is, the auxiliary network 48 as the second model or other model or algorithm is provided in addition to the main CNN as the first model. The auxiliary network 48 can be another CNN or any other suitable type of neural network, deep learning algorithm, or alternative trainable or other model.

[0031] As shown in FIG. 2, in the illustrated embodiment, the auxiliary network receives non-image data that can be used to limit or influence the space of ambiguous outputs of the main CNN (e.g., ambiguous outputs for the task of identifying particular anatomical features of interest in each set of image data).

[0032] In the exemplary embodiment shown in FIG. 2, the non-image data input to the auxiliary network is indicated as the subject's biological sex (eg, male or female), the subject's height, and the subject's weight.

[0033] It will be appreciated that such non-image data may relate to size, location, or other characteristics of anatomical features of interest. For example, properties such as the heart or blood vessels of a healthy young subject are expected to differ from those of an unhealthy, obese elderly subject. Size, shape, position, or relative location, among other properties, may differ depending on such non-image data or other conditioning data.

[0034] As shown diagrammatically in FIG. 2, the output of the auxiliary network 48 may provide scale (γ) and shift (β) parameters that are provided as inputs to processes 44, 46 at layers of the model. The scale and shift parameters transform the image Fc (e.g., one or more training image datasets, or feature maps derived from a previous layer of the CNN) according to a batch normalization process, e.g., as represented in Equation (1) below. Any suitable batch normalization process may be applied to the CNN, e.g., batch normalization processes as described in Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, Ioffe et al, 2015, arXiv.1502.03167, US2016217368.

[0035]

number

[0036] As part of processes 44, 46, separate scale (γ) and shift (β) factors can be applied to each channel of one or more relevant layers of the CNN, resulting in feature maps that are individually amplified or suppressed to affect the final predictions of the CNN. However, such batch normalization processes applied, for example, at 44, 46, typically do not provide the flexibility to adjust channels depending on spatial location; instead, the scale and shift factors modify the entire feature map.

[0037] As an important feature of the embodiment according to Fig. 2, the spatially dependent conditioning that depends on non-image data is also applied to one or more layers of the CNN. That is, the CNN as the first model comprises at least a first layer and a second layer that receives the output from the first layer as input. The processing unit 32 uses a filter on the output from the first layer. In particular, in the embodiment according to Fig. 2, an attention mechanism based on a differentiable function (e.g., Gaussian) is provided to generate a spatially dependent filter that can be applied, for example, before applying a feature-specific transformation.

[0038] The filter has the effect of limiting the region of the medical image data so that more attention (e.g., weight or importance) is given to the spatial region of the image data when training the model. For example, if an anatomical feature of interest is more likely to be found in a particular spatial region of an image (e.g., an aligned and / or normalized image) for a subject with a particular value of a non-image parameter, a model (an example of a first model) may be applied to label (e.g., segment) the anatomical feature of interest in the image dataset. When training a network model including a CNN or other neural network as such a first model, the filter acting as an attentional function may ensure that more attention is given to that region of the image.

[0039] In the embodiment according to FIG. 2, the filter is the product of two Gaussian vectors (a=a 1 a T 2 ) and fed into a layer that performs the condition instance normalization process 44. The mechanism is computationally efficient and requires only four parameters (μ 1 ,σ 1 ,μ 2 ,σ 2 ) to the existing layers of the CNN. In other embodiments, any suitable space-varying function may be used, and embodiments are not limited to functions that comprise Gaussians or combinations of Gaussians.

[0040] For example, parameter values ​​determining the shape and location of the Gaussians, such as the peak location and variance of each Gaussian, are determined in the embodiment of FIG. 2 as outputs of auxiliary network 48 based on non-image data (e.g., in the example data shown in FIG. 2, the subject's gender, height, and weight) or as inputs to auxiliary network 48 based on other conditioning data provided with the associated training image data set.

[0041] The filters generated by the auxiliary network 48 to function as attention functions can be shared across feature maps or applied per channel when training the CNN. For example, Gaussian or other filters with different parameter values ​​(e.g., peak location, variance) may be used for different feature maps and / or channels in one or more layers when training the CNN. The parameter values ​​of the filters applied as attention functions to the different feature maps and / or channels may be learned separately by the auxiliary network 48. Alternatively, in some embodiments, the same filter with the same parameter values ​​may be used as an attention function for all relevant channels and / or more than one feature map.

[0042] Although FIG. 2 shows a Gaussian filter acting as an attention function applied to a single layer of the CNN, in other embodiments the filter acting as an attention function may be applied to more than one different layer of the CNN, or different filters obtained using non-image data, each acting as an attention function, may be applied to different layers of the CNN.

[0043] Although subject height, weight, and sex are described above with respect to a specific implementation of the embodiment of FIG. 2, any suitable data may be used as conditioning data to provide spatially dependent attention. The conditioning data may be of any modality, such as, for example, an image, a one-hot coded vector, structured text, etc. The conditioning data may comprise or represent, for example, one or more of age, weight, sex, the presence or absence of a particular condition, at least one property of an imaging process or imaging device, the output of another methodology, an electronic health record (HER), metadata, DICOM data, audio data, or structured data.

[0044] As a feature of one embodiment, when the conditioning data, e.g., non-image data, does not contain spatial information or is not related to the spatial distribution of the image or of associated anatomical features or pathologies within the image, the method generalizes to the condition instance normalization layer, i.e., attention does not affect the feature maps (in practice, e.g., a Gaussian with large diffusion is generated, e.g., by the auxiliary network 48, as a filter that applies equal weights to all feature maps).

[0045] Where the conditioning data is related to the spatial distribution of the image or of associated anatomical features or pathologies within the image, in some embodiments generation of a filter (e.g., one or more Galcians) comprises application of a process that promotes non-spatial homogenization of the filter.

[0046] For example, network defaulting to a general solution using a uniform attention map is not desired, and to avoid this, a regularization penalty may be included in the cost function used in the filter generation process to facilitate learning of localizations. For example, in the case of Gaussian functions, Gaussians with large spread (e.g., with large standard deviation / variance) may be penalized in some embodiments with a training loss function such as that expressed in Equation (2) below.

[0047]

number

[0048] In the above equation (2), the first term is the main objective of optimization, e.g., Dice loss. The second term is a penalty to be imposed, e.g., a standard deviation parameter of the attention-driven Galcian filter (e.g., large standard deviation values ​​are penalized). The hyperparameter η is used to control the tradeoff between the main objective and the regularization penalty. The above method for promoting non-spatially uniformity of the spatially dependent filter may be implemented by the auxiliary network 48 in the embodiment of FIG. 2. In some embodiments, the loss function may be implemented according to the techniques described in, e.g., Numerical Co-ordinate Regression with Convolutional Neural Networks, Nibali et al, 2018, arXiv:1801.07372.

[0049] The embodiment of Figure 2 was applied to an extension of the CLEVR dataset (CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning Johnson et al, arXiv:1612.06890, 2016) modified to include segmentation masks and several conditioning scenarios. Figure 3 shows a representation of the dataset to which the method is applied, including in the bottom image various features to be labeled (square, circle, and prism-shaped objects with different colors and in different locations).

[0050] In the application shown in Figure 3, various tests were performed on the output of the CNN with the following conditions: i) location (e.g., select only objects in the bottom right corner), ii) color (e.g., select only red objects), iii) shape (e.g., select only prisms), and iv) size (e.g., select only medium-sized objects). In scenario i), the condition factors have spatial information, and in scenarios ii)-iv), the condition factors do not have spatial information.

[0051] The other images above the bottom image in FIG. 3 show segmentation masks corresponding to conditions i) to iv).

[0052] The task described in relation to Figure 3 was performed using two variants of the embodiment of Figure 2. The first uses a filter (e.g., a Garcian-style attention map) per channel, which we call the exclusive attention variant, and the second shares a filter (e.g., a Garcian-style attention map) between attention maps, which we call the shared attention variant.

[0053] The tasks were also based on known techniques, including baseline (a known CNN with no additional conditioning), FiLM (using techniques derived from FiLM: Visual Reasoning with a General Conditioning Layer, Perez et al, arXiv:1709.07871, 2017), and Guide Me (using techniques derived from Guide Me: Interacting with Deep Networks, Rupprecht et al, arXiv:1803.11544, 2018).

[0054] The results of the tasks performed under different conditioning scenarios i)-iv) using two variants of the embodiment of Fig. 2 and various known techniques are given in the table below in the form of achieved Dice values. In each case, the best performance was achieved by one of the embodiments of Fig. 2 (labeled Proposed Method in the table). In all techniques, the same base segmentation network, e.g. CNN, was used.

[0055] [Table 1]

[0056] If the conditioning factors used do not contain spatial information, e.g., are not related to absolute positions in image space, the method according to some embodiments generalizes to a condition instance normalization layer.

[0057] The task was also run on the ACDC dataset (Bernard, O., Lalande, A., Zotti, C., Cervenansky, F., Yang, X., Heng, PA, Cetin, I., Lekadir, K., Camara, O., Ballester, MAG and Sanroma, G., 2018. Deep Learning Techniques for Automatic MRI Cardiac Multi-structures Segmentation and Diagnosis: Is the Problem Solved?. IEEE Transactions on Medical Imaging, 37(11), pp. 2514-2525). The task was repeated with a limited number of training examples (e.g., 100%, 25%, 6% datasets) to mimic a scenario where the machine does not have access to a huge number of annotated examples.

[0058] In our application, we performed various tests on the output of the CNN under the following conditions: i) the position of the input image within the heart volume (e.g., bottom slice denoted by 1); ii) the cardiac phase during acquisition of the input images (e.g., end systole).

[0059] [Table 2]

[0060] According to an embodiment, the non-image data based filters and, for example, providing a location-dependent attention map, may be provided or implemented as layers of a CNN or other model. The locations of the filters, for example, implemented as layers, may vary in embodiments and may be included in various locations of the network as desired. The layer with the non-image data based filters and, for example, providing a location-dependent attention map, may appear more than once in the network. FIG. 4 illustrates an embodiment in which the filter layer 62 is provided twice to the CNN 60. Other layers or other elements of the CNN 60 are block 64 (labeled Part A) and block 66 (labeled Part B).

[0061] Once the model is trained, the CNN may be applied to any suitable dataset to perform the task it was trained for, such as labeling anatomical features or pathology in an image dataset, for example.

[0062] In some embodiments, a visual representation of the space-varying function is displayed together with, e.g., superimposed on, an image from the image dataset. The image may comprise a representation of a segmentation or other labeling generated by a trained model, e.g., a CNN. By displaying a representation of the filter, e.g., a Gaussian, the user can see where, e.g., to which part of the image space, attention was directed in training the model. This can be useful to help the user understand the training process or to gain the user's confidence that the model training process is reliable. The superimposition of a representation of the filter used to provide location-dependent attention is shown in Fig. 5a-b. Fig. 5a shows the image without the superimposition of the filter representation, and Fig. 5b shows the image with the superimposition of the filter representation.

[0063] In the embodiment of FIG. 2, the filter process 42 is used to direct attention based on absolute position in image space. In an alternative embodiment, a filter, such as one or more Gaussians, may be used to direct attention to regions of interest based on relative position, such as relative position between anatomical features and / or pathologies. In this manner, the positions of objects may be coded or represented relative to one another (e.g., "the right ventricular cavity is next to the myocardium" as well as "position (150,75) in pixel space" or instead). Such an alternative embodiment is shown diagrammatically in FIG. 6, where like reference numbers represent like features. The embodiment of FIG. 6 is similar to that of FIG. 2, but includes an additional auxiliary network 70 that is used to generate the filters (e.g., to determine one or more Gaussian parameters) used to code or direct attention based on relative position.

[0064] In one embodiment, a method of processing data is provided, the method comprising: determining parameter values ​​for a spatially-variant function using additional conditioning data related to a spatial distribution of values ​​of the data, applying the spatially-variant function to the data, and making processing of the data dependent on the spatially-variant function.

[0065] Parameter values ​​for the spatially-varying function may be determined using both the data and the additional conditioning data, which may comprise a plurality of data sets, each of which may be associated with conditioning data, at least some, and optionally all, of the data sets.

[0066] The spatially-varying function may comprise multiple spatially-varying functions.

[0067] The data may comprise a set of imaging data and the additional conditioning data may comprise non-imaging data.

[0068] The data may comprise medical imaging data, text data, or any suitable type of data. The data may comprise ECG or other sensor data, acceleration data, wearable device data, and / or any suitable series of signals or suitable time-dependent signals.

[0069] The data may comprise at least one of magnetic resonance (MR) data, computed tomography (CT) data, x-ray data, ultrasound data, positron emission tomography (PET) data, single photon emission computed tomography (SPECT) data, or patient record data.

[0070] The set of data may comprise a labelled set of data, where the label may comprise or represent a classification and / or segmentation and / or identification of an anatomical feature or pathology.

[0071] The data may comprise imaging data representative of an image of a patient, and the additional conditioning data may, for at least some patients, represent at least one characteristic of the patient or at least one property of an imaging process or imaging device used to obtain the image.

[0072] The conditioning data may comprise or represent at least one of age, weight, sex, and the presence or absence of a particular condition.

[0073] The conditioning data may comprise or represent one or more outputs of other data, an electronic health record (EHR), or another method of processing any data that correlates to visual properties found in the image (e.g., pathology, acquisition settings, etc.).

[0074] The spatially-varying function may comprise an attention function. Application of the spatially-varying function may be such that data is given greater or less importance in processing determined depending on the spatially-varying function.

[0075] The spatially-varying function may comprise a filter.

[0076] The method may comprise a method of training a model. The model may comprise a plurality of layers. The data may comprise a set of training data. Applying a space-variant function may comprise applying a space-variant function to an output of a layer of the model as part of training the model.

[0077] The model may comprise a machine learning model.

[0078] The model may comprise a neural network.Determining parameter values ​​for the spatially varying function may comprise training an auxiliary model to obtain the parameter values, and the auxiliary model may comprise an auxiliary neural network.

[0079] Determining parameter values ​​for the spatially-varying function may comprise application of a process that promotes non-spatially uniformity of the spatially-varying function.

[0080] A process for promoting non-spatially uniformity of a spatially varying function may comprise the use of a penalty in the loss function, such as, for example, penalizing larger values ​​of the standard deviation of the function compared to smaller values ​​of the standard deviation.

[0081] Applying the space-variant function to the output of a layer of the model may comprise applying the space-variant function as part of a further layer of the model, the output of which may be used as an input to the further layer.

[0082] The further layer may comprise a plurality of feature maps and / or channels, and applying the spatially-varying function may comprise applying different function parameter values ​​and / or different functions to different ones of the feature maps and / or channels.

[0083] The method may comprise determining respective parameter values ​​of a space-varying function for each of a plurality of channels and / or feature maps.

[0084] The method may further comprise applying a process to different ones of the multiple channels and / or feature maps that promotes a change in the location of a peak or other feature of the spatially-varying function.

[0085] Determining the individual parameter values ​​of the spatially-varying function may comprise using an auxiliary neural network to learn parameter values ​​separately for each of the multiple channels and / or feature maps.

[0086] Applying the spatially-varying functions to the outputs of the layers of the model may comprise applying different spatially-varying functions to different feature maps and / or channels.

[0087] The method may comprise determining parameter values ​​for a plurality of space-variant functions and applying the space-variant functions to outputs of a plurality of different layers of the model as part of training the model.

[0088] The application of the spatially variant function to the output of a number of different layers of the model may comprise the application of said spatially variant functions to a number of further layers of the model.

[0089] The method may comprise representing the relative position of two or more features of interest using a space-varying function.

[0090] Determining parameter values ​​for the spatially-variant function may further comprise using additional conditioning data together with the data and / or at least one of volumetric data, image data, text data, metadata, and audio data to determine the parameter values.

[0091] The method may further comprise displaying a visual representation of the space-varying function.

[0092] The spatially-variant function may comprise a differentiable function and / or a compact parameterized function. The spatially-variant function may comprise a Gaussian function or a Student's T function or any efficiently parameterized smooth function or mixture of functions. The spatially-variant function may comprise a product of at least two vectors, optionally at least two Gaussian vectors and / or matrices. The spatially-variant function may comprise a multivariate function, optionally a multivariate Gaussian. The additional conditioning data may comprise at least one of image data, text data, metadata, audio data and / or structured data.

[0093] The method may comprise applying the model to a dataset to generate an output.

[0094] The data may comprise an imaging dataset and the output may comprise an image, and the method may further comprise including a visual representation of the spatially varying function.

[0095] The output may comprise at least one of a label and / or a classification and / or a segmentation and / or an identification of an anatomical feature or pathology. The output may be used to correspond to and / or comprise an output representing a disease classification, an anatomy or pathology description, or an anatomy or pathology detection.

[0096] In an embodiment there is provided a data processing apparatus comprising a processing circuitry configured to use a plurality of input data sets and additional conditioning data relating to the spatial distribution of values ​​of the input data sets to determine parameter values ​​of a spatially-variant function, to apply the spatially-variant function to at least one of the input data sets, and optionally to all of the input data sets, and to make processing of the at least one of the input data sets dependent on the spatially-variant function.

[0097] In one embodiment, a neural network model is provided that includes a plurality of layers and a spatially dependent conditioning layer between one layer and another layer of the plurality of layers, the spatially dependent conditioning layer applying a spatially-varying function determined based on a combination of a set of imaging data and associated non-imaging data.

[0098] The spatially dependent conditioning layer may be implemented with or comprise an auxiliary network.

[0099] In one embodiment, a medical image processing device is provided, comprising a processing circuit configured to receive medical image data of a subject and non-medical image data that are parameters related to the subject, and generate a filter to be applied to the medical image data based on the non-medical image data, the filter being a filter for limiting a region of the medical image data.

[0100] The non-medical image may comprise at least one of: gender, height, weight, BMI, disease.

[0101] The filter may comprise a Gaussian filter.

[0102] The processing circuitry may be further configured to determine parameters of the Gaussian filter based on the non-medical image data.

[0103] The processing circuitry may be further configured to determine a peak location and variance of the Gaussian filter based on the non-medical image data.

[0104] The processing circuitry may be further configured to include a neural network that outputs parameters of the Gaussian filter based on the non-medical image data input.

[0105] An embodiment provides a method for incorporating spatial information, comprising: a) a neural network; b) a set of training examples and additional conditioning data spatially related to the training points; c) the additional information is incorporated in the form of attentional training from a differentiable function (e.g., Gaussian, Student's T-distribution, etc.); and / or d) the attentional training parameters are predicted using an auxiliary network; and / or e) the attentional training mechanism may be trained separately for each channel; and / or f) a penalty in the loss function may be used to encourage the training of an informative (non-uniform) function, for example by penalizing large standard deviations in the case of a Gaussian function, encouraging the Gaussian means of different channels to be spread out; and g) the additional information may be one or more of images, text, metadata, audio, or other structured data.

[0106] The Gaussian parameters may be replaced by any suitable compact parameterized function.

[0107] The auxiliary network, responsible for learning the parameterized attention function, may also be fed with sources of volume, images, text, metadata and audio to better localize features within the original data.

[0108] A user interface may be provided to allow the user to apply non-imaging data to the network and see how the resulting final attention map changes.

[0109] Although particular circuits are described herein, in alternative embodiments, the functionality of one or more of those circuits may be provided by a single processing resource or other component, or the functionality provided by a single circuit may be provided by a combination of two or more processing resources or other components. Reference to a single circuit encompasses multiple components that provide the functionality of that circuit, whether or not such components are separate from one another. Reference to multiple circuits encompasses single components that provide the functionality of those circuits.

[0110] While certain embodiments have been described, these embodiments are presented for illustrative purposes only and are not intended to limit the scope of the invention. In fact, the novel methods and systems described herein may be embodied in a variety of other forms. Furthermore, various omissions, substitutions, and changes in the form of the methods and systems described herein may be made without departing from the spirit of the invention. The appended claims and their equivalents are intended to cover such forms and modifications as fall within the scope of the invention. [Explanation of symbols]

[0111] 20 Data Processing Device 22 Computing devices (medical information processing devices) 24 Scanner 26 Display Screen 28 Input Devices 30 Data storage unit 32 Processing equipment 34 Model Training Circuit 36 Data Processing / Labeling Circuit 38 Interface Circuit 60 CNN 62 Filter Layer Blocks 64 and 66

Claims

1. an acquisition unit for acquiring medical image data having a set of training data and non-image data for at least one subject; A processing unit that generates a filter that limits a region of the medical image data based on the non-image data, and processes the medical image data using the filter; the processing unit inputs the medical image data and outputs output information to train a first model having a plurality of layers, and applies the filter to an output of a layer of the plurality of layers of the first model as part of training the first model. Medical information processing equipment.

2. the first model comprises a neural network; The processing unit trains a second model, the second model being a neural network different from the first model, to obtain at least one parameter value of the filter, and generates the filter. The medical information processing device according to claim 1 .

3. The first model includes at least a first layer and a second layer that receives an output from the first layer as an input; the processing unit applies the filter to the output from the first layer. The medical information processing device according to claim 1 .

4. The processing unit uses different parameter values ​​and different functions for the filter for at least one of a plurality of feature maps generated as an output from a layer included in the first model and a plurality of channels included in the first model. The medical information processing device according to claim 3 .

5. The processing unit determines individual parameter values ​​of the filter for each of a plurality of feature maps generated as an output from a layer included in the first model and / or a plurality of channels included in the first model. The medical information processing device according to claim 3 .

6. The processing unit includes: a) determining parameter values ​​for a number of filters and applying the filters to outputs of a number of different layers of the first model as part of the training of the first model; b) applying a process that promotes the change in peak location or characteristics of the filter to different ones of a plurality of feature maps and a plurality of channels; The medical image processing apparatus according to claim 1 , which executes at least one of the following:

7. determining the parameter values ​​of the plurality of filters includes learning the parameter values ​​for each of the plurality of channels and / or the plurality of feature maps using a neural network; The medical information processing device according to claim 6 .

8. the non-image data includes at least one of the following: age, weight, sex, presence or absence of a particular medical condition of the at least one subject; at least one property of an imaging procedure or imaging device; an electronic health record; text data; structured data; metadata; DICOM data; DICOM metadata; and information generated using image data acquired by an optical imaging device; The medical image processing device according to claim 1 .

9. The processor determines a peak position and a variance of the filter as a Gaussian filter based on the non-image data. The medical image processing device according to claim 1 .

10. The processing unit has a neural network that outputs parameters of a Gaussian filter as the filter based on the non-image data. The medical image processing device according to claim 1 .

11. The processing unit performs a non-spatially uniform process of the filter. The medical image processing device according to claim 1 .

12. The non-spatial homogenization process of the filter includes a regularization process using a penalty in a loss function. The medical information processing device according to claim 11.

13. The filter performs at least one of an extraction process to extract a position of a feature of interest and an extraction process to extract relative positions of a plurality of features of interest. The medical image processing device according to claim 1 .

14. a display unit for displaying a visual representation of the filter; The medical image processing device according to claim 1 .

15. a) the filter comprises at least one of a differentiable function, a smooth function, and a parameterized smooth function; b) the filter comprises a Gaussian function or a Student's T distribution; c) the filter uses at least one of a vector and a matrix multiplication of at least two vectors; d) the non-image data comprises at least one of text data, metadata, audio data, and / or structured data; At least one of the following: The medical image processing device according to claim 1 .

16. The method of claim 1, further comprising: applying the trained first model to a dataset. The medical information processing device according to claim 1 .

17. acquiring medical image data having a training data set and non-image data for at least one subject; generating a filter for limiting a region of the medical image data based on the non-image data; processing the medical image data using the filter; training a first model having a plurality of layers by inputting the medical image data and outputting output information; applying the filter on an output of a layer of the plurality of layers of the first model as part of training the first model; A medical information processing method comprising:

18. On the computer, acquiring medical image data and non-image data for at least one subject; a function of generating a filter for limiting a region of the medical image data based on the non-image data, and processing the medical image data using the filter; a function of inputting the medical image data and outputting output information to train a first model having a plurality of layers; applying the filter on an output of a layer of the plurality of layers of the first model as part of training the first model; A medical information processing program that makes this possible.

Images