Multi-modal base model of medical images using modal-specific and cross-modal expert sub-networks

Abstract

Systems and methods are provided for performing medical imaging analysis tasks using machine learning-based models (e.g., base models). One or more input medical images are received. One or more modality-specific expert sub-networks of a machine learning-based model are selected to perform a first processing of one or more input medical images. A first processing of the one or more input medical images is performed to perform a medical imaging analysis task using the one or more selected modality-specific expert sub-networks. The results of the first processing are merged into one or more merged result sets. One or more cross-modal expert sub-networks of the machine learning-based model are selected to perform a second process of the one or more consolidated result sets. A second process of the one or more consolidated result sets is performed to perform a medical imaging analysis task using the one or more selected cross-modal expert sub-networks. And outputting a result of the second processing.

Description

Technical Field

[0001] The present invention generally relates to large-scale machine learning based ground models, and in particular to multimodal ground models for medical images using modality-specific and cross-modality expert sub-networks. Background Art

[0002] Base models are large-scale machine learning models that can be adapted to perform many different applications. Base models are trained on a wide range of datasets and fine-tuned for a variety of tasks. Conventionally, training of multimodal base models typically requires paired training data from different modalities, such as, for example, text and image, text and speech, speech and image, etc. However, the medical field presents unique and novel challenges for training base models due to the heterogeneity of the data, especially for medical images of different modalities acquired using different types of scanners. It is not feasible to collect large-scale pairs of medical images of different modalities because the costs associated with imaging the same patient with multiple scanners are prohibitively high, especially when there is no clinical reason to do so.

[0003] Recently, auxiliary tasks that do not require labels have been proposed for training base models of medical images of different modalities via self-supervised learning. However, in the absence of medical image pairs, all images must be processed by the same encoder, which leads to entangled representations across modalities and reduces the representation power of the base model. Contrastive learning methods, which are often used for self-supervised learning, are also insufficient because they require negative anchors both within and across modalities. Such contrastive learning methods may lead to representations that are significantly different within the modality, but not too different across modalities. Summary of the invention

[0004] According to one or more embodiments, a system, method, and computer-readable medium storing computer program instructions for performing a medical imaging analysis task using a machine learning-based model (e.g., a base model) are provided. One or more input medical images are received. One or more modality-specific expert subnetworks based on a machine learning model are automatically selected to perform a first processing of the one or more input medical images. The first processing of the one or more input medical images is performed to perform the medical imaging analysis task using the one or more selected modality-specific expert subnetworks. The results of the first processing are merged into one or more merged result sets. One or more cross-modality expert subnetworks based on a machine learning model are selected to perform a second processing of the one or more merged result sets. The second processing of the one or more merged result sets is performed to perform the medical imaging analysis task using the one or more selected cross-modality expert subnetworks. The results of the second processing are output.

[0005] In one embodiment, the machine learning based model is trained via self-supervised learning.

[0006] In one embodiment, the first processing of the one or more input medical images is performed by processing the modality specific data of the one or more input medical images using the one or more selected modality specific expert sub-networks.

[0007] In one embodiment, the second processing of the one or more merged result sets is performed by integrating the one or more merged result sets between each other.

[0008] In one embodiment, during training of the machine learning based network, multiple modality specific expert sub-networks in the machine learning based network are learned for each modality. In another embodiment, multiple modality specific expert sub-networks in the machine learning based network are predefined for each modality.

[0009] In one embodiment, during training of the machine learning based network, multiple cross-modal expert sub-networks in the machine learning based network are learned.

[0010] In one embodiment, the multiple layers and the multiple parameters of each layer of each modality-specific expert sub-network and each cross-modality expert sub-network in the machine learning based network are defined based on resource availability.

[0011] In one embodiment, the machine learning based model is the base model.

[0012] These and other advantages of the present invention will become apparent to those of ordinary skill in the art by reference to the following detailed description and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 A method for performing a medical imaging analysis task using a machine learning based model including a modality specific expert sub-network and a cross-modality expert sub-network according to one or more embodiments is shown;

[0014] Figure 2 A workflow for performing a medical imaging analysis task using a machine learning-based model including a modality-specific expert subnetwork and a cross-modality expert subnetwork according to one or more embodiments is shown;

[0015] Figure 3 A workflow for efficiently performing a medical imaging analysis task on an input medical image of a single modality using a machine learning-based model including a modality-specific expert subnetwork and a cross-modality expert subnetwork according to one or more embodiments is shown;

[0016] Figure 4An exemplary artificial neural network that can be used to implement one or more embodiments is shown;

[0017] Figure 5 shows a convolutional neural network that can be used to implement one or more embodiments; and

[0018] Figure 6 A high-level block diagram of a computer that can be used to implement one or more embodiments is shown. DETAILED DESCRIPTION

[0019] The present invention generally relates to methods and systems for multimodal base models for medical images, which use modality-specific and cross-modal expert subnetworks. Embodiments of the present invention are described herein to give a visual understanding of such methods and systems. Digital images are typically composed of digital representations of one or more objects (or shapes). In this article, the digital representation of an object is typically described in terms of the identification and manipulation of the object. Such manipulations are virtual manipulations performed in the memory or other circuit systems / hardware of a computer system. Therefore, it will be understood that embodiments of the present invention can be performed within a computer system using data stored within the computer system. In addition, references to image pixels herein may equally refer to voxels of the image, and vice versa.

[0020] Embodiments described herein provide a base model that uses a mixture of intra-modal and cross-modal expert training. Such a base model includes a modality-specific expert subnetwork that preserves modality-specific properties and a cross-modal expert subnetwork that integrates cross-modal features, together with an intra-modality expert selection module for selecting an appropriate modality-specific expert subnetwork and a cross-modal expert selection module for selecting an appropriate cross-modal expert subnetwork. Advantageously, the base model according to the embodiments described herein 1) promotes untangled representations during training, 2) ensures scalability of the number of parameters, 3) ensures fast runtime for modality-specific inference, and 4) enables cross-modal information exchange.

[0021] Figure 1 A method 100 for performing a medical imaging analysis task using a machine learning-based model including a modality-specific expert subnetwork and a cross-modality expert subnetwork according to one or more embodiments is shown. The steps of method 100 may be performed by one or more suitable computing devices, such as, for example, Figure 6 Computer 602. Figure 2 A workflow 200 for performing a medical imaging analysis task using a machine learning based model including a modality specific expert subnetwork and a cross-modality expert subnetwork is shown according to one or more embodiments. Figure 1 and Figure 2 . Figure 1 Method 100 and Figure 2 The workflow 200 is executed during an online or inference phase to perform a medical imaging analysis task using a machine learning based model that was trained during a previous offline or training phase.

[0022] exist Figure 1 In step 102, one or more input medical images are received. The one or more input medical images may depict anatomical objects (one or more) of interest to the patient, such as, for example, organs, bones, blood vessels, lesions, etc. The one or more input medical images may have any suitable one or more modalities, such as, for example, CT (computed tomography), MRI (magnetic resonance imaging), US (ultrasound), X-ray, or any other medical imaging modality or combination of medical imaging modalities. The one or more input medical images may be 2D (two-dimensional) images and / or 3D (three-dimensional) volumes. The one or more input medical images may be directly obtained from an image acquisition device (e.g., a computer) when the medical image is acquired. Figure 6 The image acquisition device 614) can be received by the computer system (e.g., Figure 6 The medical image may be received by loading a previously acquired medical image from a storage device or memory of a computer 602, or may be received from a remote computer system (e.g., Figure 6 Computer 602) receives it.

[0023] In one example, as in Figure 2 As shown in the workflow 200, one or more input medical images may be modality-specific data 202-A, 202-B, and 202-C (collectively referred to as modality-specific data 202) corresponding to a CT medical image, an MRI medical image, and a US medical image, respectively.

[0024] exist Figure 1 In step 104, one or more modality-specific expert subnetworks of the machine learning-based model are selected to perform a first processing of the one or more input medical images. The intra-modality expert selection module of the machine learning-based model receives the one or more input medical images, automatically selects the one or more modality-specific expert subnetworks, and routes the one or more input medical images to the one or more selected modality-specific expert subnetworks to perform the first processing. In one example, as in Figure 2 As shown in the workflow 200 , the modality-specific expert sub-networks 206 -A, 206 -B, and 206 -C (collectively referred to as the modality-specific expert sub-networks 206 ) of the base model are selected by the intra-modality expert selection module 204 of the base model to perform a first processing of the modality-specific data 202 .

[0025] In one embodiment, the machine learning based model is the base model. However, the machine learning based model can be implemented according to any suitable machine learning based architecture (e.g., a neural network architecture). One or more modality specific expert subnetworks are branches of the machine learning based model, which is trained to process modality specific data for one or more input medical images of a specific modality. One or more modality specific expert subnetworks can be implemented using any suitable machine learning based architecture, such as, for example, a convolutional neural network layer or a transformer neural network layer. The architecture of one or more modality specific expert subnetworks can be the same or different.

[0026] In one embodiment, one or more modality-specific expert subnetworks are learned during prior training of the machine learning-based model and are not predefined (e.g., by a user). For example, during training of the machine learning-based model, multiple modality-specific expert subnetworks to be included in the machine learning-based model and specialized tasks performed by each modality-specific expert subnetwork are learned. Such specialized tasks may include, for example, tasks specialized for different modalities, specialized for different anatomical objects, specialized for different imaging resolutions, specialized for determining edges, organs, anatomical information, etc., based on modality, robustness to modality-specific contrast changes, robustness to modality-specific artifacts (e.g., due to image acquisition technology), etc. Information learned from performing specialized tasks can be shared across modalities by modality-specific expert subnetworks. In other embodiments, a user may predefine the number of modality-specific expert subnetworks for each modality, and the specialized tasks performed by each modality-specific expert subject are learned during training of the machine learning-based model. In either embodiment, the number of modality-specific expert subnetworks may vary from modality to modality, as some modalities may be more difficult to represent.

[0027] The architecture of the modality-specific expert subnetwork is predefined. For example, in one embodiment, the multiple layers of each modality-specific expert subnetwork and the multiple parameters in each layer (i.e., depth and width) can be predefined (e.g., by a user) based on resource availability. In another embodiment, the multiple layers of each modality-specific expert subnetwork and the multiple parameters in each layer can be automatically determined, for example, using any suitable known method. The number of layers of the modality-specific expert subnetwork can be a single layer or multiple layers.

[0028] exist Figure 1 In step 106, a first processing of one or more input medical images is performed to perform a medical imaging analysis task using one or more selected modality-specific expert subnetworks. Figure 2As shown in the workflow 200 of , a first process is performed using a modality specific expert subnetwork 206. Each of the one or more selected modality specific expert subnetworks performs a different specialized task that contributes to the performance of the first process. Each of the one or more selected modality specific expert subnetworks receives as input at least one of the one or more input medical images and generates as output a result that contributes to the first process.

[0029] exist Figure 1 In step 108, the results of the first processing are merged into one or more merged result sets. The results of the first processing are merged by one or more expert data merging modules. The expert data merging module is a branch of the machine learning-based model that learns to appropriately merge the results of the first processing during the training of the machine learning-based model. In one example, as in Figure 2 As shown in the workflow 200 of , the first processing results output by the modality-specific expert sub-network 206 are merged into one or more merged result sets by the expert data merging modules 208-A and 208-B (collectively referred to as the expert data merging modules 208). In particular, the expert data merging module 208-A merges the first processing results output by the modality-specific expert sub-network 206-A into a first merged result set, and the expert data merging module 208-B merges the first processing results output by the modality-specific expert sub-network 206-B and 206-C into a second merged result set.

[0030] exist Figure 1 In step 110, one or more cross-modal expert sub-networks based on the machine learning model are selected to perform a second process on the one or more merged result sets. The cross-modal expert selection module based on the machine learning model receives the one or more merged result sets as input, selects one or more cross-modal expert sub-networks, and routes the one or more merged result sets to the one or more selected cross-modal expert sub-networks for performing the second process. In one example, as in Figure 2 As shown in the workflow 200, the cross-modal expert sub-networks 212-A, 212-B and 212-C (collectively referred to as the cross-modal expert sub-networks 212) of the base model are selected by the cross-modal expert selection module 210 of the base model to perform a second processing on one or more merged result sets output from the expert data merging module 208.

[0031] One or more cross-modal expert subnetworks are branches of a machine learning-based model that is trained to integrate the merged results of one or more modality-specific expert subnetworks across modalities (i.e., between each other). The cross-modal expert subnetworks enable similar representations of similar concepts across different modalities. For example, similar organs (or other anatomical objects of interest) depicted in medical images of different modalities will result in similar representations. One or more cross-modal expert subnetworks can be implemented using any suitable machine learning-based architecture, such as, for example, a convolutional neural network layer or a transformer neural network layer. The architecture of one or more cross-modal expert subnetworks can be the same or different.

[0032] In one embodiment, one or more cross-modal expert subnetworks are learned during the training of the machine learning-based model. For example, during the training of the machine learning-based model, multiple cross-modal expert subnetworks are included in the machine learning-based model, and specialized tasks performed by each cross-modal expert subnetwork are learned (e.g., specialized for feature integration between specific modalities, different anatomical objects, different image resolutions, etc.). Such specialized tasks may include, for example, tasks specialized for feature integration between specific modalities, specialized for different anatomical objects, specialized for different image resolutions, specialized for organ identification of different attributes of a given modality, specialized for relationships between anatomical structures (e.g., the stomach is close to the liver), specialized for edge detection, etc.

[0033] The architecture of the cross-modal expert sub-network is predefined. For example, the number of layers of each cross-modal expert sub-network and the number of parameters in each layer (i.e., depth and width) can be predefined by the user based on resource availability. The number of layers of the cross-modal expert sub-network can be a single layer or multiple layers.

[0034] exist Figure 1 In step 112, a second processing of one or more merged result sets is performed to perform the medical imaging analysis task using one or more selected cross-modality expert subnetworks. Figure 2 As shown in the workflow 200 of , the second processing is performed using a cross-modal expert subnetwork 212. Each of the one or more selected cross-modal expert subnetworks performs a different task, which contributes to the second processing performance. Each of the one or more selected cross-modal expert subnetworks receives at least one of the one or more merged result sets as input and generates a result that contributes to the second processing as output. The result output by the one or more selected cross-modal expert subnetworks represents the result of the medical imaging analysis task. The medical imaging analysis task may include any suitable medical imaging analysis task, such as, for example, detection, classification, segmentation, etc.

[0035] exist Figure 1 In step 114, the result of the second processing is output. In one embodiment, the result of the second processing can be output to another layer of expert subnetwork (e.g., intra-modality expert subnetwork or cross-modality expert subnetwork). In another embodiment, for example, in the case where one or more selected cross-modality expert subnetworks are the final layers of a machine learning-based model, the result of the second processing can be output as the final result of the medical imaging analysis task. The result of the second processing can be output by, for example, displaying the result of the evaluation on a display device of the computer system, storing the result of the second processing on a memory or storage device of the computer system, or by transmitting the result of the second processing to a remote computer system.

[0036] The machine learning based model is trained in a previous offline or training phase to perform medical imaging analysis tasks based on a set of training images of multiple different modalities. The machine learning based model can be trained via self-supervised learning, for example, using contextualized target predictions. Once trained, the machine learning based model can be applied online or during inference to perform medical imaging analysis tasks, for example, to perform Figure 1 Method 100 or Figure 2 Workflow 200.

[0037] In one embodiment, one or more input medical images of a single modality may be subjected to Figure 1 Method 100 and Figure 2 The workflow 200 of FIG. 1 is to efficiently perform medical imaging analysis tasks. When one or more input medical images have only a single modality, the intra-modality and cross-modality expert selection modules will select and run specific modality-specific expert sub-networks and cross-modality expert sub-networks only for that modality. Depending on the complexity of the modality, both the intra-modality and cross-modality expert selection modules can potentially select more or fewer specific modality-specific expert sub-networks and cross-modality expert sub-networks. Because only some expert sub-networks will process the data, the base model gains efficiency while maintaining scale. Figure 3 A workflow for efficiently performing medical imaging analysis tasks on input medical images of a single modality is shown in FIG.

[0038] Figure 3A workflow 300 is shown for efficiently performing a medical imaging analysis task on an input medical image of a single modality using a machine learning-based model according to one or more embodiments, the machine learning-based model including a modality-specific expert subnetwork and a cross-modality expert subnetwork. In the workflow 300, CT data 302 (e.g., an image) is received by an intra-modality expert selection module 304 of the machine learning-based model. The intra-modality expert selection module 304 selects CT modality-specific expert subnetworks 306-A and 306-B (collectively referred to as CT modality-specific expert subnetworks 306), and the CT modality-specific expert subnetworks 306 perform a first processing of the CT data 302 for performing the medical imaging analysis task. The merging module 308 merges the first processing results output by the CT modality-specific expert subnetworks 306 into one or more merged result sets. The cross-modality expert selection module 310 selects the cross-modality expert sub-networks 312 -A and 312 -B (collectively referred to as the cross-modality expert sub-networks 312 ), and the cross-modality expert sub-networks 312 perform a second processing of one or more merged result sets to perform the medical imaging analysis task.

[0039] Advantageously, the machine learning based models (e.g., base models) according to the embodiments described herein provide a combination of different expert subnetworks, which ensures modularity. In one embodiment, the trained machine learning based model can be fine-tuned to process medical images of new modalities. In order to expand the trained machine learning based model, new modality-specific and cross-modality expert subnetworks are added to the machine learning based model and fine-tuned. In order to ensure continuous learning and avoid catastrophic forgetting, when the newly added expert subnetwork is trained, the already trained expert subnetwork can be fine-tuned at a slower rate than the newly added expert subnetwork. The flexibility enabled by the machine learning based model can significantly reduce maintenance costs while enabling continuous learning.

[0040] Although the embodiments described herein are described with respect to input medical images of different medical imaging modalities, the embodiments described herein are not so limited. In one embodiment, one or more input medical images may alternatively be any type or types of data (e.g., text, speech, images, etc.). For example, one or more input medical images may be text-based data, and the language of the text may be considered a modality. Modality-specific expert subnetworks may be specialized for studying specific languages, and cross-modality expert subnetworks may integrate general language representations. In another example, one or more input medical images may be multimodal data including speech in different languages; text and speech; text, speech, and vision; or any other mixture of modalities relevant to performing the task.

[0041] The embodiments described herein are described with respect to the claimed systems and with respect to the claimed methods. Features, advantages, or alternative embodiments herein may be assigned to other claimed objects, and vice versa. In other words, claims to the system may be improved with features described or claimed in the context of the method. In this case, the functional features of the method are embodied by the target unit providing the system.

[0042] In addition, certain embodiments described herein are described with respect to methods and systems for utilizing trained machine learning-based models and with respect to methods and systems for training machine learning-based models. Features, advantages, or alternative embodiments herein may be assigned to other claimed objects, and vice versa. In other words, claims to methods and systems for training machine learning-based models may be improved with features described or claimed in the context of methods and systems for utilizing trained machine learning-based models, and vice versa.

[0043] In particular, the trained machine learning-based models applied in the embodiments described herein can be adjusted by methods and systems for training machine learning-based models. In addition, the input data of the trained machine learning-based models can include advantageous features and embodiments of the training input data, and vice versa. In addition, the output data of the trained machine learning-based models can include advantageous features and embodiments of the output training data, and vice versa.

[0044] Generally speaking, trained machine learning-based models mimic the cognitive functions that humans associate with other human minds. In particular, trained machine learning-based models are able to adapt to new environments and detect and extrapolate patterns through training based on training data.

[0045] In general, the parameters of a machine learning based model can be adjusted by means of training. In particular, supervised training, semi-supervised training, unsupervised training, reinforcement learning and / or active learning can be used. In addition, representation learning (an alternative term is "feature learning") can be used. In particular, the parameters of a trained machine learning based model can be iteratively adapted through several training steps.

[0046] In particular, the trained machine learning based model may include a neural network and / or a Bayesian network, and / or the trained machine learning based model may be based on k-means clustering, Q learning, a genetic algorithm and / or association rules. In particular, the neural network may be a deep neural network, a convolutional neural network or a convolutional deep neural network. In addition, the neural network may be an adversarial network, a deep adversarial network and / or a generative adversarial network.

[0047] Figure 4An embodiment of an artificial neural network 400 is shown in accordance with one or more embodiments. Alternative terms for "artificial neural network" are "neural network", "artificial neural network" or "neural network". Artificial neural network 400 may be used to implement: a machine learning network described herein - such as, for example Figure 1 The machine learning-based model used in method 100 comprises Figure 2 The intra-modality expert selection module 204, the modality-specific expert sub-network 206, the expert data merging module 208, the cross-modality expert selection module 210 and the cross-modality expert sub-network 212 in the workflow 200, and the machine learning-based model include Figure 3 The intra-modality expert selection module 304, the CT modality specific expert sub-network 306, the merging module 308, the cross-modality expert selection module 310 and the cross-modality expert sub-network 312 in the workflow 300 of FIG.

[0048] The artificial neural network 400 includes nodes 402-422 and edges 432, 434, ..., 436, wherein each edge 432, 434, ..., 436 is a directed connection from a first node 402-422 to a second node 402-422. Generally speaking, the first node 402-422 and the second node 402-422 are different nodes 402-422, but it is also possible that the first node 402-422 and the second node 402-422 are the same. For example, in Figure 4 , edge 432 is a directed connection from node 402 to node 406, and edge 434 is a directed connection from node 404 to node 406. Edges 432, 434, ..., 436 from a first node 402-422 to a second node 402-422 are also labeled as "incoming edges" for the second node 402-422 and as "outgoing edges" for the first node 402-422.

[0049] In this embodiment, the nodes 402-422 of the artificial neural network 400 may be arranged in layers 424-430, wherein each layer may include an inherent order introduced by edges 432, 434, ..., 436 between the nodes 402-422. In particular, the edges 432, 434, ..., 436 may only exist between adjacent node layers. Figure 4In the embodiment shown in , the input layer 424 includes only nodes 402 and 404, no incoming edges, the output layer 430 includes only node 422, no outgoing edges, and there are hidden layers 426, 428 between the input layer 424 and the output layer 430. In general, the number of hidden layers 426, 428 can be selected arbitrarily. The number of nodes 402 and 404 within the input layer 424 is generally related to the number of input values ​​of the neural network 400, and the number of nodes 422 within the output layer 430 is generally related to the number of output values ​​of the neural network 400.

[0050] In particular, a (real) number may be assigned as a value to each node 402-422 of the neural network 400. Here, x (n) i Indicates the value of the i-th node 402-422 of the n-th layer 424-430. The value of the node 402-422 of the input layer 424 is equivalent to the input value of the neural network 400, and the value of the node 422 of the output layer 430 is equivalent to the output value of the neural network 400. In addition, each edge 432, 434, ..., 436 may include a weight as a real number, in particular, the weight is a real number in the interval [-1, 1] or in the interval [0, 1]. Here, w (m，n) i，j The weight of the edge between the i-th node 402-422 of the m-th layer 424-430 and the j-th node 402-422 of the n-th layer 424-430 is indicated. (n，n+1) i，j Definition of the abbreviation w (n) i，j .

[0051] Specifically, to calculate the output value of the neural network 400, the input value is propagated through the neural network. Specifically, the value of the nodes 402-422 of the (n+1)th layer 424-430 can be calculated based on the value of the nodes 402-422 of the nth layer 424-430 by the following formula:

[0052]

[0053] In this article, function f is a transfer function (another term is "activation function"). Known transfer functions are step functions, sigmoid functions (e.g., logistic function, generalized logistic function, hyperbolic tangent function, inverse tangent function, error function, smoothed step function) or rectifier functions. Transfer functions are mainly used for normalization purposes.

[0054] In particular, the values ​​are propagated layer by layer through the neural network, where the value of the input layer 424 is given by the input of the neural network 400, where the value of the first hidden layer 426 can be calculated based on the value of the input layer 424 of the neural network, where the value of the second hidden layer 428 can be calculated based on the value of the first hidden layer 426, and so on.

[0055] To set the value w for edge (m，n) i，j , training data must be used to train the neural network 400. In particular, the training data includes training input data and training output data (denoted as t i ). For the training step, the neural network 400 is applied to the training input data to generate the calculated output data. In particular, the training data and the calculated output data include a plurality of values, the number of which is equal to the number of nodes of the output layer.

[0056] In particular, the comparison between the calculated output data and the training data is used to recursively adjust the weights within the neural network 400 (back propagation algorithm). In particular, the weights are changed according to the following formula

[0057]

[0058] where Y is the learning rate, and if the (n+1)th layer is not the output layer, then the number δ (n) j Based on δ (n+1)j is recursively calculated as

[0059]

[0060] And if the (n+1)th layer is the output layer 430 then

[0061]

[0062] where f' is the first derivative of the activation function, and y (n+1) j is the comparison training value for the j-th node of the output layer 430 .

[0063] Figure 5 A convolutional neural network 500 is shown in accordance with one or more embodiments. The convolutional neural network 500 may be used to implement: a machine learning network described herein—such as, for example, Figure 1 The machine learning based model used in method 100, the machine learning based model - comprising Figure 2 The intra-modality expert selection module 204, the modality-specific expert sub-network 206, the expert data merging module 208, the cross-modality expert selection module 210 and the cross-modality expert sub-network 212 in the workflow 200, and the machine learning-based model - including Figure 3 The intra-modality expert selection module 304, the CT modality specific expert sub-network 306, the merging module 308, the cross-modality expert selection module 310 and the cross-modality expert sub-network 312 in the workflow 300 of FIG.

[0064] exist Figure 5 In the embodiment shown in , the convolutional neural network 500 includes an input layer 502, a convolutional layer 504, a pooling layer 506, a fully connected layer 508, and an output layer 510. Alternatively, the convolutional neural network 500 may include several convolutional layers 504, several pooling layers 506, and several fully connected layers 508, as well as other types of layers. The order of the layers can be selected arbitrarily, and the fully connected layer 508 is usually used as the last layer before the output layer 510.

[0065] In particular, within the convolutional neural network 500, the nodes 512-520 of a layer 502-510 can be considered to be arranged as a d-dimensional matrix or a d-dimensional image. In particular, in the two-dimensional case, the values ​​of the nodes 512-520 indexed by i and j in the nth layer 502-510 can be denoted as X (n) [i，j] However, the arrangement of the nodes 512-520 of a layer 502-510 has no effect on the computations performed within such a convolutional neural network 500, as these are given solely by the structure and weights of the edges.

[0066] In particular, the convolution layer 504 is characterized by the structure and weights of the incoming edges that form the convolution operation based on a specific number of kernels. In particular, the structure and weights of the incoming edges are selected so that the value x of the node 514 of the convolution layer 504 is (n) k is calculated as the value x of node 512 based on the previous layer 502 (n-1) The convolution of x (n) k =K k *x (n-1) , where convolution* is defined in two dimensions as

[0067]

[0068] Here, the kth kernel K kis a d-dimensional matrix (a two-dimensional matrix in this embodiment), which is typically small compared to the number of nodes 512-518 (e.g., a 3×3 matrix or a 5×5 matrix). In particular, this implies that the weights of the incoming edges are not independent, but are selected so that they produce the convolution equation. In particular, for the kernel as a 3×3 matrix, there are only 9 independent weights (each entry of the kernel matrix corresponds to an independent weight), regardless of the number of nodes 512-520 in the corresponding layer 502-510. In particular, for a convolutional layer 504, the number of nodes 514 in the convolutional layer is equal to the number of nodes 512 in the previous layer 502 multiplied by the number of kernels.

[0069] If the nodes 512 of the previous layer 502 are arranged as a d-dimensional matrix, the use of multiple kernels can be interpreted as adding another dimension (denoted as the "depth" dimension) so that the nodes 514 of the convolution layer 504 are arranged as a (d+1)-dimensional matrix. If the nodes 512 of the previous layer 502 are already arranged as a (d+1)-dimensional matrix including the depth dimension, the use of multiple kernels can be interpreted as expanding along the depth dimension so that the nodes 514 of the convolution layer 504 are also arranged as a (d+1)-dimensional matrix, where the size of the (d+1)-dimensional matrix relative to the depth dimension is a factor of the number of kernels in the previous layer 502.

[0070] The advantage of using convolutional layers 504 is that the spatial local correlation of the input data can be exploited by enforcing local connectivity patterns between nodes in adjacent layers, specifically by each node being connected to only a small region of nodes in the previous layer.

[0071] exist Figure 5 In the embodiment shown in , the input layer 502 includes 36 nodes 512 arranged as a two-dimensional 6×6 matrix. The convolution layer 504 includes 72 nodes 514 arranged as two two-dimensional 6×6 matrices, each of which is the result of the convolution of the values ​​of the input layer with the kernel. Equivalently, the nodes 514 of the convolution layer 504 can be interpreted as being arranged as a three-dimensional 6×6×2 matrix, where the last dimension is the depth dimension.

[0072] The pooling layer 506 can be characterized by the structure and weights of the incoming edges and the activation function of its nodes 516, which form a pooling operation based on a nonlinear pooling function f. For example, in the two-dimensional case, the value x of the node 516 of the pooling layer 506 (n) It can be based on the value X of the node 514 of the previous layer 504 (n-1) To calculate

[0073] x (n) [i, j] = f(x (n-1) [id 1 , jd 2 ], ..., x(n-1) [id 1 +d 1 -1, jd 2 +d 2 -1]).

[0074] In other words, by using the pooling layer 506, the number of adjacent nodes 514 in the previous layer 504 can be replaced by a single node 516 calculated as a function of the value of the number of adjacent nodes in the pooling layer, thereby reducing the number of nodes 514, 516. In particular, the pooling function f can be a maximum function, an average value, or an L2 norm. In particular, for the pooling layer 506, the weights of the incoming edges are fixed and are not modified by training.

[0075] The advantage of using the pooling layer 506 is that the number of nodes 514, 516 and the number of parameters are reduced. This results in a reduction in the amount of computation in the network and controls overfitting.

[0076] exist Figure 5 In the embodiment shown in , the pooling layer 506 is a maximum pool, which replaces four neighboring nodes with only one node, and the value is the maximum of the four neighboring node values. The maximum pool is applied to each d-dimensional matrix of the previous layer; in this embodiment, the maximum pool is applied to each of the two 2D matrices, thereby reducing the number of nodes from 72 to 18.

[0077] The fully connected layer 508 is characterized by the fact that most, in particular all, edges exist between the nodes 516 of the previous layer 506 and the nodes 518 of the fully connected layer 508, and the weight of each edge can be adjusted individually.

[0078] In this embodiment, the nodes 516 of the previous layer 506 of the fully connected layer 508 are all displayed as a two-dimensional matrix, and are additionally displayed as unrelated nodes (indicated as a row of nodes, where the number of nodes is reduced for better presentability). In this embodiment, the number of nodes 518 in the fully connected layer 508 is equal to the number of nodes 516 in the previous layer 506. Alternatively, the number of nodes 516, 518 may be different.

[0079] Furthermore, in this embodiment, the values ​​of the nodes 520 of the output layer 510 are determined by applying the Softmax function to the values ​​of the nodes 518 of the previous layer 508. By applying the Softmax function, the sum of the values ​​of all the nodes 520 of the output layer 510 is 1, and all the values ​​of all the nodes 520 of the output layer are real numbers between 0 and 1.

[0080] The convolutional neural network 500 may also include a ReLU (rectified linear unit) layer or activation layer having a nonlinear transfer function. In particular, the number of nodes and the node structure contained in the ReLU layer are equivalent to the number of nodes and the node structure contained in the previous layer. In particular, the value of each node in the ReLU layer is calculated by applying a rectification function to the value of the corresponding node of the previous layer.

[0081] The inputs and outputs of different CNN blocks can be wired using sums (residual / dense NNs), element-wise multiplications (attention), or other differentiable operators. Thus, if the entire pipeline is differentiable, the CNN architecture can be nested instead of sequential.

[0082] In particular, the convolutional neural network 500 can be trained based on a back-propagation algorithm. To prevent overfitting, regularization methods can be used, such as dropout of nodes 512-520, random pooling, use of artificial data, weight decay based on L1 or L2 norm or maximum norm constraints. Different loss functions can be combined for training the same neural network to reflect joint training objectives. A subset of neural network parameters can be excluded from optimization to retain weights pre-trained on another dataset.

[0083] The systems, devices, and methods described herein can be implemented using digital circuit systems or using one or more computers using known computer processors, memory units, storage devices, computer software, and other components. Typically, a computer includes a processor for executing instructions and one or more memories for storing instructions and data. A computer may also include or be coupled to one or more mass storage devices, such as one or more magnetic disks, internal hard disks and removable disks, magneto-optical disks, optical disks, and the like.

[0084] The systems, devices and methods described herein can be implemented using computers operating in a client-server relationship. Typically, in such a system, the client computer is located remotely from the server computer and interacts via a network. The client-server relationship can be defined and controlled by computer programs running on the respective client and server computers.

[0085] The systems, devices, and methods described herein can be implemented within a network-based cloud computing system. In such a network-based cloud computing system, a server or another processor connected to the network communicates with one or more client computers via the network. For example, a client computer can communicate with a server via a web browser application that resides and runs on the client computer. The client computer can store data on the server and access the data via the network. The client computer can transmit a request for data or a request for an online service to the server via the network. The server can perform the requested service and provide the data to (one or more) client computers. The server can also transmit data suitable for causing the client computer to perform a specific function, such as performing a calculation, displaying specific data on a screen, etc. For example, the server can transmit a request suitable for causing the client computer to perform one or more steps or functions of the methods and workflows described herein, including Figure 1-3 One or more steps or functions of the methods and workflows described herein - including Figure 1-3 One or more steps or functions of the method and workflow described herein may be performed by a server or another processor in a network-based cloud computing system. Certain steps or functions of the method and workflow described herein include Figure 1-3 One or more steps of the method and workflow described herein may be performed by a client computer in a network-based cloud computing system. The steps or functions of the method and workflow described herein include Figure 1-3 One or more steps of - may be performed by a server and / or a client computer in a network-based cloud computing system in any combination.

[0086] The systems, apparatus, and methods described herein may be implemented using a computer program product tangibly embodied in an information carrier, such as in a non-transitory machine-readable storage device, for execution by a programmable processor; and the methods and workflow steps described herein - including Figure 1-3 One or more steps or functions of a program may be implemented using one or more computer programs executable by such a processor. A computer program is a set of computer program instructions that can be used, directly or indirectly, in a computer to perform a specific activity or bring about a specific result. A computer program may be written in any form of programming language, including compiled or interpreted languages, and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0087] Figure 6A high-level block diagram of an example computer 602 that can be used to implement the systems, apparatus, and methods described herein is depicted in FIG. The computer 602 includes a processor 604 operably coupled to a data storage device 612 and a memory 610. The processor 604 controls the overall operation of the computer 602 by executing computer program instructions that define such operations. The computer program instructions may be stored in the data storage device 612 or other computer-readable medium and loaded into the memory 610 when execution of the computer program instructions is desired. Thus, Figure 1-3 The methods and workflow steps or functions may be defined by computer program instructions stored in the memory 610 and / or the data storage device 612 and controlled by the processor 604 executing the computer program instructions. For example, the computer program instructions may be implemented as computer executable code programmed by a person skilled in the art to execute Figure 1-3 Thus, by executing the computer program instructions, the processor 604 performs Figure 1-3 The computer 602 may also include one or more network interfaces 606 for communicating with other devices via a network. The computer 602 may also include one or more input / output devices 608 (e.g., a display, keyboard, mouse, speaker, buttons, etc.) that enable a user to interact with the computer 602.

[0088] The processor 604 may include both general and special purpose microprocessors, and may be the sole processor or one of multiple processors of the computer 602. For example, the processor 604 may include one or more central processing units (CPUs). The processor 604, the data storage device 612, and / or the memory 610 may include, be supplemented by, or incorporate one or more application specific integrated circuits (ASICs) and / or one or more field programmable gate arrays (FPGAs).

[0089] The data storage device 612 and the memory 610 each include a tangible, non-transitory computer-readable storage medium. The data storage device 612 and the memory 610 may each include a high-speed random access memory, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a double data rate synchronous dynamic random access memory (DDR RAM), or other random access solid-state memory devices, and may include non-volatile memory, such as: one or more disk storage devices, such as internal hard disks and removable disks, magneto-optical disk storage devices, optical disk storage devices, flash memory devices; semiconductor memory devices, such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), compact disk read-only memory (CD-ROM), digital versatile disk read-only memory (DVD-ROM) disks; or other non-volatile solid-state storage devices.

[0090] Input / output devices 608 may include peripheral devices such as printers, scanners, display screens, etc. For example, input / output devices 608 may include a display device such as a cathode ray tube (CRT) or liquid crystal display (LCD) monitor for displaying information to a user, a keyboard, and a pointing device such as a mouse or trackball through which a user can provide input to computer 602.

[0091] The image acquisition device 614 can be connected to the computer 602 to input image data (e.g., medical images) to the computer 602. It is possible to implement the image acquisition device 614 and the computer 602 as one device. It is also possible that the image acquisition device 614 and the computer 602 communicate wirelessly via a network. In a possible embodiment, the computer 602 can be located remotely relative to the image acquisition device 614.

[0092] Any or all of the systems and devices discussed herein may be implemented using one or more computers, such as computer 602 .

[0093] Those skilled in the art will recognize that the actual computer or computer system implementation may have other structures and may also include other components, and Figure 6 is a high-level representation of some components of such a computer for purposes of illustration.

[0094] Independent of the use of the grammatical term, individuals with male, female or other identities are also included in the term.

[0095] The foregoing detailed description is to be understood as being illustrative and exemplary in every respect, rather than restrictive, and the scope of the invention disclosed herein is not determined from the detailed description, but from the claims interpreted as the full breadth permitted under patent law. It will be understood that the embodiments shown and described herein are merely illustrative of the principles of the invention, and that various modifications may be implemented by those skilled in the art without departing from the scope and spirit of the invention. Various other feature combinations may be implemented by those skilled in the art without departing from the scope and spirit of the invention.

[0096] The following is a list of non-limiting exemplary embodiments disclosed herein:

[0097] Illustrative Embodiment 1. A computer-implemented method, comprising: receiving one or more input medical images; selecting one or more modality-specific expert subnetworks of a machine learning-based model to perform a first processing of the one or more input medical images; performing the first processing of the one or more input medical images to perform a medical imaging analysis task using the one or more selected modality-specific expert subnetworks; merging the results of the first processing into one or more merged result sets; selecting one or more cross-modality expert subnetworks of a machine learning-based model to perform a second processing of the one or more merged result sets; performing the second processing of the one or more merged result sets to perform the medical imaging analysis task using the one or more selected cross-modality expert subnetworks; and outputting the results of the second processing.

[0098] Illustrative Embodiment 2. The computer-implemented method of Illustrative Embodiment 1, wherein the machine learning based model is trained via self-supervised learning.

[0099] Illustrative Embodiment 3. The computer-implemented method of any of Illustrative Embodiments 1-2, wherein performing a first processing of one or more input medical images so as to perform a medical imaging analysis task using one or more selected modality-specific expert subnetworks comprises: processing modality-specific data of the one or more input medical images using the one or more selected modality-specific expert subnetworks.

[0100] Illustrative Embodiment 4. The computer-implemented method of any of Illustrative Embodiments 1-3, wherein performing a second processing of the one or more merged result sets so as to perform the medical imaging analysis task using the one or more selected cross-modality expert subnetworks comprises: integrating the one or more merged result sets among each other.

[0101] Illustrative Embodiment 5. The computer-implemented method of any of Illustrative Embodiments 1-4, wherein during the training of the machine learning based network, a plurality of modality-specific expert sub-networks in the machine learning based network are learned for each modality.

[0102] Illustrative Embodiment 6. The computer-implemented method of any of Illustrative Embodiments 1-5, wherein a plurality of modality-specific expert sub-networks in the machine learning based network are predefined for each modality.

[0103] Illustrative Embodiment 7. The computer-implemented method of any of Illustrative Embodiments 1-6, wherein during the training of the machine learning based network, a plurality of cross-modal expert sub-networks in the machine learning based network are learned.

[0104] Illustrative Embodiment 8. The computer-implemented method of any of Illustrative Embodiments 1-7, wherein the plurality of layers and the plurality of parameters of each layer of each modality-specific expert sub-network and each cross-modality expert sub-network in the machine learning based network are defined based on resource availability.

[0105] Illustrative Embodiment 9. The computer-implemented method of any one of Illustrative Embodiments 1-8, wherein the machine learning based model is the base model.

[0106] Illustrative embodiment 10. An apparatus comprising: a component for receiving one or more input medical images; a component for selecting one or more modality-specific expert subnetworks of a machine learning-based model to perform a first processing of the one or more input medical images; a component for performing the first processing of the one or more input medical images to perform a medical imaging analysis task using the one or more selected modality-specific expert subnetworks; a component for merging the results of the first processing into one or more merged result sets; a component for selecting one or more cross-modality expert subnetworks of a machine learning-based model to perform a second processing of the one or more merged result sets; a component for performing the second processing of the one or more merged result sets to perform the medical imaging analysis task using the one or more selected cross-modality expert subnetworks; and a component for outputting the second processing results.

[0107] Illustrative Embodiment 11. The apparatus of Illustrative Embodiment 10, wherein the machine learning based model is trained via self-supervised learning.

[0108] Illustrative embodiment 12. The apparatus of any of illustrative embodiments 10-11, wherein the means for performing a first processing of one or more input medical images so as to perform a medical imaging analysis task using one or more selected modality-specific expert subnetworks comprises: means for processing modality-specific data of one or more input medical images using one or more selected modality-specific expert subnetworks.

[0109] Illustrative Embodiment 13. The apparatus of any of Illustrative Embodiments 10-12, wherein the means for performing a second processing of one or more merged result sets in order to perform a medical imaging analysis task using one or more selected cross-modality expert subnetworks comprises: means for integrating the one or more merged result sets with each other.

[0110] Illustrative Embodiment 14. The apparatus of any of Illustrative Embodiments 10-13, wherein during training of the machine learning based network, a plurality of modality-specific expert sub-networks in the machine learning based network are learned for each modality.

[0111] Illustrative embodiment 15. A non-transitory computer-readable medium storing computer program instructions that, when executed by a processor, cause the processor to perform operations including: receiving one or more input medical images; selecting one or more modality-specific expert subnetworks of a machine learning-based model to perform a first processing of the one or more input medical images; performing the first processing of the one or more input medical images to perform a medical imaging analysis task using the one or more selected modality-specific expert subnetworks; merging the results of the first processing into one or more merged result sets; selecting one or more cross-modality expert subnetworks of a machine learning-based model to perform a second processing of the one or more merged result sets; performing the second processing of the one or more merged result sets to perform the medical imaging analysis task using the one or more selected cross-modality expert subnetworks; and outputting the results of the second processing.

[0112] Illustrative Embodiment 16. The non-transitory computer readable medium of Illustrative Embodiment 15, wherein the machine learning based model is trained via self-supervised learning.

[0113] Illustrative Embodiment 17. The non-transitory computer readable medium of any of Illustrative Embodiments 15-16, wherein a plurality of modality-specific expert sub-networks in the machine learning based network are predefined for each modality.

[0114] Illustrative Embodiment 18. The non-transitory computer readable medium of any of Illustrative Embodiments 15-17, wherein during the training of the machine learning based network, a plurality of cross-modal expert sub-networks in the machine learning based network are learned.

[0115] Illustrative Embodiment 19. The non-transitory computer readable medium of any of Illustrative Embodiments 15-18, wherein the plurality of layers and the plurality of parameters of each layer of each modality-specific expert sub-network and each cross-modality expert sub-network in the machine learning based network are defined based on resource availability.

[0116] Illustrative Embodiment 20. The non-transitory computer readable medium of any one of Illustrative Embodiments 15-19, wherein the machine learning based model is the base model.

Claims

1. A computer-implemented method comprising: receiving one or more input medical images; selecting one or more modality-specific expert subnetworks of the machine learning based model to perform a first processing of one or more input medical images; performing a first processing of one or more input medical images to perform a medical imaging analysis task using one or more selected modality-specific expert subnetworks; merging the results of the first processing into one or more merged result sets; selecting one or more cross-modal expert sub-networks of the machine learning-based model to perform a second processing of the one or more merged result sets; performing a second processing of the one or more merged result sets to perform a medical imaging analysis task using the one or more selected cross-modality expert sub-networks; and The result of the second process is output.

2. The computer-implemented method of claim 1 , wherein: Machine learning based models are trained via self-supervised learning.

3. The computer-implemented method of claim 1 , wherein: Performing a first processing of the one or more input medical images in order to perform a medical imaging analysis task using the one or more selected modality specific expert subnetworks comprises: The modality specific data of the one or more input medical images are processed using the one or more selected modality specific expert sub-networks.

4. The computer-implemented method of claim 1 , wherein: Performing a second processing of the one or more merged result sets so as to perform a medical imaging analysis task using the one or more selected cross-modality expert sub-networks comprises: Combines one or more merged result sets with each other.

5. The computer-implemented method of claim 1 , wherein: During training of the machine learning based network, a plurality of modality-specific expert sub-networks in the machine learning based network are learned for each modality.

6. The computer-implemented method of claim 1 , wherein: Multiple modality-specific expert sub-networks in the machine learning-based network are predefined for each modality.

7. The computer-implemented method of claim 1 , wherein: During training of the machine learning based network, a plurality of cross-modal expert sub-networks in the machine learning based network are learned.

8. The computer-implemented method of claim 1 , wherein: Multiple layers and multiple parameters of each layer of each modality-specific expert sub-network and each cross-modality expert sub-network in the machine learning based network are defined based on resource availability.

9. The computer-implemented method of claim 1 , wherein: The machine learning based model is the basic model.

10. An apparatus comprising: means for receiving one or more input medical images; means for selecting one or more modality-specific expert subnetworks of a machine learning based model to perform a first processing of one or more input medical images; means for performing a first processing of one or more input medical images so as to perform a medical imaging analysis task using one or more selected modality-specific expert subnetworks; means for merging the results of the first processing into one or more merged result sets; means for selecting one or more cross-modal expert sub-networks of the machine learning-based model to perform a second processing of the one or more merged result sets; means for performing a second processing of the one or more merged result sets to perform a medical imaging analysis task using the one or more selected cross-modality expert sub-networks; and A means for outputting a second processing result.

11. The device according to claim 10, wherein: Machine learning based models are trained via self-supervised learning.

12. The device according to claim 10, wherein: The means for performing a first processing of the one or more input medical images in order to perform a medical imaging analysis task using the one or more selected modality specific expert subnetworks comprises: Means for processing modality specific data of one or more input medical images using one or more selected modality specific expert sub-networks.

13. The device according to claim 10, wherein: The means for performing a second processing of one or more merged result sets to perform a medical imaging analysis task using one or more selected cross-modality expert sub-networks comprises: A component used to integrate one or more merged result sets between each other.

14. The device according to claim 10, wherein: During training of the machine learning based network, a plurality of modality-specific expert sub-networks in the machine learning based network are learned for each modality.

15. A non-transitory computer readable medium storing computer program instructions which, when executed by a processor, cause the processor to perform operations comprising: receiving one or more input medical images; selecting one or more modality-specific expert subnetworks of the machine learning based model to perform a first processing of one or more input medical images; performing a first processing of one or more input medical images to perform a medical imaging analysis task using one or more selected modality-specific expert subnetworks; merging the results of the first processing into one or more merged result sets; selecting one or more cross-modal expert sub-networks of the machine learning-based model to perform a second processing of the one or more merged result sets; performing a second processing of the one or more merged result sets to perform a medical imaging analysis task using the one or more selected cross-modality expert sub-networks; and The result of the second process is output.

16. The non-transitory computer readable medium of claim 15, wherein: Machine learning based models are trained via self-supervised learning.

17. The non-transitory computer readable medium of claim 15, wherein: Multiple modality-specific expert sub-networks in the machine learning based network are predefined for each modality.

18. The non-transitory computer readable medium of claim 15, wherein: During training of the machine learning based network, a plurality of cross-modal expert sub-networks in the machine learning based network are learned.

19. The non-transitory computer readable medium of claim 15, wherein: Multiple layers and multiple parameters of each layer of each modality-specific expert sub-network and each cross-modality expert sub-network in the machine learning based network are defined based on resource availability.

20. The non-transitory computer readable medium of claim 15, wherein: The machine learning based model is the basic model.

Images