Computer-implemented method, data processing apparatus, and computer program for image analysis

Abstract

To provide a computer-implemented method and the like for training a machine learning model for image analysis.SOLUTION: A method comprising: receiving an input training dataset including real training data and augmented training data; and iteratively training an encoder of a machine-learning model to obtain a trained encoder, the iterative training comprising training a discriminator model by minimizing a discriminator loss function using the input training dataset to obtain a trained discriminator, the trained discriminator being configured to discriminate whether input data is real data or augmented data; and training the encoder by maximizing a discriminator error using the training dataset and the trained discriminator to obtain a trained encoder, the trained encoder being configured to invariantly encode real data and augmented data to a same representation space.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a method of image analysis in the field of computer vision. More specifically, the present invention relates to a method of training a machine learning model for image analysis, a method of image analysis, and related data processing apparatus, computer programs, and computer readable storage media. [Background technology]

[0002] The computer vision technique of image segmentation refers to the process of dividing an image into image segments by classifying pixels in the image and marking or masking target regions. Image segmentation is a technique that is widely applied in technical fields such as medical image analysis, object detection in unmanned driving, and geographical and geological monitoring.

[0003] For example, in the field of geographic monitoring, flood monitoring is an active area of ​​research. Flood monitoring by analysis of real-time video camera footage and image datasets typically requires segmentation machine learning models. The resulting segments or masks may be used to determine the extent of the flood, which may then be used to inform emergency warning and disaster management systems.

[0004] Flooding is the most common natural hazard in the UK and is recognised as the costliest natural hazard in Europe, with damages estimated to run into tens of millions of euros in recent years. Flood detection is considered an important safety measure and has attracted widespread interest from various research and technology groups around the world. Existing approaches to flood detection are typically based on sensors, satellite imagery and video camera footage.

[0005] However, sensors and satellite imagery have several drawbacks. Sensors can be expensive and difficult to deploy. Satellite and airborne optical techniques are hampered by being limited to daytime operation and their inability to map floodwaters beneath clouds and vegetation. SAR imaging could alleviate some of the limitations above, but SAR satellite overpasses are infrequent (once or twice a day) and therefore rarely capture rising water levels.

[0006] As a result, river video cameras have emerged as a popular alternative, with advantages including the ability to continuously broadcast live images, low maintenance and installation costs, and the ease with which footage can be communicated over standard or mobile broadband.

[0007] To exploit video camera footage of rivers, it is usually necessary to use a water body segmentation model. Figure 1 shows an example of the application of a water body segmentation model adapted from the work of Vandaele, R. et al. (Hydrology and Earth System Sciences. 25. 4435-4453. 10.5194 / hess-25-4435-2021). Panel A shows an image of a river (note that the original image contains color data). Panel B shows the segmentation result, where the areas of concern classified as water by the water body segmentation model are masked in white.

[0008] In the context of water body segmentation, the current state of the art in segmentation is believed to be achieved using AquaNET (see paper Erfani, S. et al,ATLANTIS: A benchmark for semantic segmentation of waterbody images. Environmental Modelling & Software,105333), which uses a convolutional neural network (CNN) based model, PSPNet, as the network backbone.

[0009] However, water can appear in widely different forms, making it impossible to collect enough real-world data to faithfully represent all forms. Figure 2 illustrates the different variations in water imagery that a water body segmentation model may encounter. This includes variations in image quality compared to the best-case scenario (increased ISO noise, raindrops obscuring the camera's view), variations in image acquisition technology (infrared illumination), variations in lighting quality (reflective water, nighttime image acquisition), the presence or absence of precipitation (snow, rain, fog), and the nature of the water itself of interest (muddy or murky water, bubbling water, water with submerged objects).

[0010] Therefore, an ideal water body segmentation model needs to generalize to all possible out-of-distribution (OoD) scenarios in which water may appear, despite only having access to a subset of real in-distribution images during inference. Simulation or extension to synthetically model OoD scenarios helps to fill this gap and address data scarcity concerns by generating artificial images in a controlled manner. Although structurally similar to real images, such artificial images exhibit a modality shift as the data generation process is fundamentally different from real data.

[0011] For example, Figure 3 shows a simulation of changing the color attributes of water (note that again, the original image contains color data). The turbidity of the water is simulated by manipulating the color of the image.

[0012] Existing approaches that ensure generalization (or generalizability) of segmentation models to multiple modalities are done in the context of domain generalization and adaptation. Known techniques have demonstrated that synthetic images can be used to overcome data scarcity problems in computer vision tasks such as object detection for robotic grasping. However, the inventors have noticed that no techniques have been considered that consider downstream OoD segmentation or classification tasks (rather than domain generalization and adaptation).

[0013] Of course, the problem of data scarcity for downstream OoD segmentation tasks is not unique to the field of flood management: although existing approaches are known to combine real and simulated data to train large-scale image segmentation models (and, indeed, other image analysis models), it is desirable to develop techniques that ensure effective use of the initial modality data (simulated or other augmented data) when it is known that downstream tasks will be performed on different modality data. Summary of the Invention

[0014] The invention is defined in the independent claims, to which reference should be made here. Further characteristics are set out in the dependent claims.

[0015] According to an aspect of the invention, there is provided a computer-implemented method of training a machine learning model for image analysis, comprising the steps of: receiving an input training dataset including actual training data corresponding to an actual modality and augmented training data; and iteratively training an encoder of the machine learning model to obtain a trained encoder; The repeated training includes: training a discriminator model by minimizing a discriminator loss function using the input training data set to obtain a trained discriminator, the trained discriminator being configured to discriminate whether input data is real data or augmented data; training the encoder by maximizing a discriminator error using the input training data set and the trained discriminator to obtain the trained encoder, the trained encoder being configured to invariantly encode real data and augmented data into a real representation space such that the representation preserves information about the real modality; having A method is provided.

[0016] According to a further aspect of the present invention there is provided a computer implemented method of image analysis comprising the steps of: Receiving an input image; processing the input image using a trained machine learning model to obtain an analysis result of the input image; The trained machine learning model is a trained encoder that is iteratively trained using an input training dataset that includes real training data corresponding to a real modality and augmented training data; and a trained additional downstream layer trained by a multi-objective optimization procedure using the input training dataset and the trained encoder; The repeated training includes: training a discriminator model by minimizing a discriminator loss function using the input training data set to obtain a trained discriminator, the trained discriminator being configured to discriminate whether input data is real data or augmented data; training the encoder by maximizing a discriminator error using the input training data set and the trained discriminator to obtain the trained encoder, the trained encoder being configured to invariantly encode real data and augmented data into a real representation space such that the representation preserves information about the real modality; having A method is provided. [Brief description of the drawings]

[0017] By way of example only, reference is made to the accompanying drawings in which: [Figure 1] 1 is an example of image analysis in the form of water body segmentation. [Diagram 2] 1 is a set of exemplary OoD scenarios for water body segmentation. [Diagram 3] 1 is an example of image augmentation showing control of water color attributes through simulation or synthesis. [Figure 4] Schematic diagram of a DPT-based machine learning architecture for water body segmentation. [Diagram 5] 1 is a representation of the modality shift in the representation space induced by a conventional encoder. [Figure 6] 1 is an overview of NCI techniques implemented in accordance with embodiments. [Figure 7] 1 is a flowchart illustrating a general method for training a machine learning model for image analysis, according to an embodiment. [Figure 8] 1 is an exemplary training process for a water body segmentation model including NCI, according to an embodiment. [Figure 9] 1 is an exemplary training process for a medical image classification model including NCI, according to an embodiment. [Figure 10] 1 is a flow chart illustrating a general method of image analysis according to an embodiment. [Figure 11] 1 is an exemplary inference process for a water body segmentation model including NCI, according to an embodiment. [Figure 12] 1 is an exemplary inference process for a medical image classification model including NCI, according to an embodiment. [Figure 13a] 13 is a further exemplary training process for a water body segmentation model including NCI, according to an embodiment. [Figure 13b] 13 is a further exemplary training process for a water body segmentation model including NCI, according to an embodiment. [Figure 13c] 13 is a further exemplary training process for a water body segmentation model including NCI, according to an embodiment. [Figure 14] 13 is a further exemplary training process for medical image classification including NCI, according to an embodiment. [Figure 15] 1 is a collection of water images and corresponding water segments determined according to a trained model trained according to an embodiment. [Figure 16] FIG. 2 is a block diagram of a computing means for the implementation of a method according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] According to an aspect of the invention, a computer-implemented method of training a machine learning model for image analysis is provided. The method includes receiving an input training dataset including real training data or, equivalently, training data corresponding to, acquired from, or representative of a first modality. The input training dataset further includes augmented training data (or, equivalently, training data corresponding to, acquired from, or representative of a second modality). The augmented training data can be considered as extending or supplementing information derivable from the real training data. The augmented training data can be manipulated real data, simulated or synthetic data, or other supplementary data. In the context of image analysis in the form of water body segmentation, by way of example, the real training data can be real, acquired, unaltered images of water in various environments, and the augmented data can be synthetic images that are entirely artificially generated or synthesized by manipulating real images of water (e.g., by color or texture manipulation).

[0019] The term "modality" in this context refers to the manner in which data is experienced or captured. Thus, actual modality refers to the manner in which actual training data (and actual data) is captured. Augmented data may similarly correspond to an augmented modality that is different from the actual modality. Although not required, the augmented data may also be "actual" in the sense that the data is not real or artificial. An augmented modality is different from an actual modality.

[0020] The method also includes iterative training of an encoder of the machine learning model. That is, a (complete) machine learning model for image analysis includes an encoder component. In general, an encoder receives an input sequence (e.g., image data) and converts the sequence into a fixed-size representation or context vector (or embedding or projection). This representation ideally contains relevant information derived from the input sequence and serves as a high-level summary of the input. By training the encoder (i.e., optimizing the weights and biases and other parameters) until it converges, a trained encoder can be obtained. A trained encoder is sometimes called a Non-Communitative Invariant (NCI) encoder.

[0021] The iterative training of the encoder includes training a discriminator model and training the encoder itself. Training the discriminator model may be performed by minimizing a first loss function value (discriminator loss function). In effect, the discriminator model can be considered as a classifier. The computational means passes the input data set to the discriminator (in the form of a representation or encoding obtained from the encoder following encoding of the input data), which may perform reinforcement learning and / or adversarial processing to classify both the actual training data and the augmented data. The discriminator loss function penalizes the discriminator when it misclassifies the actual training data as augmented data and the augmented training data as actual data. The discriminator may update the weights and biases, for example, by backpropagation from the discriminator loss function through the discriminator model. The trained discriminator model is configured to discriminate whether new unknown input data (and training data) is actual data or augmented data.

[0022] Training the encoder can be performed by maximizing a second Muramuro function (discriminator error or encoder loss function). The objective here is to configure the encoder such that the discriminator is partially fooled, in the sense that the real data is mapped into a representation space (encoding space or latent space) corresponding to the real data, and the augmented data is also mapped into a representation space corresponding to the real data. Therefore, the invariance (within the encoder) achieved by this process is non-commutative. The trained encoder is therefore configured to invariantly encode the real and augmented data into the same real representation space (corresponding to the real modality).

[0023] Thus, the trained encoder maps the augmented data to the real representation space, but not the real data to the augmented representation space. In this sense, the trained encoder encodes or generates the representation in a non-commutative manner. The trained encoder fixes the direction of invariance to the real data modality. While known techniques learn invariances by eliminating modality-specific information from all modalities (both real and augmented), the embodiments make the learning invariant to the augmented modality, preserving all real modality-specific information in the process. Thus, learning the invariances in a non-commutative manner preserves the real modality information. Thus, the method according to the embodiments provides an approach to effectively exploit or use augmented data to train large-scale models, such as vision transformers to achieve OoD generalization, in scenarios where a limited amount of real data is available.

[0024] The performance of machine learning models such as deep neural networks is known to scale with the size of the training dataset, which is especially true for encoder-based architectures. The present disclosure enables tuning the size of the training dataset to aid in the generalization ability of image analysis machine learning models to various use cases such as water body segmentation and medical image classification.

[0025] As the modality discriminator trains, it becomes better at distinguishing between the real and augmented modalities. Training the encoder to fool the discriminator causes it to generate representations that do not contain information specific to the synthetic modality.

[0026] Related work includes the Gradient Reversal Layer (GRL) for domain adaptation (the reader is referred to the work of Ganin, Y. et al. Domain-adversarial training of neural networks. J. Mach. Learn. Res. 17, 1 (January 2016), 2096-2030). GRL forces the backbone encoder to be robust to modality shifts by enforcing commutative form invariance. In other words, the GRL method tricks the discriminator by flipping the modality labels, making the model invariant to both real and synthetic modality information. However, if the modality-shifted data is used as an augmentation at training time, there is no need to enforce bidirectional (commutative) invariance (imposed by the GRL method). That is, if data from another modality is used as an augmentation of the training dataset (not encountered during inference), this commutativity is not necessary. Instead, by having the encoder only map the augmented data to the real representation space to maximize the discriminator error, and not vice versa, models trained according to the present disclosure use the modality shift as a regularizer, allowing the encoder-based model to leverage multi-modal data sources to aid in OoD generalization, resulting in models with significantly improved parameter efficiency.

[0027] Combining real and augmented data in a modality-invariant manner improves the accuracy of downstream segmentation and classification. In the example of an encoder-based (or transformer-encoder-based) segmentation model, sampling from the proposed non-commutative invariant (NCI) representation space improves both in-distribution and OoD segmentation performance. Modality invariance allows the encoder to better exploit the increased number of synthetic samples.

[0028] Optionally, the machine learning model may have additional downstream layers, i.e., the machine learning model may have layers in addition to the encoder. Once trained, the additional downstream layers, together with the trained encoder, may be configured (by training) to output an analysis result of the input data, the input data corresponding to the same (or similar) real modality as that of the real training data.

[0029] Optionally, the method for training a machine learning model for image analysis may further comprise iterative training of any additional downstream layers, for example by a multi-objective optimization procedure (or Pareto optimization) may be used to jointly optimize the encoder and any additional layers.

[0030] Optionally, the machine learning model may have a transformer encoder, a reassembly module, and a fusion module. Once trained, the trained machine learning model may have a trained transformer encoder, a trained reassembly module, and a trained fusion module. This configuration is well suited to processing representations with a constant and relatively high resolution, and ensures a global receptive field at every stage. Such properties allow the machine learning model to provide finer-grained and more globally consistent predictions compared to fully convolutional networks.

[0031] Optionally, the method may include the use of a trained machine learning model. That is, the method may include receiving an input image (or images). The input image may be input from a repository of images, such as a database. Additionally or alternatively, the input image may be input in real time, for example from live video footage. The method may then process the input image with the trained machine learning model to obtain an analysis result (e.g., image segment or image classification) of the input image. The input data corresponds to the same (or similar) actual modality as that of the actual training data. In this way, inference can be performed on the input data using information derived from the augmented training data in a manner that takes into account OoD generalization.

[0032] According to a further aspect of the invention, there is provided a computer-implemented method of image analysis, which receives an input image and obtains an analysis result using a machine learning model trained in accordance with the techniques disclosed above and further herein.

[0033] In general, the techniques presented here provide an efficient way to exploit data augmentation for any kind of task that requires invariance to the augmented modality.

[0034] For example, optionally, the method of image analysis and / or the method of training a machine learning model for image analysis is for the segmentation of water images. The input training dataset in this case thus includes real training water images and augmented or simulated training water images. Because water appears quite differently under different conditions, it is not feasible to collect enough real images that faithfully represent all such conditions. The techniques presented herein can identify synthetic data as a valid alternative to real data for modeling the distributional changes in water appearance.

[0035] Optionally, when the method is for segmentation of water images, the input water images may be acquired in real time (e.g., from a live video stream). The real-time images may be processed using a trained machine learning model to obtain water body segmentation results.

[0036] Optionally, the water body segmentation result may be used to determine whether an alarm needs to be issued. For example, the water body segmentation result (e.g., water image segments) (or derived data thereof) may be compared to a threshold. For example, an alarm is triggered when a segment occupying 15% of the image frame is compared to a predefined threshold of 10%. The alarm may be an audible signal or may be a data signal for transmission to appropriate authorities or personnel.

[0037] Combining real and augmented water images to train machine learning water body segmentation models significantly improves performance compared to a direct mixture of the two modalities. The technique presented here provides a state-of-the-art model for water body segmentation that can be used to inform emergency warning and disaster management systems addressing flooding, one of the most common causes of natural disasters.

[0038] Optionally, the method of image analysis and / or the method of training a machine learning model for image analysis is for medical image analysis, such as segmentation or classification. Thus, the input training dataset in this case includes medical images for training and augmented medical images, such as genomic data, corresponding to the same subjects as the subjects of the medical images. Genetic or genomic information can be available during training, along with non-genetic data (such as medical images) that are easy to obtain, but during inference, only non-genetic data may be available.

[0039] In such multimodal genomics applications, existing approaches that use genetic information as augmented data during training learn commutative invariance through contrastive learning. Use cases arising in multimodal genomics applications: For multimodal genomics applications where both genetic information and medical images may only be used during training, the model may learn non-commutative invariance from genetic modality to imaging modality and discard that modality-specific information while leveraging the complementary semantics offered by the genetic data. The non-commutative nature of the invariance improves the efficiency of inference during real-world deployments that are typically performed only on image data without access to genetic information.

[0040] Optionally, when the method is for medical image segmentation or classification, the input medical images may be acquired in real time (e.g., from a live video stream or a live medical scan). In this way, image segmentation may be efficiently performed on-site so that adaptive radiation therapy or other treatment techniques may be immediately implemented. Similarly, in this way, image classification may be performed in real time.

[0041] Another aspect embodiment includes a data processing apparatus having a memory storing computer readable instructions and a processor, the processor (or controller circuitry) configured to execute the instructions to train a machine learning model for image analysis, perform a computer implemented method, and / or perform a computer implemented method of image analysis.

[0042] Other aspect embodiments include a computer program having instructions which, when executed by a computer, cause the computer to perform a computer-implemented method and / or a computer-implemented method of image analysis.

[0043] Other aspect embodiments include a non-transitory computer-readable storage medium having instructions that, when executed by a computer, cause the computer to perform a computer-implemented method and / or perform a computer-implemented method of image analysis.

[0044] The invention can be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. The invention may also be implemented as a computer program or computer program product, i.e. a computer program tangibly embodied in a non-transitory information carrier, for example in a machine-readable storage device or in a propagating signal, for execution by or to control the operation of one or more pieces of hardware.

[0045] A computer program may be in the form of a stand-alone program, a computer program portion, or more than one computer program, written in any type of programming language, including compiled or interpreted languages, which may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a data processing environment. A computer program may be deployed to be executed on one module or on multiple modules at one site or distributed across multiple sites and interconnected by a communication network.

[0046] The method steps of the present invention may be performed by one or more programmable processors executing a computer program to perform the functions of the present invention by manipulating input data and generating output. Apparatus of the present invention may be implemented as programmed hardware or as special purpose logic circuitry including, for example, a Field Programmable Gate Array (FPGA) or an Application-Specific Integrated Circuit (ASIC).

[0047] Processors suitable for the execution of a computer program include, by way of example, both general purpose and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor receives instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions coupled to one or more memory devices for storing instructions and data.

[0048] The present invention has been described with reference to specific embodiments. Other embodiments are within the scope of the following claims. For example, the steps of the invention can be performed in a different order and still achieve desirable results.

[0049] Elements of the invention are described using terms such as "processor" and "input device." Those skilled in the art will recognize that such functional terms, and their equivalents, may refer to parts of a system that are spatially separate but combine to perform the defined functions. Similarly, the same physical part of a system may provide two or more of the defined functions. For example, separately defined means may be implemented using the same memory and / or processor, where appropriate.

[0050] We show that learning modality-invariant representations is a good way to leverage data augmentations from different modalities, such as simulations.

[0051] In the context of water body segmentation, as mentioned above, the current state of the art for segmentation is achieved using the AquaNet model. However, as will become clear, we have demonstrated that CPNet (the reader is referred to the work of Yu, C. et al. (2020) Context Prior for Scene Segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)) performs better in water body segmentation with OoD scenarios. Furthermore, we have also demonstrated that the use of DPT (the reader is referred to the work of Ranftl, R. et al. (2021) Vision Transformers for Dense Prediction. ICCV, 2021), one of the most commonly used transformer models for segmentation, outperforms the aforementioned models for OoD generalization of water body segmentation by more than 12%. Therefore, in the following discussion, DPT will be considered as the baseline segmentation model and also used as an example to demonstrate the inefficiencies that currently exist.

[0052] Figure 4 represents the architecture of a DPT model applied in the context of water body segmentation. When the model is trained, the input image is converted into tokens after passing the input image through a patch encoder. This conversion is performed by extracting non-overlapping patches and then linearly projecting their flattened representations (or embeddings or projections) or by applying a ResNet-50 feature extractor. Patches in this context are represented as grid-like overlays on the input image. Image embeddings may be extended with positional embeddings to add patch-independent readout tokens. The tokens go through multiple transformer stages (T1, T2, T3, T4). The tokens are then reassembled into an image-like representation at multiple resolutions from different stages by reassembly operations (R1, R2, R3). The reassembly operations allow for assembling the tokens into feature maps at a fraction of the spatial resolution of the input image. Fusion modules (F1, F2, F3, F4) progressively fuse and upsample the representations to generate fine-grained predictions. The fusion module combines the features using a residual convolution unit and upsamples the feature maps. As shown in the figure, the result of the fine-grained prediction is a segmentation mask corresponding to the input image.

[0053] However, as depicted in Figure 5, the DPT model does not provide any built-in mechanism for combining real and synthetic training data. Therefore, the representations from the patch encoder are susceptible to modality gaps between different sets of training data, which can adversely affect inference and test-time performance. That is, images of a first modality (top, e.g., real data), when passed through the patch encoder (and possibly other layers of the DPT model), will produce a representation at one location in the latent or representation space. Images of a second modality (bottom, e.g., simulated data), when passed through the same patch encoder, will produce a representation at a different location in the latent space.

[0054] In the absence of pairwise training data, the best currently known method to combine data from multiple modalities is by learning a modality-invariant representation using a Gradient Reversal Layer (GRL). GRL forces the backbone encoder to be robust to modality shifts by enforcing commutative form invariance. In other words, GRL tries to fool the discriminator by flipping the modality labels, making the model invariant to, for example, both real and synthetic modality information.

[0055] As proven here, this commutativity is not necessary if data from one modality (e.g., simulated data) is used only as an augmentation and is never encountered during inference. In this case, the technique performs non-commutative modality invariance, where only simulated data is mapped into the space of real data, and not vice versa. This results in models that are significantly more parameter-efficient.

[0056] In summary, existing approaches for water body segmentation suffer from distribution shifts in real-world water images. Training models using simulated data that reproduce the distribution shifts could be a potential solution, but state-of-the-art models for water body segmentation lack a built-in mechanism to consume simulated data and may exhibit inefficiencies caused by the modality gap between real and simulated data.

[0057] As mentioned before, the techniques presented here are not limited to the field of flood management. When applying computer vision to healthcare, such as the automated analysis of fundus or x-ray images, it is possible to create training datasets that provide genetic information along with medical images. However, the data is not available for the patient. Since DNA sequencing is a time-consuming and expensive process, it is not realistic to expect genetic information to be available during real-time diagnosis (inference). It is more reasonable to train models only on genetic data and medical images. In this case, the genetic data serves as a source of augmentation (much like synthetic or simulated images in flood detection scenarios). Medical image segmentation or classification models can then be deployed to leverage the knowledge the models learned by looking at genetic information during training, while performing inference, possibly in real time, using easily available medical images. Thus, commutative invariance leads to parameter-inefficient models in multimodal genomics use cases.

[0058] Figure 6 shows the Non-Commutative Invariance (NCI) for water body segmentation. Fig. 1 is a schematic diagram of using NCI representations (or embeddings or projections) to efficiently exploit synthetic augmentation in the training data. When trained with NCI representations (or embeddings or projections), otherwise general segmentation models can better exploit training set augmentation (e.g., augmentation or supplementation with synthetic samples) than known existing methods such as combining multiple modalities. Thus, training with synthetic data is an effective approach to achieve OoD generalization in water body segmentation.

[0059] As illustrated, an NCI encoder (i.e., an encoder for an image segmentation machine learning model trained to enforce non-commutative invariance) processes images from different modalities. While known approaches generate representations that are sensitive to modality, approaches using an NCI encoder generate representations that are non-commutatively modality invariant. The representations are then processed by subsequent layers of a conventional network to generate water body segmentation masks.

[0060] FIG. 7 illustrates a general method for training a machine learning model for image segmentation, the method including training an encoder configured to invariantly encode data of a first modality and data of a second modality into the same representation space corresponding to the first modality.

[0061] In step 71, the computer receives an input training dataset that includes actual training data (i.e., data of a first modality) and augmented training data (i.e., data of a second modality, which in various contexts may be referred to as other data, manipulated data, simulated data, supplementary data, genomic data, etc.).

[0062] In step 72, the computer iteratively trains the encoder until the encoder converges. This process results in a trained non-commutative invariant encoder.

[0063] The training process 72 includes, at step 721, training a discriminator model. Training the discriminator model includes minimizing a discriminator loss function, using embeddings obtained from the encoder (after initialization, and of course before the final convergence of the encoder). The training process uses an input training data set. Once trained, the discriminator is configured to discriminate whether the input data is real data (data of the first modality) or augmented data (data of the second modality).

[0064] The training process also includes training the encoder by maximizing the discriminator error, step 722. The training process uses an input training data set and a trained discriminator (possibly not a final version of the trained discriminator; rather, an iterative process means that the discriminator weights can be iteratively updated after updating the encoder weights). Once trained, the encoder is configured to invariantly encode real and augmented data into the same real representation space.

[0065] As explained above, the output of early layers of machine learning networks that do not enforce modality invariance (patch encoding in the specific case of DPT, as described herein) is susceptible to modality shifts. Ideally, the early layers (transformer patch representations) covary with changes in semantics and not with changes in modality. For example, in the context of water body segmentation, the early layers of a water body segmentation model should covary with the presence or absence of flooding, but not with changes in the mode of image acquisition, changes in light conditions, changes in precipitation, etc. Making the representations of the early layers invariant to modality-specific information helps to integrate a diverse set of multimodal data sources for training machine learning models, such as deep neural networks, for segmentation, for example, using large amounts of easily accessible simulated data. This is particularly useful for transformer-based models, since performance is directly proportional to the amount of training data. Modality invariance allows the transformer to sample from a "modality-independent" representation space and encode only changes in semantics.

[0066] The technique presented here makes the representations in the early layers of a segmentation model invariant to the imaging modality (real / synthetic) from which the inputs originate. However, during evaluation or inference, the model only encounters images of the real modality, as the synthetic data is only used as a training-time augmentation. Thus, unlike existing approaches that enforce invariance, which achieve this goal by removing modality-specific information from all modalities, the technique presented here maps only synthetic images to the real representation space, without perturbing the representations of real images. This simplifies the learning objective and allows the model to leverage modality-specific information (e.g., texture) that corresponds to the real modality during inference, resulting in significant parameter efficiency improvements.

[0067] Figure 8 is a schematic diagram showing the training process for a segmentation model trained with NCI. The proposed invariance is applied to embeddings obtained after propagating input training images to the first few layers (initial encoder) of the neural network for segmentation. The input training images in this example are real and simulated training images. The training images are fed with the ground truth segmentation mask and also with an indicator regarding the modality (real or simulated).

[0068] The first step in enforcing non-commutative modality invariance is performed by the introduction of a discriminator network. The discriminator network can be adversarially trained to distinguish between real and simulated or synthetic modalities using fixed patch representations. That is, the encoder that computes the patch representations for processing by the discriminator is fixed, and the encoder weights are not updated by backpropagation from the discriminator loss. The discriminator loss for each sample is computed using the discriminator loss function as follows:

number

[0069] Once updated, the initial embedding layer is then trained to partially fool the modality discriminator (by removing modality-specific information). This involves the discriminator mapping synthetic images to a representation that the discriminator recognizes as real. However, the real images are also mapped to a representation space where they are still considered real. The discriminator is partially fooled, since it recognizes the real data as real, and the synthetic data as real. The invariance achieved by this algorithm is therefore non-commutative. The patch coding loss is computed using the discriminator error as follows:

number

[0070] For NCI training, the discriminator weights D and the patch encoder weights E can be updated by gradient descent using the following assignment:

number

[0071] Once trained, subsequent layers of the segmentation model can then be trained in a conventional manner using the NCI representations obtained by the trained NCI encoder.

[0072] Similar to water body segmentation, the NCI technique can be applied to use cases such as multimodal genomics image segmentation. The initial encoder computes L for data points of genetic modalities ��.E This maps samples containing only genetic information to representations or non-genetic (at inference time) modalities within the representation space.

[0073] FIG. 9 illustrates the training process for a segmentation or classification model with a genomics use case, trained at NCI. The reader will notice that the logic is very similar to the training process described above using real and simulated training data. However, the input training data in this example are real medical images (e.g., x-rays, MRI scans, etc.) and corresponding genomic data (i.e., genomic data for the same subject as the subject of the medical images). The training data is fed with ground truth segmentation masks or classifications, and also fed with an indicator (genomic data or medical images) for modality. In this context, a discriminator network can be adversarially trained to distinguish between imaging modalities and genetic modalities. In the case of a segmentation model, the trained model outputs segments or masks for the input medical images. In this case, segmentation is performed with the additional context provided by the genomics training data. In the case of a classification model, the trained model outputs classifications for the input medical images. In this case, classification is performed with the additional context provided by the genomics training data.

[0074] 10 illustrates a general method for use of a trained machine learning model for image analysis, the trained machine learning model includes a trained encoder configured to invariantly encode data of a first modality and data of a second modality into the same representation space corresponding to the first modality.

[0075] In step 101, a computer receives an input image. In step 102, the computer processes the input image using a trained machine learning model to obtain analysis data for the input image.

[0076] The trained encoder of the trained machine learning model is iteratively trained using an input training dataset including the actual training data and the augmented training data. The iterative training includes training a discriminator model by minimizing a discriminator loss function using the input training dataset to obtain a trained discriminator, the trained discriminator configured to discriminate whether the input data is the actual data or the augmented data. The iterative training also includes training an encoder by maximizing a discriminator error using the training dataset and the trained discriminator to obtain a trained encoder, the trained encoder configured to invariantly encode the actual data and the augmented data into the same representation space.

[0077] The trained machine learning model also includes additional trained downstream layers, which are trained, for example, by a multi-objective optimization procedure using the training dataset and using the trained encoder. As mentioned above, the additional trained downstream layers are trained in a conventional manner using the NCI representations obtained by the trained NCI encoder.

[0078] Figure 11 is a schematic diagram showing the inference (or testing) process using a segmentation model trained with NCI in the context of water body segmentation. Inference is performed similarly to conventional approaches, except for the replacement of the initial code layer by the NCI layer. The test set contains only real, unsimulated images of water. The trained segmentation model outputs mask or segment predictions that indicate that water is predicted to be present in the images of the test set.

[0079] Figure 12 is a schematic diagram illustrating the inference (or testing) process using an NCI trained image analysis model in the context of medical image classification. The test set contains only medical image data. However, the trained NCI encoder was trained on both medical image data and genomics data, as described above. In this way, the encoder learns non-commutative invariance from genetic modality to imaging modality. The trained classification model outputs a prediction of the presence of a genetic condition inferred from the test set images.

[0080] Figure 13 shows an example of implementing NCI by non-commutatively invariantly training a DPT transformer model for water body segmentation, where the training data is again real image data (labeled “0”) and simulated image data (labeled “1”).

[0081] Figure 13a represents the first step in the training process. The computer trains a discriminator CNN on the patch representations from the DPT. The patch representations may be passed through additional layers, and the results may be concatenated before input to the discriminator. This training may be performed in an adversarial manner so that the discriminator can classify between real and simulated modalities. This training involves minimizing the discriminator loss function value. More specifically, the training involves minimizing the cross-entropy loss during adjustment of the discriminator weights.

[0082] Figure 13b represents the second step in the training process. The computer trains the DPT patch encoder to maximize the discriminator error by mapping all representations to the "real" modalities. Thus, the patch encoder is made invariant to modality information, albeit non-commutatively. That is, the simulated images are mapped to the real modalities in the representation space, while the modality information in the real images is preserved. Training involves maximizing the discriminator error by maximizing the cross-entropy loss while tuning the encoder weights.

[0083] Figure 13c represents the third step in the training process. The non-commutatively invariant (NCI) patch representations are then used to train downstream transformer layers of the DPT. The discriminator has no role in this aspect of the training process. In effect, the trained patch encoder is "frozen" (i.e., its weights and biases are frozen) and used in the full network training process. As shown, the effect of the trained NCI patch encoder is to encode images from either modality into a shared representation space in a modality-invariant manner.

[0084] Once trained, inference can be performed as usual, using the NCI patch encoder instead of the vanilla patch encoder.

[0085] Figure 14 shows an example of the operation of NCI used for medical image classification. The training data in this case are medical images (specifically, fundus images acquired by OCT or the like) and genomics data.

[0086] In the case of predicting diabetic retinopathy (a complication of diabetes that causes damage to the back of the eye), numerous genetic and non-genetic factors are known to be determinants. To train a model to predict or classify diabetic retinopathy, various genetic data sources and acquisition modalities, such as Single Nucleotide Polymorphisms (SNPs) and Polygenic Risk Score (PRS), are used along with non-genetic fundus images. However, in practice, when deploying such prediction or classification models in the real world, genetic data sources cannot usually be used for inference. Thus, by mapping SNP and / or PRS modalities to imaging modalities, non-commutative invariance (NCI) can be learned. This allows the final prediction or classification model to mine semantically related knowledge from both genetic and imaging modalities while discarding genetics-specific information that is not encountered at testing or inference time. During inference, only fundus images are provided to the diabetic retinopathy prediction model.

[0087] As shown, the input training dataset in this example includes fundus images labeled to indicate when a subject is known to have diabetic retinopathy. The training dataset further includes genomics data for the same subjects in the form of SNP and PRS data. Training of the encoder (here represented by f) may be performed in a manner equivalent to the training methods described above. Once trained, the trained encoder may be implemented into a classification or image recognition model, such as an Inception-3 deep CNN. Although incorporating the trained encoder, this model may be trained according to conventional techniques and may then be configured to accept fundus images from new, unknown subjects and output an analysis result in the form of a classification or probability (γ) of diabetic retinopathy.

[0088] Table 1 below provides an overview of the quantitative performance of NCI when used with the DPT model for water body segmentation. Performance is quantified using the Intersection over Union (IoU) metric, which quantifies the degree of overlap between the predicted water segment (or segments) and the ground truth segment. [Table 1]

[0089] All hyperparameters and experimental settings are also based on the implementation of Domain Adversarial Neural Networks Ganin & Lempitsky (2015) in DomainBed (Gulrajani & Lopez-Paz, 2021), which provides a fair basis for comparison with existing techniques. In short, the following hyperparameters were found to be suitable for the model trained according to the technique (NCI-DPT): - Optimizer: Adam - Learning rate: 0.001 - Scheduler: Exponential (gamma set to 0.99) - Weight decay: 0.0001 - Architecture-specific choices such as activation functions, number of hidden units, layers, dropout rates, etc. We used standard DPT with ResNet50 as the patch encoder. - Pretrained: ResNet50 patches pretrained on ADE20K DPT using the encoder. We fine-tuned a pre-trained model provided by the authors of DPT. - Training and testing split: The total number of real samples was about 4,000, and the number of augmented simulated samples varied from about 4,000 (small) to 12,000 (large). The test sets were fixed: the in-distribution test set had about 300 examples, and the out-of-distribution test set had about 600 examples.

[0090] As shown, among all prior art models (AquaNet, CPNet, DPT, Finetuned-DPT), the DPT-based model performs the most accurate task of image segmentation on training / test datasets containing real water images in image analysis. This finding holds true for test dataset 1, which consists only of in-distribution images (i.e., all water images share similar patterns with the training data and are all acquired under comparable lighting conditions), and training dataset 2, which consists of OoD images (i.e., water images contain previously unseen features such as novel lighting conditions). For this reason, the DPT-based architecture is selected as a suitable candidate benchmark for evaluating the effectiveness of NCI in image segmentation.

[0091] The benchmark in this example involves the preparation of small and large mixed real and synthetic datasets. The small training dataset used in the example consists of about 4,000 water images (including about 2,000 real and 2,000 synthetic images), and the large training dataset used in the example consists of 12,000 water images (including about 2,000 real and 10,000 synthetic images). In line with the general consensus that the performance of deep neural networks (such as DPT-based models) is known to scale with the size of the training dataset, DPT models trained on larger benchmark mixed datasets provide higher accuracy when testing on both in-distribution and OoD data. The DPT benchmark trained on more synthetic images shows accuracy improvements of 0.06 and 0.12, respectively, compared to the DPT benchmark trained on fewer synthetic images on both Test Dataset 1 and Test Dataset 2.

[0092] The NCI-DPT model (i.e., a DPT model incorporating an NCI encoder trained according to the techniques disclosed herein) shows improved accuracy compared to both benchmark models for the OoD test data. The number of real images in the mixed training dataset, which contains both real and synthetic images, contains half the number of synthetic images (i.e., real images:synthetic images=1:2), with approximately 2,000 real images and approximately 4,000 synthetic images. Note that the performance of In-Dist segmentation accuracy is slightly degraded compared to the benchmark DPT model. The inventors speculate that this is due to the imbalance of real and synthetic images in the dataset.

[0093] Oversampling can be used to balance the training dataset. That is, datasets (both small and large) that contain real and synthetic data may contain a disproportionate number of real images compared to synthetic images. For example, oversampling may be used to replicate the number of real images to match the number of real and synthetic images in the mixed training dataset. This is advantageous to avoid the model disproportionately overfitting the synthetic dataset. The NCI-DPT model trained using more synthetic images in a balanced manner shows accuracy improvements of 0.05 and 0.60, respectively, compared to the NCI-DPT model trained using fewer synthetic images on both Test Dataset 1 and Test Dataset 2.

[0094] In particular, the accuracy of the NCI-DPT model (irrespective of the size of the training dataset) exceeds the accuracy of all known models in correct segmentation of OoD test images.

[0095] Figure 15 provides a qualitative demonstration of NCI when used in a DPT model for water body segmentation. The left column contains three figures from the ATLANTIS dataset (the reader is referred to the paper Erfani, S. et al., ATLANTIS: A benchmark for semantic segmentation of waterbody images. Environmental Modelling & Software, 105333). (Note that the original images contain color information.) As shown, qualitatively, a DPT model incorporating a trained NCI encoder trained according to the techniques disclosed herein is suitable for image segmentation of urban flood images (Panel A), images with reflective water (Panel B), and images with turbid water (Panel C).

[0096] In the context of water images, the image segmentation data may be processed for comparison with a threshold value. If the threshold value is exceeded, the computation means outputs an alarm indicating the presence of excessive water. The alarm may be a signal for transmission to the appropriate authorities. As a simple example, real-time images obtained from a video surveillance camera may be sent to the computation means and processed using the NCI-DPT model. A predefined threshold may be set to indicate that a safe level of water present in the image corresponds to 10% of the field of view. If the image segment generated using the NCI-DPT model exceeds 10% of the field of view or image frame, the computation means may be configured to output an audible alarm or to transmit a signal to emergency services personnel.

[0097] In summary, the techniques presented here demonstrate that training with synthetic data is a valid approach to achieve OoD generalization in use cases such as water body segmentation and medical image classification. The NCI approach, which efficiently combines synthetic and real data, delivers state-of-the-art OoD water body segmentation performance. Furthermore, the techniques presented here provide a framework for leveraging NCI to improve performance in multimodal genomics use cases where genetic information is only available during training and not during inference.

[0098] 16 is a block diagram of a computing device, such as a data processing server, that may be used to embody the present invention and to implement the embodiment method of training a machine learning model for image analysis and / or the embodiment method of augmented analysis. The computing device includes a processor 993 and a memory 994. Optionally, the computing device also includes a network interface 997 for communication with other computing devices, such as other computing devices of the embodiments of the present invention.

[0099] For example, an embodiment may consist of a network of such computing devices. Optionally, the network of computing devices also includes one or more input mechanisms 996, such as a keyboard and mouse, and one or more display units 995, such as a monitor. The components may be connected to each other via a bus 992.

[0100] The memory 994 may include a computer-readable medium, which term may refer to a single medium or multiple media (e.g., centralized or distributed databases and / or associated caches and servers) configured to carry computer-executable instructions or store data structures. Computer-executable instructions may include, for example, instructions and data accessible to a general purpose computer, a special purpose computer, or a special purpose processing device (e.g., one or more processors) that cause such a computer to perform one or more functions or operations. Thus, the term "computer-readable storage medium" may also include any medium capable of storing, encoding, or carrying a set of instructions for execution by a machine, causing the machine to perform any one or more of the method steps of the present disclosure. The term "computer-readable storage medium" may accordingly be interpreted to include, but is not limited to, solid-state memory, optical media, and magnetic media. By way of example, and not limitation, such computer-readable media may include non-transitory computer-readable storage media including random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disk read-only memory (CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory devices (e.g., solid-state memory devices).

[0101] The processor 993 is configured to control the computing device and execute processing operations, such as computer program code stored in the memory 994, to perform various different functions of the machine learning models and components described herein. The memory 994 stores data that is read and written by the processor 993. As referred to herein, a processor may include one or more general-purpose processing devices, such as a microprocessor, a central processing unit, and the like. A processor may include a multiple instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor that implements other instruction sets or a combination of instruction sets. A processor may also include one or more special-purpose processing devices, such as an ASIC, an FPGA, a digital signal processor (DSP), a network processor, and the like. In one or more embodiments, the processor is configured to execute instructions to perform the operations and operations discussed herein.

[0102] The display unit 995 may display representations of data stored by the computing device, and may also display cursors and dialog boxes and screens that allow interaction between a user and programs and data stored on the computing device. The input mechanism 996 may allow a user to input data and instructions into the computing device.

[0103] A network interface (network I / F) 997 may be connected to a network such as the Internet, and can be connected to other such computing devices via the network. The network I / F 997 may control data input from other devices or data output to other devices via the network. Other peripheral devices such as a microphone, a speaker, a printer, a power supply unit, a fan, a case, a scanner, a trackball, etc. may be included in the computing device.

[0104] The encoder training module may consist of processing instructions stored in a portion of memory 994, a processor 993 that executes the processing instructions, and a portion of memory 994 that stores encoder weights, biases, and other information related to the encoder architecture during execution of the processing instructions. The final trained weights and biases of the encoder may be stored in memory 994 and / or a connected storage unit and may be transmitted, forwarded, or otherwise communicated to other components.

[0105] The discriminator training module may consist of processing instructions stored in a portion of memory 994, a processor 993 that executes the processing instructions, and a portion of memory 994 that stores discriminator weights, biases, and other information related to the discriminator architecture during execution of the processing instructions. The final trained weights and biases of the discriminator may be stored in memory 994 and / or a connected storage unit and may be transmitted, forwarded, or otherwise communicated to other components.

[0106] The image analysis model training module may consist of processing instructions stored in a portion of memory 994, a processor 993 that executes the processing instructions, and a portion of memory 994 that stores weights, biases, and other information related to the architecture of the image analysis machine learning model during execution of the processing instructions. The final trained weights and biases of the model may be stored in memory 994 and / or a connected storage unit and may be transmitted, transferred, or otherwise communicated to other components.

[0107] Methods embodying the invention may be performed on a computing device such as that depicted in Figure 16. Such a computing device need not have all of the components depicted in Figure 16, but may consist of some of these components. For example, methods embodying the invention may be performed by a single computing device in communication with one or more data storage servers over a network. The computing device may be the data storage itself, storing trained model weights and / or biases, training data sets, model hyperparameters, test images, segmentation and / or classification outputs, etc.

[0108] Methods embodying the invention may be performed by multiple computing devices in cooperation with one another, where one or more of the multiple computing devices may be data storage servers that store at least a portion of the trained model weights and / or biases, training data sets, model hyper-parameters, test images, segmentation and / or classification outputs, etc.

[0109] The various methods described above may be implemented by a computer program. The computer program may include computer code (e.g., instructions) arranged to instruct a computer to perform one or more functions of the various methods described above. For example, the steps of the methods described with respect to FIG. 7 and / or FIG. 10 may be performed by the computer code. The steps of the methods described above may be performed in any suitable order. The computer program and / or code for performing such methods may be provided to an apparatus, such as a computer, on one or more computer readable media, or more generally, on a computer program product. The computer readable medium may be transitory or non-transitory. The one or more computer readable media may be, for example, an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system, or a propagation medium for data transmission, for example, downloading code via the Internet. Alternatively, the one or more computer-readable media may take the form of one or more physical computer-readable media, such as a semiconductor or solid state memory, magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), a rigid magnetic disk, and an optical disk, such as a CD-ROM, CD-R / W or DVD. The instructions may also reside, completely or at least partially, within memory 913 and / or within controller circuitry 911 while being executed by computing system 910, with memory 913 and controller circuitry 911 also constituting computer-readable storage media.

[0110] In implementation, the modules, components, and other features described herein may be implemented as discrete components or may be integrated into the functionality of hardware components such as an ASIC, FPGA, DSP, or similar device.

[0111] A "hardware component" is a tangible (e.g., non-transient) physical component (e.g., a set of one or more processors) capable of performing specific operations and may be configured or arranged in a specific physical manner. A hardware component may include dedicated circuitry or logic that is permanently configured to perform specific operations. A hardware component may have a dedicated processor such as an FPGA or ASIC. A hardware component may also include programmable logic or circuitry that is temporarily configured by software to perform specific operations.

[0112] Additionally, the modules and components may be implemented as firmware or functional circuitry within a hardware device, or may be implemented in any combination of hardware devices and software components, or solely in software (e.g., code stored or otherwise embodied in a machine-readable medium or a transmission medium).

[0113] Unless specifically stated otherwise, as will be apparent from the discussion that follows, discussions throughout this specification using terms such as "receive," "determine," "compare," "enable," "hold," "identify," "obtain," "access," and the like refer to operations and processes of a computer system or similar electronic computing device that manipulate and transform data represented as physical (electronic) quantities in the computer system's registers and memory into other data similarly represented as physical quantities in the computer system's memory or registers, or other such information storage, transmission, or display device.

[0114] Although specific embodiments have been described, these embodiments are presented by way of example only and are not intended to limit the scope of the invention. Indeed, the novel methods and apparatus described herein may be embodied in a variety of other forms, and various omissions, substitutions and changes in the forms of the methods and apparatus described herein may be made.

[0115] The following remarks are useful in understanding the present invention.

[0116] (Appendix 1) 1. A computer-implemented method for training a machine learning model for image analysis, comprising: receiving an input training dataset including actual training data corresponding to an actual modality and augmented training data; and iteratively training an encoder of the machine learning model to obtain a trained encoder; The repeated training includes: training a discriminator model by minimizing a discriminator loss function using the input training data set to obtain a trained discriminator, the trained discriminator being configured to discriminate whether input data is real data or augmented data; training the encoder by maximizing a discriminator error using the input training data set and the trained discriminator to obtain the trained encoder, the trained encoder being configured to invariantly encode real data and augmented data into a real representation space such that the representation preserves information about the real modality; having method.

[0117] (Appendix 2) The machine learning model further comprises an additional downstream layer, the additional downstream layer being configured by training together with the trained encoder to output an analysis result of the input data. The method described in Appendix 1.

[0118] (Appendix 3) and iteratively training the additional downstream layers by a multi-objective optimization procedure using the input training dataset and the trained encoder to obtain trained additional downstream layers. The method described in Appendix 2.

[0119] (Appendix 4) The additional downstream layers include a transformer encoder, a reassembly module, and a fusion module. 4. The method according to claim 2 or 3.

[0120] (Appendix 5) Receiving an input image; and processing the input image using a trained machine learning model to obtain an analysis result of the input image. 5. A method according to any one of claims 1 to 4.

[0121] (Appendix 6) 1. A computer-implemented method of image analysis comprising the steps of: Receiving an input image; processing the input image using a trained machine learning model to obtain an analysis result of the input image; The trained machine learning model is a trained encoder that is iteratively trained using an input training dataset that includes real training data corresponding to a real modality and augmented training data; and a trained additional downstream layer trained by a multi-objective optimization procedure using the input training dataset and the trained encoder; The repeated training includes: training a discriminator model by minimizing a discriminator loss function using the input training data set to obtain a trained discriminator, the trained discriminator being configured to discriminate whether input data is real data or augmented data; training the encoder by maximizing a discriminator error using the input training data set and the trained discriminator to obtain the trained encoder, the trained encoder being configured to invariantly encode real data and augmented data into a real representation space such that the representation preserves information about the real modality; having method.

[0122] (Appendix 7) the machine learning model is for segmentation of water images, and the input training dataset includes real training water images and simulated training water images; 7. A method according to any one of claims 1 to 6.

[0123] (Appendix 8) receiving an input water image in real time; and processing the input water image with a trained machine learning model to obtain a water body segmentation result for the input water image. The method described in Appendix 7.

[0124] (Appendix 9) and outputting an alarm in response to determining that the water body segmentation result exceeds a threshold. The method described in Appendix 8.

[0125] (Appendix 10) the machine learning model is for segmentation or classification of medical images, and the input training data set includes training medical images and corresponding genomic training data; 7. A method according to any one of claims 1 to 6.

[0126] (Appendix 11) Receiving input medical images in real time; and processing the input medical image using a trained machine learning model to obtain a medical segmentation or classification result for the input medical image. 11. The method according to claim 10.

[0127] (Appendix 12) and outputting an alarm in response to determining that the medical segmentation or classification result exceeds a threshold. 12. The method according to claim 11.

[0128] (Appendix 13) 13. A data processing apparatus comprising a memory storing computer-executable instructions for performing a method according to any one of claims 1 to 12, and a processor configured to execute said instructions.

[0129] (Appendix 14) A computer program comprising instructions, The computer program, when executed by a computer, causes the computer to perform a method according to any one of claims 1 to 12. Computer program.

[0130] (Appendix 15) A non-transitory computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform a method according to any one of claims 1 to 12. [Explanation of symbols]

[0131] 992 Bus 993 Processor 994 Memory 995 Display Unit 996 Input Mechanism 997 Network Interface

Claims

1. 1. A computer-implemented method for training a machine learning model for image analysis, comprising: receiving an input training dataset including actual training data corresponding to an actual modality and augmented training data; and iteratively training an encoder of the machine learning model to obtain a trained encoder; The repeated training includes: training a discriminator model by minimizing a discriminator loss function using the input training data set to obtain a trained discriminator, the trained discriminator being configured to discriminate whether input data is real data or augmented data; training the encoder by maximizing a discriminator error using the input training data set and the trained discriminator to obtain the trained encoder, the trained encoder being configured to invariantly encode real data and augmented data into a real representation space such that the representation preserves information about the real modality; having method.

2. The machine learning model further comprises an additional downstream layer, the additional downstream layer being configured by training together with the trained encoder to output an analysis result of the input data. The method of claim 1.

3. and iteratively training the additional downstream layers by a multi-objective optimization procedure using the input training dataset and the trained encoder to obtain trained additional downstream layers. The method of claim 2.

4. The additional downstream layers include a transformer encoder, a reassembly module, and a fusion module. The method according to claim 2 or 3.

5. Receiving an input image; and processing the input image using a trained machine learning model to obtain an analysis result of the input image. The method of claim 1.

6. the machine learning model is for segmentation of water images, and the input training dataset includes real training water images and simulated training water images; The method of claim 1.

7. receiving an input water image in real time; and processing the input water image with a trained machine learning model to obtain a water body segmentation result for the input water image. The method according to claim 6.

8. and outputting an alarm in response to determining that the water body segmentation result exceeds a threshold. The method according to claim 7.

9. the machine learning model is for segmentation or classification of medical images, and the input training data set includes training medical images and corresponding genomic training data; The method of claim 1.

10. Receiving input medical images in real time; and processing the input medical image using a trained machine learning model to obtain a medical segmentation or classification result for the input medical image.

10. The method of claim 9.

11. and outputting an alarm in response to determining that the medical segmentation or classification result exceeds a threshold. The method of claim 10.

12. 1. A computer-implemented method of image analysis comprising the steps of: Receiving an input image; processing the input image using a trained machine learning model to obtain an analysis result of the input image; The trained machine learning model is a trained encoder that is iteratively trained using an input training dataset that includes real training data corresponding to a real modality and augmented training data; and a trained additional downstream layer trained by a multi-objective optimization procedure using the input training dataset and the trained encoder; The repeated training includes: training a discriminator model by minimizing a discriminator loss function using the input training data set to obtain a trained discriminator, the trained discriminator being configured to discriminate whether input data is real data or augmented data; training the encoder by maximizing a discriminator error using the input training data set and the trained discriminator to obtain the trained encoder, the trained encoder being configured to invariantly encode real data and augmented data into a real representation space such that the representation preserves information about the real modality; having method.

13. a memory storing computer executable instructions; receiving an input training dataset including actual training data corresponding to an actual modality and augmented training data; and a processor configured to execute the computer-executable instructions to: iteratively train an encoder of a machine learning model to obtain a trained encoder; The repeated training includes: training a discriminator model by minimizing a discriminator loss function using the input training data set to obtain a trained discriminator, the trained discriminator being configured to discriminate whether input data is real data or augmented data; training the encoder by maximizing a discriminator error using the input training data set and the trained discriminator to obtain the trained encoder, the trained encoder being configured to invariantly encode real data and augmented data into a real representation space such that the representation preserves information about the real modality; having Data processing device.

14. When executed by a computer, the computer receiving an input training dataset including actual training data corresponding to an actual modality and augmented training data; iteratively training the encoder of the machine learning model to obtain a trained encoder; The repeated training includes: training a discriminator model by minimizing a discriminator loss function using the input training data set to obtain a trained discriminator, the trained discriminator being configured to discriminate whether input data is real data or augmented data; training the encoder by maximizing a discriminator error using the input training data set and the trained discriminator to obtain the trained encoder, the trained encoder being configured to invariantly encode real data and augmented data into a real representation space such that the representation preserves information about the real modality; having A computer program containing instructions.

15. When executed by a computer, the computer receiving an input training dataset including actual training data corresponding to an actual modality and augmented training data; iteratively training the encoder of the machine learning model to obtain a trained encoder; The repeated training includes: training a discriminator model by minimizing a discriminator loss function using the input training data set to obtain a trained discriminator, the trained discriminator being configured to discriminate whether input data is real data or augmented data; training the encoder by maximizing a discriminator error using the input training data set and the trained discriminator to obtain the trained encoder, the trained encoder being configured to invariantly encode real data and augmented data into a real representation space such that the representation preserves information about the real modality; having A non-transitory computer-readable storage medium that stores instructions.

Images