Image data processing based on machine learning base model

Abstract

The domain-specific machine learning model (231) is used to generate contextual information for the generic machine learning base model (241). The output (292) thereof may in turn be used to retrain the domain-specific machine learning model (231).

Description

Technical Field

[0001] Various examples disclosed relate to techniques for processing image data. Various examples relate to techniques for predicting image features, for example, particularly the location of a particular structure in an image. Various examples relate to techniques for generating training data for a machine learning model based on image data processing in order to obtain such predictions. Background Art

[0002] Automatic processing of image data plays an important role in microscopy for various applications (fields). For example, it can help the user to position the sample in the field of view of the microscope. For this purpose, the overview image is evaluated by an algorithm. Data evaluation can also be performed near the recording. Other aspects involve instance segmentation of cells, for example for determining the degree of confluence or for automatic cell counting. Machine learning methods, in particular machine learning / deep learning models such as deep neural networks, are used in all these fields.

[0003] Training such models requires annotated training data, i.e. input image data and associated ground truth values ​​(annotations). The machine learning model then learns to generate outputs for new and unseen images that correspond to the annotations seen in the training data (inference phase).

[0004] In this case, generating annotations is a largely manual process and is often a bottleneck when creating new models. Given the multitude of domains in which machine learning models are used, this effort increases with each new task to be solved. Summary of the invention

[0005] The following describes how to enable a first machine learning model with lower complexity to provide contextual information to a second machine learning model with higher complexity.

[0006] Since it has been trained to a certain extent for domain-specific applications, the first machine learning model (hereinafter referred to as the first model) can be called a domain-specific model or a dedicated model. Due to the limited complexity of the first model, it is impossible or only possible to use the domain-specific model to perform inference in another domain to a certain extent. The first machine learning model or domain-specific model will be referred to as the first model hereinafter.

[0007] The second machine learning model (hereinafter referred to as the second model) can be called a base model or basic model, which is general enough to solve tasks in different fields. This is achieved by the relatively high complexity of the base model. Hereinafter, this base model is referred to as the second model.

[0008] The technology disclosed in this article can be applied to the processing of different image data. For example, microscope image data can be processed. However, volume image data can also be processed. Medical image data or biological image data can be processed. For example, a Z stack of multiple images can be processed. The image data can be a microscopic image or a macroscopic (overview) image. There is no restriction on the recording system (wide field, LSM, lattice light sheet, etc.), contrast type (bright field, DIC, fluorescent staining), etc. Magnetic resonance images or computer tomography image data can be processed.

[0009] In this process, the first model is first applied to the image data to process the image data, which gives a first prediction of the image features in the image data.

[0010] The first prediction may, for example, relate to the location of the structure in the image data. Localization information may be provided. However, other predictions are also possible. For example, the number of structures may be counted (e.g., cell count). The localization information may, for example, be provided in the form of a center point marking of the corresponding structure, a center of gravity marking of the corresponding structure, by a bounding box, or as a segmentation.

[0011] As an alternative or in addition to the localization of structures in the image data, the first prediction and / or the second prediction may include a classification of one or more structures in the image data.

[0012] Global parameters (eg, cell confluency or whether or not it is a tumor) can be determined.

[0013] Context information for the second model is then determined based on the first prediction. The context information configures the second model; thus the context information may also be referred to as a hint or configuration command.

[0014] In one example, the first prediction of the first model is directly used as context information. The context information can also be derived from the first prediction, for example by determining the encoding and / or by modifying the first prediction. User input can be considered here. The user can interactively intervene in the method by correcting or refining the context information.

[0015] By doing so, a portion of the first prediction can be selectively presented to the user via the user output so that the user modifies this portion of the first prediction. This portion of the first prediction can be determined by an active learning process. For example, a certain portion of the first prediction can be selected based on confidence or a trade-off between exploration and use in the input data space. This makes it possible to achieve a particularly steep learning curve.

[0016] Based on the contextual information, the second model can then provide a second prediction for the image feature by means of a second machine learning model that processes the image data.

[0017] In this case, the second prediction may correspond to the first prediction. For example, if the first prediction comprises a first localization of structures in the image data, the second prediction also comprises a second localization of these structures in its image data. Thus, this means that the first prediction and the second prediction solve the same task (e.g. localization of a specific structure, such as a cell). However, the first prediction may differ from the second prediction in terms of quantity and / or quality.

[0018] The second prediction may be more accurate, more complete, have a higher resolution, or be more comprehensive than the first prediction. For example, the first prediction may locate a particular structure (e.g., a cell) in the image data only incompletely and / or with relatively low accuracy (false positives or false negatives) and / or with low resolution (e.g., only center point localization). The second prediction may then locate all cells with high accuracy and resolution (e.g., using a segmentation mask for on-the-fly segmentation).

[0019] Thus, in general, the second model makes it possible to improve the quality of the predictions. This can then be exploited in various ways. For example, training data can be generated; the training data contains ground truth values ​​that can correspond to the second predictions or be determined based on the second predictions. Other applications of the second predictions include, for example, determining model confidence or model selection.

[0020] The features listed above and the features described below can be used not only in the respectively explicitly listed combination but also in other combinations or alone, without departing from the scope of protection of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A data processing pipeline for processing image data using two sequential models (a first model and a second model) according to various examples is schematically illustrated.

[0022] Figure 2 A data processing device according to various examples is schematically illustrated.

[0023] Figure 3 is a flow chart of an exemplary method.

[0024] Figure 4 Deployment scenarios of the basic model according to various examples are schematically illustrated. DETAILED DESCRIPTION

[0025] The above-mentioned nature, features and advantages of the present invention and the manner of achieving them will become more apparent and more clearly understood in conjunction with the following description of more detailed explanation of exemplary embodiments in conjunction with the accompanying drawings.

[0026] The present invention will be described in more detail below with reference to the accompanying drawings based on preferred embodiments. In the drawings, the same reference numerals represent the same or similar elements. These figures are schematic representations of various embodiments of the present invention. The elements shown in the figures are not necessarily shown in true proportion. On the contrary, the various elements shown in the figures are presented in a manner that those skilled in the art can understand their functions and general uses. The connection and coupling between the functional units shown in the figures and the elements can also be implemented as indirect connections or couplings. The connection or coupling can be implemented in a wired or wireless manner. The functional units can be implemented as hardware, software, or a combination of hardware and software.

[0027] A description of a technique for processing image data, particularly microscope image data, is given below. The processing is performed by a machine learning model. Specifically, an artificial deep neural network may be used. Convolutional layers may be used.

[0028] The convolutional layers in a convolutional neural network (CNN) slide over an input image or input map using one or more filters. These filters, often called "kernels," are small matrices that are able to recognize certain features or patterns in an image. At each step, the filters are point-by-point multiplied by a local region of the image and summed to create a value in the output map. This process is repeated for the entire image, creating a "feature map." Convolutional layers allow CNNs to learn the spatial hierarchy of features in the data. The kernels and concatenation between layers, as well as the number of layers, are parameters; the more parameters a CNN (or more generally a model) has, the more complex it is.

[0029] For example, in various examples described herein, a CNN with a U-net architecture can be used to process image data. Such a CNN with a U-net architecture allows, in particular, domain-specific processing of image data. The architecture of U-net is described in detail in the following paper: Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. "U-net: Convolutional networks for biomedical image segmentation." Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18. Springer International Publishing, 2015. In short, U-net is an architecture for segmenting image data, characterized by its characteristic "U" shape. It consists of a contracting path (encoder) that extracts information from the input image data and an expanding path (decoder) that uses this information to create a detailed segmentation map. While information is lost in the contracting path due to the max pooling layers, the dilating path uses upsampling operations. Skip connections between corresponding layers of the encoder and decoder ensure that spatial information is preserved.

[0030] Another architecture that can be used is the transformer architecture. The transformer network is based on the self-attention mechanism, which correlates the input sequence to weight the contextual information. The main components are multi-head self-attention and feed-forward neurons. In self-attention, the weight of each element in the sequence is calculated based on how closely it is linked to other elements. Multi-head attention allows the network to record different types of relationships simultaneously. Finally, the weighted output passes through a feed-forward layer and is then forwarded to the next block in the network.

[0031] In various examples, two machine learning models have the property of being connected in sequence. In this case, the image data is first processed using a first machine learning model (hereinafter referred to as the first model). This gives a first prediction of image features in the image data. Then, the image data is also processed in a second machine learning model (hereinafter referred to as the second model), and the second model generates a second prediction for the image features.

[0032] The first and second models are concatenated using context information. The context information of the second model is determined based on the first prediction of the first model.

[0033] Context information is used here to control specific types of information processing or response generation in the second model. While the input is the primary data element (e.g., an image) that is analyzed or interpreted by the second model, the context information provides guidance on how the second model should consider or respond to the input. For example, the input could be an image of an animal and the context information could be: "Determine the color of the animal." This context information instructs the second model to focus on a specific aspect of the image, namely the color of the animal, rather than just providing a general description or category of the image.

[0034] In this way, the context information can specify the domain in which the second model is to operate.

[0035] In order to enable the second model to operate substantially in different fields, the second model has a high complexity. The second model may have a higher complexity than the first model. For example, the number of parameters in the second model may be significantly greater than the number of parameters in the first model. For example, the number of parameters in the second model may be at least E3 times greater or at least E6 times greater or more than the number of parameters in the first model.

[0036] Since the first model has a small number of parameters, its generalization ability is relatively poor, that is, it is difficult to use it to solve other tasks besides the specific task it was trained for.

[0037] The first model may be a domain-specific model. This means that the first model may provide a first prediction for a specific application case. For example, the specific application case may be: locating cells in a microscope image acquired by phase contrast microscopy. Using the context information, the second model may be configured to provide a second prediction in the domain of the first model.

[0038] The second model may be a generic model. This means that the second model is not domain-specific. Due to its high complexity, the second model can be used in a variety of application cases. The second model may in particular be a basic model.

[0039] The base model is trained on a large amount of data and forms the basis of more specific models for specific application areas. The base model is general enough to handle a wide variety of tasks. The base model typically has a transformer architecture. The Transformer architecture can have an encoder and a decoder. The hidden layer has one or more "self-attention layers" and one or more feed-forward layers. The number of layers and the size of each layer (i.e. the number of "neurons" or nodes in each layer) are model parameters that can be adjusted during training. The base model is typically task-independent, or at least can be applied to a large number of different tasks. In addition to the input, the base model is provided with contextual information describing the task to be solved for this purpose. For example, the base model "SAM" described in Kirillov, Alexander, et al. "Segment anything." ArXiv Preprint arXiv:2304.02643 (2023) requires point annotations, which (i) show the SAM base model which objects in the image should be segmented, and (ii) correct potential incorrect segmentation areas. The base model is not particularly domain-specific. The base model is particularly applied to segmentation tasks. See Kirillov, Alexander, et al. "Segment anything." arXiv preprint arXiv:2304.02643 (2023). Different structures can be flexibly segmented here. The input image data can also vary. For example, Ma, Jun, and Bo Wang. "Towards foundation models of biological image segmentation." Nature Methods 20.7 (2023): 953-955 describes how the foundation model can solve various segmentation tasks related to structure segmentation in microscopy image data. In summary, the foundation model is an extremely complex model that often does not work in real time due to its long inference time. It has been trained on a large amount of data covering a variety of fields. As a result, it has learned the ability to recognize what are generally considered objects or regions and is able to generalize well. In order to know exactly what to segment, detect, or recognize by other means, the model requires contextual information in the form of prompts in addition to the input image.

[0040] Various examples are based on the finding that using a base model directly to solve a specific task can be problematic. This is because such a base model has a large number of parameters, so reasoning with such a base model is particularly computationally intensive. As a result, such a base model is often not suitable for rapid use in the field, for example on a microscope computer. Since the base model is not domain-specific but rather generic, the prediction quality of the base model can be lower than that of a domain-specific model. In particular, the prediction quality of the base model can be related to the quality of the provided contextual information (cues).

[0041] A description of techniques for how to effectively use a base model in conjunction with a domain-specific model to implement image data processing is given below. More generally, a description of techniques for how a first model with low complexity (the domain-specific model) and a second model with high complexity (e.g., the base model) interact to implement image data processing is given below.

[0042] The first model may be pre-trained to some extent. The first model may be trained on a relatively elementary basis. This means that the results of the first prediction of the first model may be relatively inaccurate or incomplete. For example, it is conceivable that the first (domain-specific) model only provides incomplete domain-specific predictions. For example, if cell instances are localized, it is conceivable that the first model cannot correctly identify each cell instance. For example, the first model may not distinguish between "cells" and "non-cells" for each pixel in the microscope image, but only provide point annotations (such as geometric center points) for a portion of all cells.

[0043] Various examples are based on the following discovery: despite the limited quality of a first prediction of a first model, it may be possible to determine contextual information of a second model (particularly a base model) from the first prediction. The second model may then generate a second prediction that is of higher quality than the first prediction.

[0044] For example, the second localization generated by the second model may have a higher accuracy than the first localization provided by the first model, that is, contain fewer false positive results, etc. For example, it is conceivable that the second localization provides a more complete localization, such as detecting all instances of a structure (such as cells), or at least a much larger portion than the first localization. It is also conceivable that the second localization has a higher level of detail than the first localization: for example, the first localization may include only point localizations, while the second localization provides segmentation. Segmentation is used not only to locate the center point or center of gravity of a specific structure, but also to indicate the extent and shape of the corresponding structure.

[0045] In various examples, localization can be provided as point localization or bounding box localization or segmentation. This can provide semantic segmentation. Semantic segmentation aims to classify each pixel in the image according to its associated object class, without distinguishing between individual instances of the same class. Another example is instance segmentation, which assigns unique identifiers if they belong to the same class. An application example is identifying and separating individual cells in multiplexed immunofluorescence images so that each cell can be analyzed separately. Panoptic segmentation combines elements of semantic segmentation and instance segmentation, with the goal of identifying the semantic class of each pixel and assigning a unique identifier to each object of the same class.

[0046] Thus, the second prediction enables different applications. For example, the first machine learning model can be retrained based on the second prediction. Thus, the second model can be used to generate ground truth values ​​for training data with relatively little user effort, which can then be used to improve the first model.

[0047] Figure 1 The data processing pipeline 200 is schematically shown. In the data processing pipeline 200, point localization with classification assignment (combined localization and classification prediction) is determined by a first model 231 (domain-specific model) and passed to a second model 241 (base model) as context information. The base model 241 includes an encoder 242 and a decoder 244 that generate a latent representation 243. The base model 241 delivers the resulting mask of semantic segmentation. This will be described in detail below.

[0048] The image data 211 is passed to the domain specific machine learning model 231 (at 281). The domain specific model 231 may be implemented according to a previously known architecture. Figure 1 The example in represents a deep neural convolutional network, such as U-net. This gives a first prediction 212 of image features in the image data (at 282). At 284, this first prediction 212 is passed to a general machine learning base model 241 as context information. The base model 241 processes the image data 281 as input at 283. The base model 241 provides a second prediction 212 of the image features at 285.

[0049] At 284, the context information is passed to the base model 241: the context information is not passed to the input layer of the base model 241 (to which the image data 211 is passed). As a general rule, the context information of the first model can be passed to the input of the second model differently from the image data. For example, the image data can pass through the encoder, but the context information does not. In the case of a (visual) transformer architecture for the second model, as described by Kirillov, Alexander et al., the reference segmentation is converted into an embedding and provided to the base model as a token downstream of the decoder; this variant can also be considered in this case. In the case where the base model is implemented as a CNN, the context information can flow into one of the hidden layers (i.e. between the input layer and the output layer) as a continuous value or similarly as an embedding.

[0050] exist Figure 1 In the example of , predictions 291, 292 are related to the localization of structures of multiple classes, in particular the localization of cells and background in a phase contrast microscopy image. Therefore, in addition to localization, classification is also performed.

[0051] The prediction 291 of the domain-specific model 231 is an incomplete point localization (black circle with white border: “background”; white circle with black border: “cell”). Not all cells are identified.

[0052] The predictions 292 of the base model are complete semantic segmentation (dark: “background”; bright: “cell”).

[0053] In general, the techniques described here are not limited to such cell localization predictions. In general, a wide range of tasks can be solved, such as: cell detection, fusion estimation, cell instance segmentation (specific cell lines only), tissue (region) segmentation, particle detection, organelle localization, overview image segmentation, and many others. Applications in completely different technical fields are also conceivable, such as bacteria, PCBs, gas and oil, neuro, metrology, material science, transcriptomics, etc.

[0054] In general, it can be assumed that the accuracy of the predictions 291 of the domain-specific model is lower than the accuracy of the predictions 292 of the general base model. For example, accuracy can here represent the number of incorrect localizations (deviations from the center of the cell) or the number of regions that are incorrectly assigned to cells (what is imaged is actually the background). Figure 1 In the specific example in , a pixel may be incorrectly classified as "background" when a cell is actually located there.

[0055] For example, it is also conceivable that the prediction 291 of the domain-specific model 231 provides localization at a lower image space density than the prediction 292 of the base model 241. For example, the prediction 291 localizes the cell localization only for a few pixels, while the prediction 292 classifies all pixels of the microscope image 211 as "cell" and "background".

[0056] For example, it can be envisioned that prediction 291 of domain-specific model 231 provides a lower level of detail than prediction 292 of general base model 241. For example, the level of detail can relate to the level of detail used to provide localization. For example, the localization of prediction 291 is point localization and thus has a lower level of detail, while the localization of prediction 292 is instance segmentation and thus has a higher level of detail.

[0057] Figure 2 The data processing device 90 is schematically shown. The data processing device 90 comprises a processor 91 and a memory 92, as well as a communication interface 93 and a human-machine interface 94.

[0058] User input may be received through the human-machine interface 94 and processed by the processor 91. For example, an image or information may be output to the user via the human-machine interface 94. The human-machine interface 94 may include one or more of the following elements: a screen, a mouse, a keyboard.

[0059] The image data may be received via the communication interface 93. For example, the image data may be received from an image data memory or an image data acquisition unit such as a microscope. It is also conceivable that the image data acquisition unit is controlled by the processor 91 to acquire the image data.

[0060] Processor 91 is capable of loading program code from memory 92 and executing it. When processor 91 executes the program code, this causes the processor to perform the described techniques, such as: applying domain-specific models; applying a general machine learning base model; applying multiple models with different complexities; training models; performing an interactive annotation process; and so on.

[0061] Figure 3 is an exemplary method. Figure 3 The method in is computer-implemented. For example, when a processor executes program code from a memory, Figure 3 The method in can be implemented by a processor. For example, Figure 3 The method in can be Figure 2 The processor 91 in is implemented. Figure 3 The method involves processing of image data.

[0062] Image data are obtained in block 905. For example, these may be microscope image data. They may be 2D image data or 3D image data. In block 905, image data may be loaded from an image data storage. For example, image data may be acquired in block 905 from an image data acquisition unit such as a microscope.

[0063] The image data may then be pre-processed in optional block 910. Image data pre-processing in block 910 may, for example, include rescaling and / or normalizing intensities and / or correcting aberrations. Other examples involve denoising and / or deconvolution.

[0064] Rescaling can consist of resizing the image so that the pixel dimensions of certain objects correspond to the desired standard size. This greatly simplifies the problem, thus reducing the required complexity of the model.

[0065] As an alternative or in addition to rescaling, other image properties can also be normalized, such as contrast and / or brightness, etc. For example, microscopy images typically have low intensity contrast if phase contrast is used as an imaging technique. It has been determined that this may impair the analysis in the model.

[0066] Sometimes, image data may contain convolutions, for example when they are acquired using magnetic resonance tomography or other imaging methods operating in the spatial frequency domain. In such cases, deconvolution methods (also called image reconstruction) can be used to remove or at least reduce such convolution artifacts.

[0067] Noise can be caused by various factors, such as camera quality, exposure or transmission errors. There are known algorithms to reduce this noise - usually based on its statistical properties.

[0068] In block 915, a first model is applied to the (possibly pre-processed) image data. The first model is a domain-specific model. Figure 1 and Model 231 discusses corresponding aspects.

[0069] Applying the first model gives a first prediction of the image features. For example, the localization of certain structures can be obtained as examples of image features. For example, corresponding localization information with classification information can be provided. For example: "The cell nucleus is located at point (23,45). The background is located at coordinates (89,1011)".

[0070] It should have as low a false positive rate as possible (i.e. objects found by the model are actually real) and deliver results with high confidence. A higher false negative rate (some objects not found) may have to be accepted here, but this can be advantageous in most cases. This is based on the finding that the false positive rate should be as low as possible so that the contextual information drawn from it is not incorrect and the second model does not receive any wrong "base points". However, the absence of points (false negative rate) is not a problem, as the second model is intended to supplement the completeness of the segmentation in any case.

[0071] In block 920, context information for the base model is determined. The context information is determined based on the first prediction from block 915. In one variation, the first prediction from block 915 may be used directly as context information. For example, the first prediction may be a point location and associated classification information, such as a combination of Figure 1 In this case, the first prediction can be used directly as context information.

[0072] It has been determined that it may be helpful for the first prediction to be incomplete. For example, it may be helpful to provide localization information corresponding to the incomplete localization of certain structures in the image data. In some cases, the first prediction may be inherently incomplete due to domain shift (that is, image data with certain properties has not been seen so far in the training of the first model).

[0073] However, as an alternative, one may consider modifying (ie changing or revising) the first prediction.

[0074] The first prediction may be subsampled, for example using a random subsampling scheme. Example: The first model delivers the coordinates of the associated nucleus for each cell in the image. An incomplete version may be obtained by randomly sampling entries from the list of all nuclei. This amounts to artificially deteriorating the first prediction. In addition to random subsampling schemes, deterministic subsampling schemes may be used, for example, to discard or retain a certain portion of the first prediction depending on the confidence level.

[0075] However, subsampling is only one option for modification. Other examples include applying noise (noising). For example, the coordinates of the point locations can be applied with noise. For example, white noise or Gaussian smearing can be applied.

[0076] As an alternative or in addition to algorithmically modifying the first prediction to determine contextual information, the first prediction may also be modified based on user input received from a user interface. Figure 2 Human-machine interface 94.

[0077] In block 925, the image data may optionally be preprocessed before being input into the second model. In particular, as an alternative or in addition, the corresponding techniques as discussed above in conjunction with block 910 may also be applied to block 925. This means that the image data may be preprocessed before being input into the first model in block 910, and / or the image data may be preprocessed in block 925 before being input into the second model. As an example, the preprocessing of the image data in block 910 may be different from that in block 925. As an example: it may be considered that in block 110, the image data is rescaled, while in block 925 the contrast is normalized.

[0078] Then, in block 930, the second model is applied to obtain a second prediction. The second prediction has the same semantic content as the first prediction in block 915 (e.g., both predictions locate cells). However, the second prediction may be of higher quality. For example, the second prediction may be more accurate, have a higher level of detail, have a higher resolution in image space, and / or be more complete than the first prediction. Corresponding aspects related to the application of the second model have been incorporated herein by reference. Figure 1 Discussion was conducted: Basic Model 241.

[0079] The prediction from block 930 then continues to block 934 . Figure 3 Some possible implementations of block 934 are shown in the form of blocks 935, 940, 945, 950, 955, 960. Individual blocks or combinations of blocks may be implemented.

[0080] For example, the prediction of the second model may be output in block 935. For example, the prediction of the second model may be output to the user via a screen in accordance with the original image data (see Figure 1 : Semantic segmentation of image data 211) superimposed (see Figure 1 : second prediction 292). In this variant, the original result from box 915 can be improved by the second model in box 930. For example, if incomplete cell localization occurs using the domain-specific model in box 915, this incomplete cell localization can be supplemented by applying the base model in box 930. However, the level of detail can also be increased. For example, if the domain-specific model in box 915 delivers localization information in the form of cell center points, the base model in box 925 can provide one or more instance segmentation masks for cells and background or even different cell types. This enhanced information can then be output to the user.

[0081] Alternatively or additionally, a confidence level for the second prediction may be determined in block 940. This may be done based on variations between multiple instances of the second prediction from block 930 ( Figure 3 It is shown that box 930 can be implemented in multiple iterations 931). Specifically, for example, multiple iterations 931 of box 930 can be performed, wherein different context information is obtained in each iteration. As an example, it has been described in conjunction with box 920 (so that box 920 can also be implemented multiple times): context information can be determined (ascertained) based on the first prediction from box 915 by modifying the first prediction. For example, different sub-scans can be performed. For example, different sub-sampling schemes can be used. The random sub-sampling scheme can also be implemented multiple times. Thus, different context information can be provided for different iterations 931 of box 930. Then, the confidence can be calculated based on the individual results of the multiple iterations 931 as follows: For example, the variance / entropy or other statistical deviation measures of the individual results can be used to determine the confidence. As an alternative or in addition, a measure of the matching situation can also be determined based on the differences between the individual results. The individual results can also be processed in a model to determine the confidence. See, for example, German patent application 102021100444.

[0082] Variations between multiple instances of the second prediction may be determined in the image space in an analytical manner. This allows a confidence in the image space of the image data to be determined in an analytical manner. For example, regions of the image data where the confidence in the second prediction is relatively low may be identified. However, in other cases, a global value of the confidence may also be output, i.e., different instances of the second prediction may be averaged by varying local properties of the second prediction, for example.

[0083] As described above, individual results for determining confidence levels can be obtained by repeatedly implementing the second model using different context information. Individual results for different iterations 931 can also originate from different instances of the second model (e.g., different base models trained using the same data but different initializations). In general, multiple instances of second predictions of image features can be obtained by the second model. These multiple second instances can be obtained by different instances of context information that have been modified relative to each other and / or different instances of first predictions of image features from the first model (e.g., the first model can be initialized with different random parameters; the first model can also have a random component, such as for sampling potential feature vectors, so that the first model can deliver different first predictions for the same input). Different configurations of the second model can also be utilized, such as different training states, etc.

[0084] For example, the confidence determined in box 940 can be output to the user. For this purpose, the human-machine interface can be controlled accordingly. It is also possible to consider using the confidence to evaluate the second model. For example, the confidence can be compared with a predefined threshold, and if the confidence is below the predefined threshold, the further use of the second prediction of the second model can be prevented. Model selection can be based on the determined confidence; see also box 950. For example, a suitable first model can be selected, through which context information that allows the second model to predict with high confidence is obtained. Therefore, this means that the confidence of the prediction of the second model can be used as a measure of the suitability of the first model to create context information for the second model.

[0085] In block 945, the predictions from block 930 may be used as input to another base model. This means that the data generated by the base model may also be used for another base model, i.e., for example (i) as training data to train a new base model and / or (ii) as context information (rather than context information obtained from a domain-specific model) in an iterative process.

[0086] Model selection can optionally be performed in block 950. If multiple models can be selected for a task (e.g., a "model zoo"), the most appropriate model can be automatically determined. Base models and / or domain-specific models can be selected here. The model with the highest confidence can be selected.

[0087] For example, the suitability of a domain-specific model for generating contextual information can be inferred based on the confidence of the prediction of the base model. Therefore, a suitable domain-specific model can be selected according to the confidence of the prediction of the base model.

[0088] In block 955, an annotation process can be optionally set based on the second prediction from block 930. The annotation process is used to generate ground truth values ​​used in block 960 for retraining the first model. Therefore, this means that, in other words, ground truth values ​​of the training data are generated by user interaction during the annotation process; by setting the annotation process based on the second prediction from block 930, this generation of ground truth values ​​can be greatly accelerated compared to pure manual annotation. For example, the localization and / or classification of the second prediction can be output to the user in a way that is superimposed with the image data. The user can then confirm or edit the second prediction. For example, since the user does not have to manually perform cell segmentation in a microscope image, but is able to check and locally change the segmentation provided by the second model, the task becomes simpler for the user. Depending on how the segmentation mask is displayed, there are various options for simple, accurate and fast user interaction. For example, the user can modify the segmentation mask by moving the spline of the segmentation mask. The user can modify the annotation by moving the points of the polygon. In the case of a pixel mask, a "brush tool" can be used to add ("paint") or remove ("erase") parts of the mask.

[0089] Thus, the user can directly modify the second prediction to adapt the reference true value. In addition to such direct manipulation of the second prediction, the user can also modify the second prediction indirectly. A corresponding example will be explained below.

[0090] Block 920 may also be implemented multiple times during the user interactive annotation process (see iteration 921). In particular, it is contemplated that the user interactive annotation process includes modifying the context information based on the user input, and accordingly outputting to the user the effect of the modification of the context information on the second prediction of the second model. For example, the user may move a specific point location predicted by the first model, and then check how this change in the context information affects the second prediction of the second model. The user may delete or add individual point locations, and in each case check how this change in the context information affects the prediction of the second model.

[0091] The user interactive annotation process may be arranged not only based on the second prediction, but also, alternatively or additionally, based on the confidence of the second prediction (see block 940). For example, in the user interactive annotation process, those parts of the second prediction with a lower confidence (see block 940) may be output with a higher priority (e.g., before other parts with a lower priority or with special emphasis, etc.). For example, if the confidence is determined analytically in the image space, then it may be considered to graphically highlight the areas in the image space associated with the lower confidence.

[0092] In summary, block 955 may enable an interactive annotation process that may be used to generate segmentation masks, instance segmentation masks, point annotations, and / or bounding box annotations for microscopy data used for model training. Here, contextual information for the second model is provided only by the user, only by the first model (in an earlier training state), or by both, and the predictions of the second model are then used as ground truth values ​​for the training data for retraining the first model. The annotation process may be interactive, wherein multiple inferences (multiple iterations 931 in one iteration 951) are performed on the second model or at least a portion of the second model during the annotation process to indicate to the user which output generated the current contextual information.

[0093] For example, it has been explained above that the second model obtains context information not at the input layer (where the image data is provided), but at a point along the data processing pipeline that is arranged closer to the output layer and at a certain distance from the input layer. In this case, in order to determine the impact of the modification of the context information on the second prediction of the second model, in each case only that part of the second model that depends on the context information can be reasoned again. In other words, the first part of the second model (see Figure 1 :241-1) can be inferred once; then only the second part of the second model that depends on the context information (see Figure 1 :241-2) can be inferred again. This saves a lot of computing resources.

[0094] By manipulating the context information, the user can delete and add annotations until a satisfactory result is obtained. In this case, the user interactive annotation process can include modifying the context information in box 920 based on user input (the corresponding user input for modification has been described in conjunction with box 920) and outputting the impact of the modification introduced by the user on the second prediction in box 930. This means that the user can, for example, change, add or delete individual elements of the second prediction, such as specific point locations in a cell or background. The user can then be shown how the predictions of the second model change due to this change. The second model can be provided to the user once; then, through an interactive annotation process for generating training data, the user can use the second model to train one or more domain-specific models themselves. During the annotation process, active learning functions can also be implemented, especially showing the user areas worth annotating. Thereby, the domain-specific model (and the optional base model) can be continuously improved over time.

[0095] Different metrics can be used to measure the region "worthy" of annotation. One exemplary metric is confidence (see box 940). Table 1 below gives further metrics.

[0096]

[0097] Table 1: Various strategies that allow the user to manipulate the first prediction through the active learning process. Based on such criteria, a portion of the prediction of the first model can be selected and presented to the user; the user can then modify or manipulate the corresponding portion of the first prediction to create changed context information. Block 955 can also include performing an annotation process. After the annotation process is completed, training data can be obtained. The training data can then be used to retrain the first model in block 960.

[0098] However, it is not necessary to use the user interactive annotation process in all variants. For example, it is conceivable that the retraining in box 960 is based on training data containing the second (unchanged) predictions from box 930 as ground truth values. These techniques are based on the assumption that the second model is much more complex and powerful than the first model and can be generalized to a variety of domains. However, knowledge for a specific task can be transferred to the first model. This approach is similar to the so-called "knowledge distillation" (a small, real-time capable model learns from a large, complex but computationally intensive model), except that the large model (here the second model from box 930) is guided by the first model from box 915 by providing contextual information (the so-called "hints"). The process of generating ground truth values ​​can also be performed iteratively.

[0099] The process of generating ground truth and retraining (box 955 and box 960) can also be iterated here: in iteration n, the first model generates context information and the second model delivers outputs, which together with the input image form an instance of training data. This instance of training data is then used to retrain the first model (retraining). The retrained first model is then implemented again to generate new context information in iteration n+1; the second model is then implemented again using the new context information in iteration n+1. This process can be repeated multiple times (see Figure 3 Iteration 951 in .

[0100] In a special case (when the first model is still untrained or does not exist at all, such as in the first iteration), the context information in box 920 comes from the user. This can generally be seen as a "assisted annotation process", where the user only needs to annotate a small part of the data compared to traditional annotation tools, and the creation of the complete dataset is largely assisted by the second model (when creating the target data).

[0101] Figure 4The use scenarios of the basic model according to various examples are schematically shown. The figure shows a central server 89 in communication connection with multiple data processing devices 81-84. For example, the server 89 can communicate with the data processing devices 81-84 via the Internet. The server 89 can be configured as a cloud server.

[0102] Each of the data processing devices 81-84 may be configured, for example, in accordance with the data processing device 90 (see Figure 2 ) to configure.

[0103] The server 89 may store a central base model, e.g. Figure 1 The base model 241 in the data processing device 81-84 is stored in the central storage of the base model 241, and is transferred to the data processing device 81-84. The central storage of the base model 241 enables the central training of the base model.

[0104] Based on these copies of the base model, each data processing device 81-84 can then each generate and train multiple domain-specific models locally. To this end, the data processing devices 81-84 are coupled to associated image acquisition units 85-88, wherein the image acquisition devices in 85-86 are configured to acquire image data. For example, the image acquisition units 85-88 can be microscopes.

[0105] Typically, different users use different data processing devices 81-84 and image acquisition units 85-88. Different users also have different requirements for corresponding domain-specific models. For example, one user may want to perform cell instance segmentation based on image data of a specific cell type recorded using a specific color filter. Another user may want to perform semantic cell segmentation using different filters or no filters at all or using another imaging technique (for example, using a light sheet microscope instead of a fluorescence microscope). For these reasons, it is necessary to create and train different domain-specific models to suit the respective requirements of different users. The central base model assists and simplifies this process.

[0106] In summary, a description has been given above of a technique for using an upstream domain-specific model with low complexity to generate context information for a general base model with high complexity. Both the domain-specific model and the base model process image data. This makes it possible, among other things, to improve and accelerate the generation of ground truth for training data used to retrain the domain-specific model.

[0107] It goes without saying that the features of the above-described embodiments and aspects of the present invention can be combined with each other. In particular, these features can be used not only in the described combinations, but also in other combinations or alone, without departing from the scope of the present invention.

[0108] As an example, a description of various aspects related to a domain-specific machine learning model with a U-net architecture has been given above. However, the specific architecture of the domain-specific machine learning model is not important for the technology described herein. The technology described herein can work with different architectures that are theoretically known to those skilled in the art.

[0109] As another example, a description of structure localization in relation to point localization and instance segmentation has been given above. However, other forms of localization information, such as bounding box localization, are also conceivable.

Claims

1. A computer-implemented method for processing image data (211), wherein the method comprises: - acquiring (905) said image data (211), - processing (281, 915) the image data (211) in a domain specific machine learning model (231) to obtain (282) a first prediction (291) of an image feature in the image data (211), wherein the first prediction comprises a first localization of a structure in the image data (211), - determining (920) context information for a common machine learning base model (241) based on said first prediction of said image features, -Based on the context information, processing (930) the image data (211) in the general machine learning base model (241) to obtain a second prediction (292) of the image feature, wherein the second prediction includes a second location of the structure in the image data (211).

2. The computer-implemented method of claim 1 , wherein the method further comprises: -Retraining the domain-specific machine learning model (231) based on training data including a baseline true value determined based on the second prediction (291).

3. The computer-implemented method of claim 1 , wherein the method further comprises: -Based on the second prediction: setting up (955) a user interactive annotation process for generating benchmark truth values ​​for retraining the domain-specific machine learning model (231).

4. The computer-implemented method of claim 3, The user interactive annotation process includes modifying the context information based on user input and accordingly outputting to the user the impact of the modification of the context information on the second prediction.

5. The computer-implemented method of claim 4, in, In order to determine the impact of the modification of the context information on the second prediction, in each case only the part (241-2) of the common machine learning base model (241) that depends on the context information is re-inferred.

6. The computer-implemented method of claim 4, The user interactive annotation process is set up including: - Based on an active learning process: selecting parts of the first prediction in order to modify the context information based on the user input.

7. The computer-implemented method of claim 1, wherein the first positioning has a first accuracy, wherein the second positioning has a second accuracy, The second precision is greater than the first precision.

8. The computer-implemented method of claim 1, wherein the first positioning has a first image space density, wherein the second positioning has a second image space density, The second image space density is greater than the first image space density.

9. The computer-implemented method of claim 1, wherein the first position has a first position detail level, wherein the second position fix has a second position fix detail level, The second positioning detail level is greater than the first positioning detail level.

10. The computer-implemented method of claim 1, The first prediction (291) and / or the second prediction (292) further comprises classifying the structure in the image data (211).

11. The computer-implemented method of claim 10, wherein the first prediction comprises a point location of the structure and a classification assignment associated with a plurality of classifications, The second prediction includes a plurality of result masks of instance segmentation or semantic segmentation of the structure, wherein the plurality of result masks correspond to the plurality of classifications.

12. The computer-implemented method of claim 1, wherein determining (920) the context information comprises: - Modifying said first prediction (291).

13. The computer-implemented method of claim 12, Wherein modifying the first prediction (291) includes sub-sampling the first prediction, optionally performing random sub-sampling.

14. The computer-implemented method of claim 12, Wherein modifying the first prediction comprises applying noise to the first prediction.

15. The computer-implemented method of claim 12, Wherein the first prediction is modified based on user input received from a user interface.

16. The computer-implemented method of claim 15, The method further comprises: - outputting a portion of the first prediction to a user, the portion being selected based on an active learning process, and - receiving said user input, said user input relating to modification of said part of said first prediction.

17. The computer-implemented method of claim 1, wherein a plurality of instances of the second prediction (291) of the image feature are obtained by the general machine learning base model (241), Wherein the multiple instances of the second prediction (291) correspond to different instances of the context information that have been modified relative to each other and / or different instances of the first prediction of the image feature and / or different configurations of the machine learning base model (241).

18. The computer-implemented method of claim 1, wherein the method further comprises: - Determining (940) a confidence level for the second prediction based on the variation between multiple instances of the second prediction.

19. The computer-implemented method of claim 18, wherein the confidence measure is determined analytically in the image space of the image data (211), The method further comprises: -Based on the confidence level of the second prediction: setting up a user interactive annotation process for generating a baseline true value for retraining the domain-specific machine learning model.

20. The computer-implemented method of claim 1, wherein the method further comprises: - Preprocessing (910, 925) the image data (211) prior to processing in the domain-specific machine learning model (231) and / or prior to processing in the general machine learning base model (241).

21. The computer-implemented method of claim 20, The preprocessing includes one or more of the following operations: rescaling, intensity normalization, aberration correction, denoising, and deconvolution.

22. The computer-implemented method of claim 20, The image data (211) is pre-processed differently before being processed in the domain-specific machine learning model (231) than before being processed in the general machine learning base model (241).

23. A data processing device (90) comprising a processor (91) and a memory (92), wherein the processor (91) is configured to load a program code from the memory (92) and to execute the program code, wherein the processor (91) implements the method according to claim 1 when executing the program code.

Images