Processing of images containing overlapping particles

Abstract

This disclosure provides a computer-implemented method for generating training data for training a machine learning model that generates a segmentation mask for an image containing overlapping particles. Training data is generated from images containing sparse particles that do not overlap. Generating a mask of non-overlapping particles is generally not a problem if the particles can be clearly identified; in many cases, simple methods such as thresholding are sufficient to produce usable masks. The sparse images can then be combined with images containing artificial overlaps. The same approach can be used for masks that generate large amounts of training data, as many combinations can be created from only a small set of images. The method is simple and efficient and can be adapted to many domains, for example, by adding style transfer to the generated images or by including additional augmentation steps.

Description

Technical Field

[0001] The present invention relates to a computer implementation method for generating training data for training a machine learning model, the machine learning model being used to generate a segmentation mask for an image containing overlapping particles; and a computer implementation method for generating a machine learning model, and a method for generating a segmentation mask for an image containing overlapping particles. Background Technology

[0002] In many medical contexts, the ability for clinicians and scientists to reliably analyze images of cells and cell populations is crucial. Such analyses can include cell counting, quantification of mean cell volume, and measurements of related cell shapes. These analytical techniques can be directly automated when images contain sparse cell populations. However, in denser cell populations, overlapping cells are frequently present in slice images, making these tasks more challenging. This is particularly true for, for example, erythrocyte populations, where overlap and / or clumping are especially prevalent problems.

[0003] Machine learning techniques have greatly aided the analysis of such images containing overlapping cells. However, training such models requires ground truth information—that is, a “correct” segmentation mask for images containing overlapping cells. Currently, generating such ground truth information is extremely time-consuming and often requires trained clinicians to manually and painstakingly annotate the images before they can be used to train machine learning models.

[0004] This invention aims to overcome the difficulty of preparing realistic information, thereby improving the efficiency of generating training data and thus the efficiency of generating robust machine learning models. While the primary application of this invention relates to images containing overlapping cells, it should be noted that it is equally applicable to any other image containing overlapping particles. Summary of the Invention

[0005] In summary, this application relates to a process for generating training data for machine learning models. At a high level, the data is generated by obtaining segmentation masks for images of non-overlapping particles. The images of non-overlapping particles are then overlaid to generate simulated composite images. Appropriate segmentation masks for these simulated composite images are then generated by overlaying segmentation masks for each of the composite images containing non-overlapping particles.

[0006] More specifically, according to a first aspect of the invention, a computer-implemented method for generating training data for training a machine learning model for generating a segmentation mask for an image containing overlapping particles is provided. The method includes the steps of: acquiring a plurality of images containing non-overlapping particles; obtaining a plurality of segmentation masks, each segmentation mask corresponding to a corresponding image among the plurality of acquired images; generating a plurality of simulated composite images by combining the images among the plurality of images, the simulated composite images including overlapping particles; and for each simulated composite image, generating a simulated segmentation mask by combining the segmentation masks, the segmentation masks being obtained for each of the constituent images, the constituent images being combined to generate the simulated composite image.

[0007] It has been shown (see below) that training data generated in this way produces highly accurate segmentation masks from images of overlapping particles.

[0008] The language used in this application is clear and well understood by those skilled in the art. However, to avoid ambiguity, we have set forth some definitions below to aid in interpreting the claims:

[0009] A "machine learning model" is an analytical model that relies on the principles of machine learning to generate meaningful outputs from inputs. Specifically, machine learning models are those whose performance improves with experience and that are able to "learn," thus improving accuracy, for example, compared to manual processing, as they are applied to more and more training data. Machine learning models can be classification models (where inputs are classified into a specific category) or regression models (where continuous values ​​of parameters are predicted based on inputs). This invention relates to models that rely on supervised learning, where training data includes inputs and "correct" outputs associated with those inputs. This can be referred to as real-world information. In some cases, machine learning models may rely on "semi-supervised" learning, where training data includes a mixture of only input data and data including the inputs and "correct" outputs as described above. Typically, machine learning models operate based on the principles of mathematical optimization. Examples of machine learning models that can benefit from the training data generation method of the first aspect of the present invention are the U-Net algorithm described in Ronneberger, Olaf et al. “U-Net: Convolutional Networks for Biomedical Image Segmentation” arXiv:1505.04597 (2015), Fast R-CNN, Faster R-CNN, and the YOLO algorithm described in Girshick, Ross et al. “Rich feature hierarchies for accurate object detection and semantic segmentation” arXiv:1311.2524 (2013).

[0010] "Training" a machine learning model is the process of improving the performance of a machine learning model. In this invention, "training data" is used to train the model. Training data includes a set of data in which inputs are associated with "correct" outputs. This may be referred to as "labeled" data. For example, in this invention, training data includes pairs of simulated composite images (inputs) and simulated segmentation masks (outputs). These are preferably stored in association with each other to ensure training efficiency.

[0011] In the context of this invention, “acquiring” multiple images of non-overlapping particles can involve directly acquiring the images, i.e., capturing images using some kind of image capturing device. Alternatively, “acquiring” multiple images can refer to receiving multiple images after they have already been captured in a previous step. Further details regarding the acquisition of these images are set forth later in this application. “Images containing non-overlapping particles” should be interpreted as meaning images containing only non-overlapping particles, or images that substantially do not contain overlap between particles.

[0012] This invention includes the step of obtaining a segmentation mask for each of a plurality of images containing non-overlapping particles. Here, the term "segmentation mask" refers to a two-dimensional array of values, each preferably corresponding to a corresponding pixel in the image, wherein the value corresponding to a pixel provides information for classifying the content displayed in that pixel. The range of values ​​that each pixel of the segmentation mask can take is preferably limited to a few values. For example, the segmentation mask can be in the form of a binary segmentation mask, where, for example, a value of 0 indicates that there is no cell in the pixel, and a value of 1 indicates that there is a cell in the pixel. As we will explain later, when considering images containing overlapping particles, the segmentation mask may take more than two values. The values ​​of pixels in the segmentation mask are different from the "pixel values" of the image (explained later). To avoid confusion, the pixel values ​​in the segmentation mask will hereafter be referred to as mask pixel values.

[0013] Further definitions can be elaborated in the full text.

[0014] In some cases, acquiring images that include only non-overlapping particles can be challenging. Therefore, to ensure this is the case, acquiring multiple images containing non-overlapping particles can include: acquiring multiple images comprising: a first subset of images including non-overlapping particles, and a second subset of images including both non-overlapping and overlapping particles; and removing the second subset of images from the multiple images. The removal of the second subset of images can be performed manually (i.e., by an experienced clinician or scientist) or automatically. For example, an independent neural network can be trained to identify images containing overlapping particles, which can then be removed. In this way, only images containing non-overlapping particles are retained. This is advantageous because it means that the segmentation mask obtained from the first subset of images is more accurate, since they are obtained only from non-overlapping images.

[0015] There are many ways to obtain a segmentation mask for multiple images. For example, obtaining a segmentation mask for each of multiple images may include applying an adaptive thresholding algorithm to the images. Alternatively or alternatively, obtaining a segmentation mask for each of multiple images may include applying a Laplacian edge detection algorithm to the images, as used in, for example, P. Vallotton, C. Sun, D. Wang, L. Turnbull, C. Whitchurch and P. Ranganathan, “Segmentation and tracking individual Pseudomonas aeruginosa bacteria indense populations of motile cells,” 2009 24th International Conference Image and Vision Computing New Zealand, 2009, pp. 221-225, doi: 10.1109 / IVCNZ.2009.5378409. It should be noted that various image segmentation methods are well known to those skilled in the art and can be found online. As outlined above, obtaining the segmentation mask preferably involves obtaining a binary segmentation mask, where a mask pixel value of 1 indicates the presence or absence of a cell at a given point in the image, and a mask pixel value of 0 indicates the opposite. Preferably, a mask pixel value of 1 indicates the presence of a cell, and a mask pixel value of 0 indicates the absence of a cell. Essentially, the segmentation mask provides an equivalent of a black-and-white image, where cells are displayed in white and the background in black, or vice versa.

[0016] We will now discuss in more detail how to generate segmentation masks, simulated combined images, and simulated segmentation masks. This is done with reference to a first and a second image among multiple images, but it should be understood that this can be applied to more than one pair of images among multiple images. For example, this process can occur for every possible combination of two images among multiple images. Or a random set of pairs can be determined, and the process can occur only for those determined pairs.

[0017] Multiple images may include a first image and a second image, each image consisting of an array of pixels, each pixel having an associated pixel value. Here, the term "pixel value" is used to refer to one or more values ​​used to define a pixel. For example, in a grayscale image, a pixel value may include a single value defining the brightness or intensity of a pixel. This may be measured in a range of, for example, 0 to 255. Alternatively, in a color image, a pixel value may include three values ​​representing the RGB values ​​of a pixel (also in the range of 0 to 255). Other three values ​​may be used, such as hue, saturation, [intensity, value, brightness, or luminance] representation, or YCbCr representation. It should be understood that a pixel value represents a characteristic of a pixel, whether it is brightness or color (or both). The step of obtaining multiple segmentation masks preferably includes: obtaining a first segmentation mask corresponding to the first image, and obtaining a second segmentation mask corresponding to the second image. Generating multiple simulated combined images may then include combining the first image and the second image. Furthermore, generating simulated segmentation masks may include combining the first segmentation mask with the second segmentation mask.

[0018] The first and second images can be combined to generate a simulated composite image in various ways. Specifically, combining the first and second images can include: for each pixel in the first image, combining the value of that pixel with the pixel value of the corresponding pixel in the second image to obtain the corresponding pixel in the simulated image. This combination can be a direct additive combination, or it can be an average of the two pixel values. However, the inventors have identified two methods for combining pixel values ​​that produce more efficient results than additive combination or averaging methods.

[0019] i. First, in some cases, combining the pixel values ​​of the first image with the corresponding pixel values ​​of the second image involves selecting the minimum pixel value among the two and using that minimum pixel value to simulate the corresponding pixel value of the image. Specifically, the "minimum pixel value" here refers to the minimum brightness or minimum intensity pixel value. Therefore, where the pixel values ​​do not explicitly include intensity pixel values ​​(e.g., where the pixel values ​​include the RGB representation of the pixel's color), this step may further include a sub-step of deriving the intensity value of the pixel in question and then selecting the minimum of the two. This step can be performed individually on each of the three image channels, for example, when the channels represent RGB values, to produce the minimum pixel with values ​​of [min(r1,r2), min(g1,g2), min(b1,b2)]. It should be understood that other types of image channels can be used. This method of combining pixel values ​​has proven to be particularly effective.

[0020] ii. Optionally, combining the pixel values ​​of the first image with the corresponding pixel values ​​of the second image includes adding the absorbance values ​​of the pixels in the first image to the absorbance values ​​of the corresponding pixels in the second image; calculating the exponent of the addition result; and using the calculated exponent as the pixel value of the corresponding pixel in the analog image. This method of combining pixel values ​​has also proven to be particularly effective. Again, as mentioned in the preceding paragraph, if the pixel values ​​of the first and second images do not include absorbance values, this step may further include a sub-step of deriving the absorbance values ​​of the pixels in question, and then adding the absorbance values ​​together.

[0021] We will now discuss the properties of segmentation masks and their combinations in more detail. A segmentation mask may comprise multiple pixels, each having a mask pixel value indicating the category to which that pixel belongs. These categories may include: a background category (e.g., a mask pixel value of either 0 or 1), indicating that no particles are present in the pixel, or a particle category (e.g., another mask pixel value of either 0 or 1), indicating that particles are present in the pixel. Combining a first segmentation mask with a second segmentation mask may include combining the mask pixel values ​​of pixels from the first segmentation mask with the mask pixel values ​​of corresponding pixels from the second segmentation mask to determine the mask pixel values ​​of corresponding pixels from a simulated segmentation mask, which indicate the category to which the pixels of the simulated segmentation mask belong. Because the first and second segmentation masks are preferably binary segmentation masks, it should be understood that the process of combining segmentation masks is a straightforward process, less labor-intensive than, for example, manually identifying overlapping regions in an image containing overlapping particles.

[0022] When both the first and second segmentation masks are binary masks, it should be understood that the simulated segmentation has three categories: combining a pixel in the background category with another pixel in the background category produces a pixel in the background category; combining a pixel in the background category with a pixel in the particle category produces a pixel in the non-overlapping pixel category; and combining a pixel in the particle category with another pixel in the particle category produces particles in the overlapping particle category. These categories can correspond to specific mask pixel values. For example, in the first and second segmentation masks: a mask pixel value of 0 corresponds to the background category; a mask pixel value of 1 corresponds to the particle category. In the simulated segmentation mask: a mask pixel value of 0 corresponds to the background category; a mask pixel value of 1 corresponds to the non-overlapping particle category; and a mask pixel value of 2 corresponds to the overlapping particle category. This demonstrates how direct the processing is, leading to a significant improvement in efficiency.

[0023] This invention can be used in many applications. Specifically, while its use in histological image analysis is beneficial, other uses are also conceivable. The particles may include one or more of the following: red blood cells; white blood cells; nanoparticles; organelles; tumor cells; circulating tumor cells; bone marrow cells; pathogenic cells, or a mixture of pathogenic cells and healthy cells; stem cells; bacteria; yeast; microparticles; bacteriophages; beads; fluorescent objects; phosphorescent objects; or translucent objects.

[0024] In some cases, realistically simulating the appearance of overlapping real particles by combining them in the manner described so far can be challenging. It is important that the training data includes images showing the same type of particles as the machine learning model that will ultimately be used for processing. In other words, the style of the simulated images may differ from the style of the images used by the machine learning model for analysis. This problem can be addressed by using style transfer algorithms.

[0025] The conversion may occur as follows:

[0026] In one scenario, the acquired images may be in a first style, different from the style of the images ultimately used in the machine learning model for analysis. The method may then further include a step of converting each of the acquired images into a transformed set of images in a second style, which is the style of the images ultimately used in the machine learning model for analysis. Subsequent steps of the method can then be performed normally on the transformed sets of images, rather than the acquired images.

[0027] The transformation can occur after the generation of the simulated composite images. Specifically, after generating multiple simulated composite images in a first style, the method can include transforming each of the simulated composite images to generate corresponding multiple transformed simulated composite images in a second style, which is the style of the images that will ultimately be used for analysis by the machine learning model.

[0028] In some cases, both of these processes may occur. For example, multiple combined images can be converted to a second style for the purpose of generating a segmentation mask. However, original (first-style) images can be combined to generate a simulated combined image, which itself can be converted to a second style (individually).

[0029] Here, the term "style" can refer to the general appearance of particles in an image. Style transfer algorithms have been developed that allow, for example, the conversion of an image of a winter landscape into its equivalent summer version, or the generation of an image of what people would look like as adults. In the context of this invention, style transfer algorithms are used to convert simulated images of overlapping objects into realistic images that will benefit the training of the algorithm, since the overlapping objects fail to accurately capture the statistical properties exhibited by the real images of those overlapping objects.

[0030] Preferably, converting the image in question from a first style to a second style involves applying a style transfer algorithm to the image. Specifically, multiple images or simulated combinations of multiple images can be input into the style transfer algorithm. The style transfer algorithm can be a machine learning algorithm, preferably involving a convolutional neural network or a generative adversarial neural network (GAN). Appropriate style transfer algorithms are discussed in Leon A. Gatys, Alexander S. Ecker, Matthias Bethge; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 2414-2423, Li, Yijun, et al. “Universal style transfer via feature transforms.” arXiv preprint arXiv:1705.08086 (2017), Pramadhan, Anugrah Akbar et al. “Cycle Generative Adversarial Networks Algorithm With Style Transfer For Image Generation” arXiv:2101.03921 (2021), and Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, Timo Aila; Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR). This is discussed in 2020, pp. 8110-8119.

[0031] The aforementioned features relate to the generation of training data for a machine learning model. A second aspect of the invention provides a computer-implemented method for generating a machine learning model for generating a segmentation mask for an image containing overlapping particles, the method comprising the steps of: providing a machine learning algorithm; receiving training data generated by the computer-implemented method of the first aspect of the invention; and training the machine learning algorithm using the received training data to generate a trained machine learning model. In some cases, the provided machine learning algorithm may be a deep learning segmentation algorithm. One type of deep learning segmentation algorithm particularly advantageous to the invention is the U-net algorithm as described in Olaf Ronneberger, Philipp Fischer, ThomasBrox Medical Image Computing and Computer-Assisted Intervention (MICCAI), Springer, LNCS, Vol. 9351: 234–241, 2015, or any similar segmentation algorithm or encoder-decoder architecture that allows classification of individual pixels. Alternatively, a deep learning algorithm, such as YOLO (“Look Only Once”), can be used to identify bounding boxes of overlapping regions, in which case the bounding boxes can be simply derived from the segmentation mask by obtaining the extrema of the segmentation mask.

[0032] In some cases, machine learning or deep learning algorithms may include a weighted cross-entropy loss function. This can be used to address dataset imbalance, which arises because background in an image may outnumber particles, and particles may occupy more regions than overlapping particles. Specifically, the weights of the weighted cross-entropy loss function can be determined using the following steps: sampling n simulated segmentation masks from the received training data; for each mask, calculating the sum of pixels in each pixel category; dividing the calculated sum by the total number of pixels in the simulated segmentation mask; and taking the average of the n simulated segmentation masks and using the inverse function of these values.

[0033] The computer implementation method may further include augmenting the received training data. The term augmentation is well-known in the art and refers to the expansion of the dataset, providing a larger amount of training data, and thus a machine learning model that is more thoroughly trained on a given amount of input images. Alternatively, augmentation means that the model can be well trained using only a small initial dataset. Augmentation may occur in two distinct phases: before the images are combined to generate a simulated composite image; and in the final phase. These will be discussed sequentially.

[0034] The computer implementation method may further include the step of generating augmented multiple images from the acquired multiple images. Generating the augmented multiple images may include applying one or more transformations to each of the acquired multiple images, each transformation resulting in a corresponding transformed image. Here, the transformation may be rotation, translation, reflection, or scaling. In a preferred embodiment, two transformed images can be generated from a given image by applying a vertical flip and (separately) a horizontal flip. This effectively doubles the number of images from which simulated combined images can be generated, thus greatly increasing the number of simulated combined images that can be generated, and consequently greatly increasing the amount of training data generated using the computer implementation method of the first aspect of the invention. It should be noted that when the computer implementation method includes this augmentation step, simulated combined images are generated by combining images from the augmented multiple images, and multiple segmentation masks are obtained for the corresponding images of the augmented multiple acquired images.

[0035] In addition, the simulated composite images and simulated segmentation masks can also be augmented by applying the same type of transformations to them, thus including further expansion in the final stage. Specifically, one or more transformations can be applied to each simulated composite image to generate one or more corresponding transformed simulated composite images. The same operation can be performed on the simulated segmentation mask to generate a corresponding transformed segmentation mask. Each transformed simulated segmentation mask can then be associated with its corresponding transformed simulated composite image. This provides an additional amount of training data.

[0036] Above, we have described the process of generating training data and how to use the training data to generate a machine learning model. Therefore, a third aspect of the present invention provides a computer-implemented method for generating a segmentation mask of an image containing overlapping particles, the method comprising the steps of: receiving an image containing overlapping particles; and applying a machine learning model to the received image, generating the machine learning model using the computer-implemented method of the second aspect of the present invention, and being configured to generate a segmentation mask based on the received image, the segmentation mask indicating regions of the image containing overlapping particles. The computer implementation may further include the step of displaying the output, i.e., the segmentation mask. Specifically, in response to the generation of the segmentation mask, the computer-implemented method may further include the step of generating computer-readable instructions, which, when received by a computer or the like, cause the computer to display the segmentation mask on a display component.

[0037] As we have explained, the machine learning model generated based on a large amount of training data according to the first aspect of the invention is necessarily more efficient and capable of providing a more accurate segmentation mask, which more accurately indicates the shape of the overlapping regions of particles. This is advantageous because the improved accuracy in identifying particle shapes (even when particles overlap) leads to an improved ability to classify particles based on their shape, size, and / or volume. This is particularly important when generating segmentation masks for images displaying overlapping cell populations. The ability to automatically and reliably classify cells based on their size, shape, and / or volume, as well as any other spatial or structural features, can lead to improvements, for example, in diagnosis.

[0038] The computer-implemented invention may further include the step of determining characteristics of one or more particles in the generated segmentation mask. These characteristics may include size, shape, area, hemoglobin content, and / or volume. Preferably, characteristics of multiple or all particles may be determined.

[0039] A fourth aspect of the invention provides a computer-implemented method for automatically classifying particles in an image including overlapping particles. This computer-implemented method includes generating a segmentation mask for the image according to a computer-implemented method according to a third aspect of the invention; and determining the class of at least one particle in the segmentation mask based on one or more of the particle's shape, size, and / or volume. Preferably, the computer-implemented method for determining the class of at least one particle includes determining the class of a plurality of particles in the image, more preferably, the class of all particles in the image. In some cases, the class of only the entire particle in the image may be determined. Determining the class may include determining features of the particles and subsequently grouping the particles based on the determined features. The computer-implemented method according to the fourth aspect of the invention preferably relates to images in which the particles are cells (types of which have been previously listed in the application). A particularly advantageous application of the computer-implemented method of the fourth aspect of the invention is the diagnosis of thalassemia, a blood disorder in which a patient's hemoglobin levels are lower than normal. Diagnosing thalassemia using known methods may fail due to red blood cell segmentation defects caused by overlap. This can then lead to an underestimation of red blood cell volume. The present invention addresses these problems.

[0040] The first to fourth aspects of the invention, as defined above, all relate to computer-implemented methods. However, other aspects of the invention are also contemplated. Another aspect of the invention provides a computer program including instructions that, when executed by a computer, cause the computer (or specifically, its processor) to perform the method of any one of the first to fourth aspects of the invention. Another aspect of the invention provides a computer-readable data carrier storing a computer program of the foregoing aspects of the invention. Another aspect of the invention provides a system including a processor configured to perform a computer-implemented method of any one of the first to fourth aspects of the invention.

[0041] The present invention includes combinations of the described aspects and preferred features, unless such combinations are clearly not permitted or explicitly avoided. Attached Figure Description

[0042] Embodiments of the present invention will now be described with reference to the accompanying drawings, in which:

[0043] Figure 1 shows an example of a system that can be used to carry out a computer-based implementation of the method of the first aspect of the present invention.

[0044] Figure 2 shows an example of a computer implementation method according to the first aspect of the present invention.

[0045] Figure 3A Figure 3E shows a series of images representing the various stages of the method in Figure 2.

[0046] Figure 4 illustrates a series of steps by which particles in an image containing overlapping particles can be classified according to a computer-implemented method according to the fourth aspect of the present invention.

[0047] Figure 5 , 6A Figures 6B and 6B show examples of results that can be obtained using a computer-implemented method according to the third aspect of the invention. Attached Figure Description

[0049] Aspects and embodiments of the invention will now be discussed with reference to the accompanying drawings. Other aspects and embodiments will be apparent to those skilled in the art. All documents mentioned herein are incorporated by reference.

[0050] Figure 1 shows an example of system 100, which can be used to perform computer implementations of the methods of the first, second, third, and fourth aspects of the present invention. Those skilled in the art will understand that this is merely an example of a system, and the components of such a system can be readily and appropriately reorganized. Such reorganized versions of system 100 should also be considered to be covered by the present invention. System 100 includes a client device 102, an image capture device 103, and an image processing system 104 connected via a network 106. Network 106 can be in the form of a wired network, a wireless network (such as a Wi-Fi network), or a cellular network. System 100 can be in the form of a local area network (LAN) or a wide area network (WAN). Preferably, client device 102 and image capture device 103 are connected to image processing system 104 via the Internet. In an alternative arrangement (not shown), the functionality of client device 102 and image processing system 104 can be provided on the same computer, which can be directly connected to image capture device 103. Here, the term "computer" should be understood to encompass conventional desktop computers, laptops, tablets, smartphones, or any other type of computing device known to a person skilled in the art. Throughout this application, we refer to various functional "modules": these modules can be in the form of physical hardware modules or software modules implemented in code. Modules can be a combination of hardware and software modules. In the context of this invention, client device 102 is a user device through which a user can upload images (containing overlapping particles) to obtain a segmentation mask using the computer-implemented method of the third aspect of this invention. Image capture device 103 may include a camera, which may be connected, for example, to a microscope (not shown) or any other laboratory equipment suitable for obtaining images of particles that may not be visible to the naked eye. In some cases (also not shown), client device 102 may include image capture device 103.

[0051] Before explaining the operation of system 100 with reference to the flowchart in Figure 2, we will now describe the structure of system 100 in more detail. The image processing system 104 includes a client interface module 107, a processor 108, an image capture device interface module 109, and a memory 110. The processor 108 includes multiple functional modules, including a machine learning model training module 111, which includes a training data generation module 112, comprising: an image filtering module 113, a segmentation module 114, a data augmentation module 115, an image combination module 116, a simulated segmentation mask generation module 118, and a style transfer module 120. The machine learning model training module 111 further includes a training module 119. The processor 104 further includes a segmentation mask generation module 122, a feature extraction module 124, and a classification module 126. The memory 110 includes a machine learning model 128, a buffer 130, and a database 132 for training data. The function of each module described above is obvious to those skilled in the art, but for completeness, we will now refer to... Figure 2 The operation of the system is described in detail in Figures 3A to 3E.

[0052] Figure 2 Figures 3A to 3E illustrate an example computer implementation method according to a first aspect of the present invention, namely, a computer implementation method for generating training data for training a machine learning model, the machine learning model being used to generate a segmentation mask for an image containing overlapping particles. In the illustrated example, the computer implementation method shown in Figure 2 is performed by a training data generation module 112 of a machine learning model training module 111:

[0053] In the first step S100, multiple images are acquired. For example, multiple images can be received from the image capture device 103 or the client device 102 via the network and the image capture device interface module 109 or the client device interface module 107, respectively. Alternatively, multiple images can be downloaded from a database or other sources. The means of acquiring multiple images in step S100 is not central to the invention, and those skilled in the art will readily understand that there are many ways to acquire them. The images in the multiple images preferably comprise relatively sparse particle swarms, as shown in FIG3A. FIG3A shows four example images A, B, C, and D, each containing sparse particle swarms. It should be understood that FIG3A through 3E show schematic versions of the images, and when implemented, the images will preferably show the types of particles on which the machine learning model 128 will ultimately be used.

[0054] The machine learning model 128 is preferably trained using images containing non-overlapping particles. To ensure this, in step S102, only those images containing only non-overlapping particles are selected. This can be done automatically using the image filtering module 113. Alternatively, the user can manually select appropriate images (this can be facilitated using the image filtering module 112). As shown in Figure 3B, only images B and D are selected because images A and C (see Figure 3A) contain overlapping particle pairs (labeled "O").

[0055] Since only images containing non-overlapping particles were selected in step S102, segmentation module 114 generates a segmentation mask for each of the selected images in step S104. It should be noted that because the selected images contain only non-overlapping particles, the generation of the segmentation mask is relatively simple (i.e., it can be done using conventional segmentation algorithms such as thresholding, edge detection, and watershed methods, all of which are well known to those skilled in the art). Figure 3C shows examples of segmentation masks generated from images B and D. These are binary masks, where areas containing particles are shown in black, and background areas are shown in white. In practice, the segmentation mask will consist of an array of pixels, each with a value of 0 or 1, depending on whether particles are present.

[0056] In step S106 (which can occur in parallel with step S104, as shown), the image combination module 116 combines pairs of selected images to generate multiple simulated combined images containing overlapping particles. The ways in which images can be combined (e.g., by selecting the minimum pixel intensity value) have been explained earlier in this application. Figure 3D shows a schematic example of the simulated combined images that can be generated. At this point, it is worthwhile to discuss the operation of the style transfer module 120. The more similar the training images are to the images on which the machine learning model 128 will ultimately be used, the more effective the machine learning model 128 will ultimately prove. Given this, it may be desirable to convert the “original” training images into images with the same style as the images on which the machine learning model 128 will ultimately be used. To achieve this, the style transfer module 120 can be used, for example, to convert selected images or simulated combined images into transformed images of the desired style. This may occur, for example, between steps S102 and S104, or after step S106 – or both. Examples of style transfer algorithms that can be performed by the style transfer module 120 are cited elsewhere in this application.

[0057] In step S108, which can occur in parallel with step S106, the simulated segmentation mask generation module 118 generates a simulated segmentation mask by combining the various segmentation masks. The resulting segmentation mask can take three values: indicating the background, non-overlapping particle regions, and overlapping particle regions. An example is shown in Figure 3E, where the two masks from Figure 3C are combined. The background region is shown in white, the non-overlapping particle regions in gray, and the overlapping particle regions in black. Specifically, pairs of segmentation masks corresponding to the selected image pairs forming each simulated combined image are combined such that one simulated segmentation mask exists for each simulated combined image.

[0058] Finally, in step S110, each simulated combined image and its corresponding simulated segmentation mask are combined and stored in the training data database 132. At this time, the training data 132 includes unlabeled input data (i.e., simulated combined images) and labeled output data (i.e., simulated segmentation masks).

[0059] After generating training data 132, according to the computer implementation method of the second aspect of the present invention, the machine learning algorithm can be trained by training module 119 using training data 132 to generate a trained machine learning model 128.

[0060] The third and fourth aspects of the present invention provide computer-implemented methods for generating a segmentation mask from an image containing overlapping particles and automatically classifying particles in the image including overlapping particles. Figure 4 is a flowchart covering the computer-implemented methods of the third and fourth aspects of the present invention:

[0061] In step S200, the image processing system 104 receives an image containing overlapping particles (e.g., from the client device 102 via the client device interface module 107, or from the image capture device 103 via the image capture device interface module 109).

[0062] In step S202, the generated machine learning model 128 is applied to the received image to generate a segmentation mask. It should be noted that throughout this application, the segmentation mask preferably includes the outlines of multiple particles within the image. In cases where particles overlap, through training, the machine learning model 128 is able to predict or infer the possible outlines of the particles, even if parts of the outline are occluded by overlapping particles. The effectiveness of the computer implementation method of this invention will be demonstrated later in this application.

[0063] In step S204, after the segmentation mask has been generated, the feature extraction module 124 can be used to extract features from the segmentation mask, such as particle shape, particle size, and particle volume. Other relevant descriptors can be found online. When using a deep learning model for classification, manual feature extraction is not required because the model can be trained directly on the image and the segmentation mask.

[0064] Finally, in step S206, the classification module 126 can be used to classify the particles based on the extracted features (i.e., place them into one of several categories). In some cases, step S206 can start directly from step S202 (i.e., classification can be performed directly based on the segmentation mask, rather than by explicitly performing the feature extraction step).

[0065] Figure 5 highlights the effectiveness of the machine learning model generated using a computer-implemented method (e.g., the second aspect of the invention). The left column shows the real-world image, i.e., the "correct" segmentation mask obtained for an image containing overlapping cell populations. Overlapping regions are shown in a lighter color. The right column shows the results of applying the machine learning model trained according to the second aspect of the invention to the same image. It should be understood that the shapes of the overlapping regions are virtually identical, thus demonstrating that the invention facilitates the generation of highly accurate segmentation masks. Figure 6A Figure 6B further demonstrates the effectiveness of this method, except that it uses nanoparticles instead of cell populations.

[0066] The features disclosed in the foregoing specification, or the manner in which the disclosed functions are expressed or implemented in a particular form in the following claims, or the methods or processes for obtaining the disclosed results, may be used individually or in any combination in various forms to achieve the present invention.

[0067] While the invention has been described in conjunction with the exemplary embodiments described above, many equivalent modifications and variations will be apparent to those skilled in the art upon presentation of this disclosure. Therefore, the exemplary embodiments of the invention described above are to be considered illustrative rather than restrictive. Various changes may be made to the described embodiments without departing from the spirit and scope of the invention.

[0068] To avoid any doubt, any theoretical explanations provided herein are intended to improve the reader's understanding. The inventor does not wish to be bound by any of these theoretical explanations.

[0069] Any chapter headings used herein are for organizational purposes only and should not be construed as limiting the subject matter described.

[0070] Throughout the specification, including the following claims, unless the context otherwise requires, the words “comprising” and “including”, as well as variations of “comprises” and “comprising”, should be understood to imply inclusion of the stated integers or steps or groups of integers or steps, but not to exclude any other integers or steps or groups of integers or steps.

[0071] It should be noted that, as used in this specification and the appended claims, the singular forms “a,” “an,” and “the” include plural referents unless the context explicitly specifies otherwise. A range herein may be expressed as from “about” one particular value and / or to “about” another particular value. When expressing such a range, another embodiment includes from one particular value and / or to another particular value. Similarly, when a value is expressed as an approximation, the use of the antecedent “about” will be understood to form another embodiment of a particular value. The term “about” in relation to numerical values ​​is optional and means, for example, + / - 10%.

Claims

1. A computer-implemented method for generating a machine learning model for generating a segmentation mask, the segmentation mask being used for an image containing overlapping particles, the method comprising: Provide machine learning algorithms; Receive training data generated using a computer-implemented method for generating training data, wherein the computer-implemented method for generating training data includes: Acquire multiple images containing non-overlapping particles; Multiple segmentation masks are obtained, each segmentation mask corresponding to a specific image among the multiple images; Multiple simulated composite images are generated by combining images from the plurality of images, the simulated composite images including overlapping particles; and For each simulated composite image, a corresponding simulated segmentation mask is generated by combining the segmentation masks, wherein the segmentation masks are obtained for each of the constituent images, and the constituent images are combined to generate the simulated composite image; and The received training data is used to train the machine learning algorithm to generate a trained machine learning model, wherein the machine learning model is a deep learning segmentation algorithm.

2. The computer implementation method according to claim 1, wherein: The plurality of images include a first image and a second image, each image being composed of an array of pixels, and each pixel having a relevant pixel value indicating the characteristics of the pixel; Obtaining the plurality of segmentation masks includes: Obtain a first segmentation mask corresponding to the first image; and Obtain a second segmentation mask corresponding to the second image; Generating the plurality of simulated combined images includes combining the first image and the second image; and Generating multiple simulated segmentation masks includes combining the first segmentation mask with the second segmentation mask.

3. The computer implementation method according to claim 2, wherein: Combining the first image and the second image includes: for each pixel in the first image, combining the pixel value of the pixel with the pixel value of the corresponding pixel in the second image to obtain the pixel value of the corresponding pixel in the simulated image.

4. The computer implementation method according to claim 3, wherein: Combining the pixel value of the pixel in the first image with the pixel value of the corresponding pixel in the second image includes selecting the minimum pixel value among the two pixel values ​​and using the minimum pixel value as the pixel value of the corresponding pixel in the simulated image.

5. The computer implementation method according to claim 2, wherein: The segmentation mask comprises multiple pixels, each pixel having a mask pixel value indicating the category to which the pixel belongs, the categories including: Background category, which indicates that no particles are present in the pixel; and Particle category, which indicates that there are particles present in the pixel.

6. The computer implementation method according to claim 5, wherein: Combining the first segmentation mask with the second segmentation mask includes: combining the pixel value of a pixel in the first segmentation mask with the pixel value of a corresponding pixel in the second segmentation mask to determine the pixel value of a corresponding pixel in the simulated segmentation mask, wherein the pixel value of the simulated segmentation mask indicates the category to which the pixel of the simulated segmentation mask belongs.

7. The computer implementation method according to claim 6, wherein: Pixels in the background category are combined with another pixel in the background category to generate pixels in the background category; Pixels in the background category are combined with pixels in the particle category to generate pixels in the non-overlapping particle category; and Pixels in the particle category are combined with another pixel in the particle category to produce pixels in the overlapping particle category.

8. The computer-implemented method according to any one of claims 1 to 7, wherein: The multiple simulated combined images are in a first style; and The method further includes the step of transforming each of the plurality of simulated composite images to generate a corresponding plurality of transformed simulated composite images in a second style, the second style being the style of the images that the machine learning model will ultimately use for analysis.

9. The computer implementation method according to claim 8, wherein: Converting the generated composite images from the first style to the second style involves applying a style transfer algorithm to each of the composite images.

10. The computer-implemented method according to any one of claims 1 to 7, wherein: The particles include one or more of the following: red blood cells; white blood cells; nanoparticles; organelles; tumor cells; circulating tumor cells; bone marrow cells; pathogenic cells, or a mixture of pathogenic cells and healthy cells; stem cells; bacteria; yeast; microparticles; bacteriophages; beads; fluorescent objects; phosphorescent objects; or translucent objects.

11. The computer implementation method according to claim 1, wherein: The deep learning segmentation algorithm includes a weighted cross-entropy loss function.

12. The computer implementation method according to claim 11, wherein: The weights of the weighted cross-entropy loss function are determined using the following steps: Sample n simulated segmentation masks from the received training data; For each mask, calculate the sum of pixels in each pixel category; Divide the calculated sum by the total number of pixels in the mask; Take the average value of the n simulated segmentation masks; as well as The reciprocals of these values ​​are used as the weights of the weighted cross-entropy loss function.

13. A computer-implemented method for generating a segmentation mask for an image containing overlapping particles, the method comprising the following steps: Receive images containing overlapping particles; A machine learning model is applied to a received image, the machine learning model is generated using a computer-implemented method according to any one of claims 1 to 12, and the machine learning model is configured to generate a segmentation mask based on the received image, the segmentation mask indicating regions of the image containing overlapping particles.

14. A system for generating a segmentation mask for an image containing overlapping particles, the system being configured to perform the method of claim 13.

Images