A cell division state classification method and system based on latent space features

Abstract

The invention discloses a method and system for classifying cell division states based on latent space features. The method comprises the following steps: constructing a classification prediction network, comprising an encoder, an inverse division image generator, an abnormal division image generator and a normal division image generator; respectively collecting a plurality of images of inverse division, abnormal division and normal division as training sets for the three generators; fixing the encoder, respectively inputting a plurality of random vectors into the three image generators to generate a plurality of virtual images, obtaining virtual image sets of the three generators, and respectively training each generator using the training set and the corresponding virtual image set; fixing the generator, extracting latent space features from the training set using the encoder, respectively inputting the extracted features into the corresponding generators to obtain a generated image set, and training the encoder using the training set and the corresponding generated image set; and inputting a plurality of images of cells to be detected into the classification prediction network to obtain classification results of the division states of the cells to be detected.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and system for classifying cell division states based on latent space features. Background Art

[0002] Cell division refers to the process of proliferation of living cells, that is, one cell divides into two cells. The cell before division is called the mother cell, and the new cell formed after division is called the daughter cell. In some cases, cells divide abnormally, that is, the number of daughter cells obtained by one cell division is greater than two. Another case is reverse division, that is, two cells merge into one cell. Detecting the state of cell division is helpful to understand the laws of cell growth and reproduction. Cell division characteristics are important indicators for evaluating cell division modes, which can reflect the differences between different division modes. By identifying the characteristics of different cell division modes, the classification of cell division modes can be achieved. Incubator technology can not only provide a stable environment for the growth of cell tissues, but also periodically monitor the growth process of cells and upload these images to the server. Cytologists can use these periodic image data to monitor the development of cell tissues and judge the mode of cell division based on this. With the development of artificial intelligence and computer vision technology, the problem of automatic identification and monitoring of cell division modes in incubators has become increasingly important.

[0003] At present, the detection of cell division mainly relies on the manual judgment of experts, or first identifying the region of interest (ROI) in the culture dish, and then further analyzing and judging these areas. With the advancement of technology, the entire image can also be directly sent to the convolutional neural network for classification and judgment.

[0004] Although existing technologies have achieved some results in cell division detection, there are still some limitations: many cell division images contain information such as granular cells and fragments that are irrelevant to the cell division method. Directly using deep learning methods may extract this interference information, affecting the accuracy of feature extraction and interfering with the final results. At the same time, the method of directly extracting features for discrimination may fail to accurately extract the essential characteristics of cell division, and sometimes the classification is correct only by chance, which affects the accuracy of the algorithm. Summary of the invention

[0005] The present invention proposes a cell division state classification method and system based on latent space features, which solves the problems of low efficiency and poor accuracy in cell division detection in the prior art.

[0006] In order to solve the above technical problems, the present invention provides a method for classifying cell division states based on latent space features, comprising the following steps:

[0007] Step S1: constructing a classification prediction network, wherein the classification prediction network includes an encoder and a generator, wherein the generator includes an inverse split image generator, an abnormal split image generator, and a normal split image generator;

[0008] Step S2: collecting a plurality of reverse split images, abnormal split images and normal split images as training sets for the three generators;

[0009] Step S3: fixing the encoder, inputting a plurality of random vectors into three image generators to generate a plurality of virtual images, obtaining virtual image sets of the three generators, and independently training each generator using the training set and the corresponding virtual image set;

[0010] Step S4: fixing the generator, using the encoder to extract latent space features from the training set, inputting the extracted latent space features into the corresponding generators to obtain a generated image set, and training the encoder using the training set and the corresponding generated image set;

[0011] Step S5: inputting multiple images of the cells to be detected into the classification prediction network to obtain classification results of the division states of the cells to be detected.

[0012] Preferably, the step S3 of training each generator using the training set and the corresponding virtual image set comprises the following steps:

[0013] Step S31: removing granular cells and debris from all cell images in the training set, and repairing the removed parts to obtain a denoised image set;

[0014] Step S32: performing latent space feature extraction and feature fusion on the denoised image set and the corresponding virtual image set respectively to obtain a first fusion feature and a second fusion feature;

[0015] Step S33: Calculate the classification cross entropy function between the first fusion feature and the second fusion feature, and use the classification cross entropy function to update the parameters of the generator.

[0016] Preferably, the step S4 of training the encoder using the training set and the corresponding generated image set comprises the following steps:

[0017] Step S41: inputting the denoised image set and the corresponding generated image set into a discriminator, wherein the discriminator is used to determine whether the input image is a generated image, and outputs 0 if it is a generated image, and outputs 1 otherwise;

[0018] Step S42: updating the parameters of the encoder using a back propagation algorithm;

[0019] Step S43: extracting latent space features from the training set using the updated encoder, inputting the extracted latent space features into the corresponding generators respectively, and updating the generated image set;

[0020] Step S44: Repeat steps S41 to S43 until the probability of the discriminator outputting 0 is less than a set probability threshold.

[0021] Preferably, the binary classification neural network in step S41 includes several fully connected layers, a hidden layer and an output layer, the hidden layer adopts a nonlinear activation function Leaky ReLU, and the output layer adopts an activation function Sigmoid.

[0022] The present invention also provides a cell division state classification system based on latent space features, which is implemented based on the above-mentioned cell division state classification method based on latent space features. The system includes a data acquisition module, a generator training module, an encoder training module and a classification prediction module; the generator training module includes an inverse division image generator training module, an abnormal division image generator training module and a normal division image generator training module;

[0023] The data acquisition module is used to respectively acquire a plurality of inverse split images, abnormal split images and normal split images as training sets for the three generators;

[0024] The generator training module: through a fixed encoder, input multiple random vectors into three generators to generate multiple virtual images as virtual image sets, and use the training set and the virtual image set to train the generator;

[0025] The encoder training module: by fixing the generator, using the encoder to extract latent space features from the training set, inputting the extracted latent space features into the corresponding generators to obtain a generated image set, and using the training set and the generated image set to train the encoder;

[0026] The classification prediction module uses the trained encoder to extract the features of the cell image of the cell to be detected, and outputs the classification result of the division state of the cell to be detected.

[0027] Preferably, the classification prediction module adopts a residual neural network ResNet.

[0028] Preferably, the generator training module updates the parameters of the generator by calculating the classification cross entropy loss between the training set and the virtual image set.

[0029] Preferably, the encoder training module uses a discriminator to determine the probability that the images in the generated image set are real images, and uses a back-propagation algorithm to update the parameters of the encoder until the average probability that the images in the generated image set are real images is greater than a set probability threshold.

[0030] The present invention also provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein the memory is used to store computer programs or instructions; the processor is used to execute the computer program or instructions so that the electronic device implements the above-mentioned cell division state classification method based on latent space features.

[0031] The present invention further provides a storage medium, applied to an electronic device, wherein the storage medium carries one or more computer programs. When the one or more computer programs are executed by the electronic device, the electronic device can implement the above-mentioned cell division state classification method based on latent space features.

[0032] The benefits of the present invention include at least:

[0033] 1. Automatically extract image features through the encoder, reducing the reliance on manual analysis by experts and improving classification efficiency; through the learning and fusion of latent space features, the ability to recognize different cell division states is improved, and a high classification accuracy can be maintained even under different influencing factors such as different cell positions in the image, the presence of fragments and granular cells in the image;

[0034] 2. By training the generator and encoder separately, the problem that the features extracted during training are not the original image features but just happen to be classified correctly is avoided, thus improving the accuracy of the algorithm;

[0035] 3. Three generators are set for different cell division states. Each cell division state has its own unique characteristics. By designing an independent generator for each division state, these characteristics can be learned and simulated more accurately, thereby improving the accuracy of the classification system. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a comparison chart of cell images under the influence of various factors;

[0037] Figure 2 A schematic diagram of a method flow of an embodiment of the present invention;

[0038] Figure 3 Schematic diagram of the structure of the generator training module of an embodiment of the present invention;

[0039] Figure 4 A schematic diagram of the structure of an encoder training module according to an embodiment of the present invention;

[0040] Figure 5 Schematic diagram of the classification prediction network structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention.

[0042] like Figure 1 As shown, even if cells have the same division mode, the positions of cells in the image may be different or there may be factors unrelated to the cell division mode, such as debris and granular cells, in the image. If the cell division features are directly extracted from the image, factors unrelated to cell division may be extracted, resulting in errors in judging the cell division mode. In order to reduce this situation, the present invention considers judging the cell division mode by extracting the latent space features of the image.

[0043] The latent space is essentially a potential, non-observable space in which features are obtained through complex nonlinear transformations and can capture the key patterns, structures or properties of the original data. Each point in the latent space can be regarded as a compressed representation of the original data. In the image generation model, the features in the latent space can be conceptual representations of the image, such as a combination of information such as color, shape, texture, etc., rather than pixel values.

[0044] like Figure 2 As shown, an embodiment of the present invention provides a method for classifying cell division states based on latent space features, comprising the following steps:

[0045] Step S1: construct a classification prediction network, the classification prediction network includes an encoder and a generator, the generator includes an inverse split image generator, an abnormal split image generator and a normal split image generator.

[0046] Step S2: Collect a plurality of reverse split images, abnormal split images and normal split images as the first training sets of the three generators.

[0047] Step S3: Fix the encoder, map the three random vectors into the latent space through linear regression, convert them into three feature vectors with the same dimension, input these feature vectors into three image generators respectively, each image generator generates three virtual images as a virtual image set, and train the generator using the first training set and the corresponding virtual image set.

[0048] In order to ensure that the extracted latent space features can accurately summarize the cell division characteristics and will not lose useful information, in the embodiment of the present invention, the image after removing interference information and the generated image under the same cell division mode are compared, with the aim of enabling the trained generator and encoder to extract features without interference information as much as possible and generate images without interference information without losing the original information.

[0049] There are two inputs when training the generator. The first input is the fusion feature generated based on the random vector, and the second input is the labeled embryo image with interference information such as debris and granulosa cells removed. The features of the embryo image are extracted through the untrained and frozen encoder, and are also fused to obtain another fusion feature. The two fusion features are input into the fully connected layer, and the classification cross entropy function is used to calculate their probability distribution. Finally, the classification cross entropy loss is used to update the parameters of the generator. The following steps are included:

[0050] Step S31: remove the granular cells and debris on all cell images in the first training set, and repair the removed parts to obtain the second training set.

[0051] Specifically, for the images in the first training set, firstly, manually divide them and use annotation tools such as LabelMe, LabelBox, etc. to manually draw polygons, rectangles or other shapes to annotate the granular cells and debris in the image. After the annotation is completed, export the annotation file in JSON, XML or image format. Use black to represent the background and white to represent the annotated foreground area to generate a binary mask for each image. After removing the annotated debris and granular cells in the mask, the embodiment of the present invention uses white to directly fill the removed part of the image, and then uses the image restoration algorithm Telea to fill the removed part in the image to obtain the second training set. In this way, the processed image is free of the above-mentioned interference information, and at the same time, each image in the second training set has a corresponding original image in the training set.

[0052] Step S32: performing latent space feature extraction and feature fusion on the second training set and the virtual image set respectively to obtain a first fused feature and a second fused feature.

[0053] Step S33: Calculate the classification cross entropy function between the first fusion feature and the second fusion feature, and use the classification cross entropy function to update the parameters of the generator. The classification cross entropy loss function evaluates the difference between the probability distribution of the true label and the probability distribution predicted by the model. In the training process of this embodiment of the present invention, when the classification cross entropy loss function value drops to close to 0.0001, it is considered that the training of the generator has been fully trained.

[0054] Step S4: fix the generator, use the encoder to extract latent space features from the training set, input the extracted latent space features into the corresponding generators to obtain a generated image set, and use the second training set and the corresponding generated image set to train the encoder.

[0055] Specifically, training the encoder using the second training set and the generated image set includes the following steps:

[0056] Step S41: Input the second training set and the corresponding generated image set into the discriminator, and the discriminator is used to determine whether the input image is a generated image. If it is a generated image, it outputs 0, otherwise it outputs 1.

[0057] Step S42: Update the encoder parameters using the back propagation algorithm.

[0058] Step S43: Use the updated encoder to extract latent space features from the training set, input the extracted latent space features into the corresponding generators respectively, and update the generated image set.

[0059] Step S44: Repeat steps S41 to S43 until the probability of the discriminator outputting 0 is close to 0.5, indicating that the discriminator has difficulty distinguishing the two, that is, the generated image achieves the expected goal, and it is judged that the training of the encoder has been completed.

[0060] The embodiment of the present invention adopts a binary classification neural network as a discriminator. The binary classification neural network includes several fully connected layers or convolutional layers, a hidden layer and an output layer, wherein the hidden layer adopts a nonlinear activation function Leaky ReLU, and the output layer adopts an activation function Sigmoid.

[0061] Step S5: input multiple images of the cells to be detected into the classification prediction network to obtain the classification results of the division status of the cells to be detected.

[0062] The embodiment of the present invention also provides a cell division state classification system based on latent space features, including a data acquisition module, a generator training module, an encoder training module and a classification prediction module. For three different cell division modes - reverse division, abnormal division and normal division, the embodiment of the present invention designs corresponding generators respectively. Each generator is trained independently to ensure that the features of the corresponding division mode can be accurately generated.

[0063] The generator training module maps multiple random vectors to the latent space through a fixed encoder to obtain the same number of latent space features, and inputs the generated latent space features into three generators to generate virtual images to obtain a virtual image set, and uses the training set and the corresponding virtual image set to train the generator. The generator uses the variational autoencoder-generative adversarial network VAE-GAN generator.

[0064] The encoder training module uses a fixed generator to extract latent space features from the training set using the encoder, and then inputs the extracted latent space features into the corresponding generator to obtain a generated image set, and uses the training set and the corresponding generated image set to train the encoder. The encoder uses the variational autoencoder-generative adversarial network VAE-GAN encoder.

[0065] The classification prediction module uses the trained encoder to extract the features of the cell image of the cell to be detected, and outputs the classification result of the division state of the cell to be detected. The classification prediction module of the embodiment of the present invention uses a VGG network or a residual neural network ResNet to implement image classification.

[0066] When the system of the embodiment of the present invention is used, two steps are required: constructing a data set and detecting the cell division type.

[0067] Construction of the data set: Collect multiple cell division images, invite multiple cell experts to annotate each image, and use the one with the most votes as the final label of the image. In order to improve the model's adaptability to different cell positions and morphological changes, the embodiment of the present invention uses image enhancement technology, including flipping and rotation operations, to expand the image data set in the training set. Such processing not only increases the diversity of the data, but also significantly improves the generalization of the model in practical applications.

[0068] Cell division type detection includes training of the generator and encoder as well as validation on the test set.

[0069] For the above image dataset, 80% of it is divided into the first training set and 20% is divided into the test set.

[0070] For the first training set, first manually divide the images in the training set by manually drawing polygons, rectangles or other shapes using annotation tools such as LabelMe and LabelBox. After drawing, export the annotation file in JSON, XML or image format. Use black to represent the background and white to represent the annotated foreground area to generate a binary mask for each image. Subtract the fragments and granular cells in the generated mask, and fill the subtracted part with white. Use the image restoration algorithm Telea to precisely fill the subtracted part to generate the second training set so that the processed image does not have the above-mentioned interference information. At the same time, each picture in the second training set has an original picture in the first training set corresponding to it.

[0071] (1) Training the Generator

[0072] The embodiment of the present invention designs a generator for each of the three different cell division modes. During the training process, each image corresponds to a random vector, which is converted into feature vectors in the latent space through linear regression mapping. The generator uses these feature vectors to generate new images. Subsequently, these newly generated images are fed into the encoder to extract feature vectors, and these vectors are fused into a single feature. At the same time, the same number of second data sets are used to extract features through the encoder, and these features are fused for comparison with the features generated by the generator. The generator is iteratively trained multiple times through the classification cross entropy loss function to optimize the parameters of the generator until the cross entropy loss of the generated image in terms of features with the second data set image is reduced to 0.0001, which is considered to meet the standard.

[0073] (2) Training the encoder

[0074] In the training phase of the encoder, the embodiment of the present invention uses the first training set of three cell division modes to train each mode independently. Each image in the first training set is fed into the encoder to extract feature vectors. These feature vectors are then fed into the previously trained generator of the corresponding division mode to generate new images. These images are then compared with the original images in the second data set after eliminating interference, and their authenticity is judged by the basic discriminator. Through multiple iterations, the parameters of the encoder are optimized to ensure that it can accurately distinguish between real images and generated images.

[0075] (3) Test set verification

[0076] In the test set verification phase, the embodiment of the present invention uses the trained encoder to process the images in the test set, extract features, and fuse these features into a single feature. Then, these fused features are sent to the fully connected layer together with the latent space features of different cell divisions extracted from the second data set, and classification prediction is performed through the classification cross entropy loss function. The test results will be used to evaluate the accuracy and reliability of the system in practical applications to ensure that the performance of the system meets the needs of practical applications.

[0077] An embodiment of the present invention also provides an electronic device, comprising at least one processor and a memory connected to the processor, the memory being used to store computer programs or instructions; the processor being used to execute the computer programs or instructions so that the electronic device implements a cell division state classification method based on latent space features as described above.

[0078] An embodiment of the present invention further provides a storage medium, which is applied to an electronic device. The storage medium carries one or more computer programs. When the one or more computer programs are executed by the electronic device, the electronic device can implement the above-mentioned cell division state classification method based on latent space features.

[0079] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. Only the preferred embodiments of the present invention are expressed. The description is more specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. As long as there is no contradiction in the combination of these technical features, they should be considered as the scope recorded in this specification.

[0080] It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these modifications and improvements all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A method for classifying cell division states based on latent space features, characterized in that it comprises the following steps: Step S1: constructing a classification prediction network, wherein the classification prediction network includes an encoder, a generator and a classification prediction module, wherein the generator includes an inverse split image generator, an abnormal split image generator and a normal split image generator; Step S2: collecting a plurality of reverse split images, abnormal split images and normal split images as training sets for the three generators; Step S3: fixing the encoder, inputting a plurality of random vectors into three image generators to generate a plurality of virtual images, obtaining virtual image sets of the three generators, and independently training each generator using the training set and the corresponding virtual image set; Step S4: fixing the generator, using the encoder to extract latent space features from the training set, inputting the extracted latent space features into the corresponding generators to obtain a generated image set, and training the encoder using the training set and the corresponding generated image set; Step S5: input multiple images of cells to be detected into the classification prediction network, use the trained encoder to extract features of the cell images of the cells to be detected, input the extracted features into the classification prediction module, and the classification prediction module outputs the classification results of the division status of the cells to be detected.

2. According to claim 1, a cell division state classification method based on latent space features is characterized in that: The step S3 of using the training set and the corresponding virtual image set to train each generator comprises the following steps: Step S31: removing granular cells and debris from all cell images in the training set, and repairing the removed parts to obtain a denoised image set; Step S32: performing latent space feature extraction and feature fusion on the denoised image set and the corresponding virtual image set respectively to obtain a first fusion feature and a second fusion feature; Step S33: Calculate the classification cross entropy function between the first fusion feature and the second fusion feature, and use the classification cross entropy function to update the parameters of the generator.

3. The method for classifying cell division states based on latent space features according to claim 2, characterized in that: The step S4 of training the encoder using the training set and the corresponding generated image set comprises the following steps: Step S41: inputting the denoised image set and the corresponding generated image set into a discriminator, wherein the discriminator is used to determine whether the input image is a generated image, and outputs 0 if it is a generated image, and outputs 1 otherwise; Step S42: updating the parameters of the encoder using a back propagation algorithm; Step S43: extracting latent space features from the training set using the updated encoder, inputting the extracted latent space features into the corresponding generators respectively, and updating the generated image set; Step S44: Repeat steps S41 to S43 until the probability of the discriminator outputting 0 is less than a set probability threshold.

4. The method for classifying cell division states based on latent space features according to claim 3, characterized in that: The discriminator in step S41 includes several fully connected layers, a hidden layer and an output layer. The hidden layer adopts a nonlinear activation function Leaky ReLU, and the output layer adopts an activation function Sigmoid.

5. A cell division state classification system based on latent space features, implemented based on a cell division state classification method based on latent space features as claimed in any one of claims 1 to 4, characterized in that: The system includes a data acquisition module, a generator training module, an encoder training module and a classification prediction module; the generator training module includes an inverse split image generator training module, an abnormal split image generator training module and a normal split image generator training module; The data acquisition module is used to respectively acquire a plurality of inverse split images, abnormal split images and normal split images as training sets for the three generators; The generator training module: through a fixed encoder, input multiple random vectors into three generators to generate multiple virtual images as virtual image sets, and use the training set and the virtual image set to train the generator; The encoder training module: by fixing the generator, using the encoder to extract latent space features from the training set, inputting the extracted latent space features into the corresponding generators to obtain a generated image set, and using the training set and the generated image set to train the encoder; The classification prediction module uses the trained encoder to extract the features of the cell image of the cell to be detected, and outputs the classification result of the division state of the cell to be detected.

6. The cell division state classification system based on latent space features according to claim 5 is characterized by: The classification prediction module adopts the residual neural network ResNet.

7. The cell division state classification system based on latent space features according to claim 5 is characterized in that: The generator training module updates the parameters of the generator by calculating the classification cross entropy loss between the training set and the virtual image set.

8. The cell division state classification system based on latent space features according to claim 5, characterized in that: The encoder training module uses a discriminator to determine the probability that the images in the generated image set are real images, and uses a back-propagation algorithm to update the parameters of the encoder until the average probability that the images in the generated image set are real images is greater than a set probability threshold.

9. An electronic device, characterized in that: It includes at least one processor and a memory connected to the processor, wherein the memory is used to store computer programs or instructions; the processor is used to execute the computer program or instructions so that the electronic device implements a cell division state classification method based on latent space features as described in any one of claims 1 to 4.

10. A storage medium, applied to an electronic device, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by the electronic device, the electronic device can implement a cell division state classification method based on latent space features as described in any one of claims 1 to 4.

Images