A classification system for cell division processes based on low-density-high-density dual-branching

Abstract

The present invention provides a low-density-high-density dual-branch cell division process classification system, including a cell division degree classification module, a video interpolation module, a feature extraction module and a classification prediction module; the cell division degree classification module obtains the degree of division change intensity; the video interpolation module is used to interpolate the original low-density real embryo image set to obtain a high-density embryo image set with high information density; when interpolating, different numbers of frames are inserted between two adjacent embryo images according to the degree of division change intensity; the feature extraction module is used to extract the features of the low-density real embryo image set and the high-density embryo image set respectively and fuse them; the classification prediction module is used to input the fused features into a fully connected layer for prediction to obtain a classification result. Extract the rules that change over time and space and use the current rules to improve the classification accuracy of different division modes.

Description

Technical Field

[0001] The present invention relates to the technical field of embryonic division recognition, and in particular to a classification system for a double-branched cell division process based on low-density-high-density. Background Art

[0002] Cell division is the process of proliferation in living cells, where one cell divides into two. The cell before division is called the mother cell, and the new cells formed after division are called daughter cells. In some cases, cells undergo abnormal division, where a single cell divides to produce more than two daughter cells. Another case is reverse division, where two cells merge into one. Monitoring cell division states can help us understand the laws governing cell growth and reproduction.

[0003] Cell division detection is an important step in assessing whether cell division is normal. Changes in the cell wall can often reflect whether cell division is normal. By comparing the number of cells before and after, the cell division method can be reflected, thus achieving cell division detection. The incubator not only provides a stable in vitro culture environment for cells, but also has the ability to periodically and continuously obtain images of the entire process of cell division. Combined with the cell photography time recorded in the incubator, cytologists need to judge each cell image based on their own cell assessment experience to obtain the cell division method, which greatly increases the workload of cytologists. Therefore, how to use computer vision methods to assist cytologists in quickly detecting the direction of cell division is of great research significance.

[0004] In actual photography testing, to address the problem of detecting cell division patterns, a culture dish typically has eight wells in a row, typically two rows per dish, for a total of 16 embryos. During the photography process, the time-lapse incubator drives the axis to move above each well, controlling the light source to illuminate before taking a picture. Taking into account the three-dimensionality and size of the embryo, multiple images are captured from top to bottom. After the images are taken, the axis moves to the next well and continues to photograph. Once all 16 embryos in the dish have been photographed, the camera moves to the first well. The camera's photography interval setting ensures that the machine captures the next image within the specified time.

[0005] If you do not consider the equipment problem and directly take photos of the same cell continuously and quickly, the frequent shooting will cause the cell exposure time to be too long, causing damage to the cell and making it impossible to continue shooting.

[0006] Because the looped recording of cell division results in a large time interval between images of the same cell, dramatic cell changes can lead to a loss of motion information and the omission of the specific division process. For example, in the case of abnormal division, one cell fails to divide while another divides into three. The final image will appear to be the same as if two cells divide into four normally, causing problems in determining the cell division method. Even if the intermediate divisions are not missed, the large time gap will result in a less smooth video frame.

[0007] Existing intelligent recognition and detection methods use existing images for processing, failing to account for the large time intervals between photos taken due to equipment issues. Instead, they process sparse video frames directly. This approach fails to account for the limited information contained in sparse video frames, making it difficult to accurately record the details of cell division over time, potentially leading to missing and inaccurate information. Summary of the Invention

[0008] The present invention proposes a low-density-high-density based dual-branch cell division process classification system to solve the technical problem of inaccurate prediction of cell division mode due to information loss caused by sparse video frames.

[0009] To solve the above technical problems, the present invention provides a low-density-high-density dual-branch cell division process classification system, which includes: a cell division degree classification module, a video interpolation module, a feature extraction module and a classification prediction module;

[0010] The cell division degree classification module is used to analyze two adjacent embryo images in the embryo image sequence to obtain the degree of division change intensity;

[0011] The video frame insertion module is used to insert frames into the original low-density real embryo image set to obtain a high-density embryo image set with high information density; when inserting frames, different numbers of frames are inserted between two adjacent embryo images according to the degree of cleavage change;

[0012] The feature extraction module is used to extract features of the low-density real embryo image set and the high-density embryo image set respectively and fuse them;

[0013] The classification prediction module is used to input the fused features into the fully connected layer for prediction to obtain the classification results.

[0014] Preferably, the feature extraction module includes a first feature extraction module and a second feature extraction module;

[0015] The first feature extraction module is used to extract spatial features of the low-density real embryo image set;

[0016] The second feature extraction module is used to extract spatial features of the high-density embryo image set.

[0017] Preferably, the channel capacity of the second feature extraction module is smaller than the channel capacity of the first feature extraction module.

[0018] Preferably, the channel capacity of the second feature extraction module is one quarter of the channel capacity of the first feature extraction module.

[0019] Preferably, the cell division state classification module obtains a degree of severity of division changes, which is divided into three levels: level one severity, level two severity, and level three severity.

[0020] Preferably, when the intensity level of the splitting change between two adjacent embryo images in the original low-density real embryo image set is level one, the video frame insertion module inserts one;

[0021] When the severity level of the splitting change between two adjacent embryo images in the original low-density real embryo image set is level 2, the video frame insertion module inserts three images;

[0022] When the severity level of the splitting change between two adjacent embryo images in the original low-density real embryo image set is level three, the video frame insertion module inserts seven images.

[0023] Preferably, the cell division degree classification module extracts feature vectors of two adjacent embryo images respectively by two weight-sharing CNN branches; performs feature fusion by direct splicing; performs training and judgment on the fused feature vectors using a fully connected layer; judges the effect of the classification network by calculating the cross entropy loss; and repeats the training to obtain a trained cell division degree classification module.

[0024] The beneficial effects of the present invention include at least:

[0025] The low-density embryo image set is based on real-world video footage. A drawback is that the dataset has a large time interval, which results in a loss of information about cell movement and the detailed process of cell division. This can also cause images of different cell division patterns to lose their division details due to the large time intervals between images. This can lead to similar images of different cell division patterns, leading to errors in the final determination of cell division patterns. The high-density dataset, on the other hand, consists of generated video frames inserted into the real dataset, and contains more details about cell division. However, the dataset also includes generated video frames, which can lead to errors that affect the results.

[0026] However, during cell division, not all images show significant changes between them. Most of the time, cells are not dividing, and the original image undergoes only minor movements, such as movement. In this case, performing multiple interpolation on all images before performing convolution to extract features can waste significant computing resources. To account for this, we pre-classify two temporally consecutive images, and vary the number of interpolated frames between them for different cell division stages.

[0027] The cell division state classification module of the present invention classifies the severity of cell division changes, inserts different numbers of frames according to the severity of division changes, and then extracts the features of the low-density real embryo image set and the high-density embryo image set respectively, effectively extracting the different features of cell division in the time dimension and spatial dimension, and fusing the features of the two different dimensions, thereby improving the classification accuracy of the cell division process while ensuring the amount of calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 Schematic diagram of the system structure of an embodiment of the present invention;

[0029] Figure 2 A comparison chart of abnormal division and normal division;

[0030] Figure 3 This is a comparison of mild and severe changes in cell division;

[0031] Figure 4 Schematic diagram of the three-level classification of different cell division intensity;

[0032] Figure 5 A network model for classifying the intensity of cell division;

[0033] Figure 6 This is the specific training process of the video frame insertion module. DETAILED DESCRIPTION

[0034] The following is a clear and complete description of the technical solutions in the embodiments of the present invention, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts are within the scope of protection of the present invention.

[0035] like Figure 2As shown, during cell division, the cell wall moves or splits over time. Different division modes can change over time, and their spatial manifestations can also vary significantly. Furthermore, cells can undergo drastic changes in a short period of time due to division, such as a single cell dividing into three. Equipment limitations can cause the captured images to lose many details of these changes. For example, the specific process of a cell dividing into three cells may not be captured, only the three cells that have completed division. In this case, the images of normal and abnormal divisions are highly similar, and direct judgment may lead to misjudgment, affecting the results. However, if new video frames are generated using simple interpolation and then fed into the calculation, the generated images may be inaccurate, causing interference from erroneous information and inaccurate prediction results.

[0036] And as Figure 3 As shown in the figure, cell division is not continuous. Most of the time, cells remain in their original state, undergoing only small movements. Conversely, cells rarely divide. In the current scenario, if multiple images are inserted between every two images in a continuous-time state, in most cases these inserted images provide no useful information for determining the cell division state; instead, they are merely redundant. Performing convolution after these insertions only increases the computational effort.

[0037] Therefore, an embodiment of the present invention provides a low-density-high-density based dual-branch cell division process classification system, which extracts the rules of change over time and space based on a large number of cell division pictures and uses the current rules to improve the accuracy of different division methods. At the same time, it makes up for the possibility that key cell division process information may be missed due to the long shooting interval, prevents the direct use of generated video frames resulting in inaccurate generation results, and may generate video frames with erroneous information, thereby making the prediction of cell division more accurate.

[0038] Specifically, if Figure 1 As shown, there are cell division degree classification module, video interpolation module, feature extraction module and classification prediction module.

[0039] The cell division degree classification module is used to analyze two adjacent embryo images in the embryo image sequence to obtain the degree of division change;

[0040] The video interpolation module is used to interpolate the original low-density real embryo image set to obtain a high-density embryo image set with high information density. During interpolation, different numbers of frames are inserted between two adjacent embryo images according to the degree of cleavage change.

[0041] A feature extraction module is used to extract features of the low-density real embryo image set and the high-density embryo image set respectively and fuse them;

[0042] The classification prediction module is used to input the fused features into the fully connected layer for prediction to obtain the classification results.

[0043] Particularly, the embodiment of the present invention is directed to the cell division video of shooting, and considering that the time interval of the original embryo image set is relatively large, the present embodiment adopts the process of shooting once every 15 minutes, and the original picture is processed as a low-density picture. For low-density picture, the feature extracted is the feature of real cell division, and the shortcoming is that the time interval is relatively large and may produce misjudgment because of missing specific division process. And in order to supplement low-density picture, adopt video frame insertion module in the present embodiment, the original picture is filled.

[0044] Since cells are not in a division state during most of the time during cell division, but in a small range of peristalsis, such as Figure 3 As shown in the figure, the cells in the first two video frames show little change, while the cell divisions in the last two frames are relatively obvious. Using the same interpolation strategy for all video frames would result in additional computational effort without any improvement in determining the cell division pattern. To address this issue, this embodiment adds a cell division state classification module before interpolating the cell division video frames to determine the degree of cell division between two consecutive video frames.

[0045] In this embodiment, the intensity of cell division between the two images is first manually determined. Figure 4 As shown, the intensity of cell division is divided into three levels. Level 1 is the lowest level of cell division, with no obvious changes in cell morphology, only small movements. Level 2 is defined as a certain degree of cell morphological change, such as the separation of two dividing cells, but no obvious change in the number of cells. Level 3 is a drastic change in morphology, such as a tendency for a single cell to divide into multiple cells. Although the cells are not completely separated, it is basically possible to judge that they are about to divide.

[0046] Specifically, the network structure of the cell division degree classification module in this embodiment is as follows: Figure 5 As shown in the figure, first, two consecutive video frames are input into two branches separately, and two corresponding feature vectors are extracted using two CNN branches with shared weights. Then, feature fusion is performed by direct splicing. The fused feature vectors are trained and judged using a fully connected layer. The effect of the classification network is judged by calculating the cross entropy loss. The training is repeated to obtain a trained classification network.

[0047] In this embodiment, the video interpolation module uses common video interpolation technologies, including but not limited to deep learning-based video interpolation technologies and motion estimation-based interpolation technologies. This embodiment uses deep learning-based video interpolation modules, such as FlowNetS, FlowNetC, and PWC-Net. FlowNetS is a classic convolutional neural network for optical flow estimation, using a simple encoder-decoder structure. It is primarily used to calculate the optical flow between two frames and predict the position of the intermediate frame by estimating the optical flow between frames.

[0048] If images are sampled at a high density, that is, the number of processed images is several times the number of low-density sampled images, this embodiment takes into account that three different levels of interpolation may result in different numbers of interpolated frames. To facilitate training as much as possible, different situations are chosen, such as inserting one, three, and seven images between two images. The extracted features include more division processes. Combining high-density cell division features with low-density cell division features can better compensate for the lack of detailed division process information caused by the large time interval between real images, thereby better determining different cell division types.

[0049] like Figure 6 As shown, in order for the video interpolation module to generate images that are closer to reality, it needs to be trained in advance. For training, a fixed camera is used to capture an image of a single well in a culture dish every 1 minute 52 seconds and 30 microseconds. After capturing a large number of images of cell division, a training set for the video interpolation module is formed. For the captured consecutive cell division frames t1, t2, t3, t4, t5, t6, t7, t8, and t9, for the first level of intensity, t5' is inserted between t1 and t9 and compared with the real t5 to calculate the L2 reconstruction loss. For the second level of intensity, t3', t5', and t7' are inserted between t1 and t9 and compared with the real t3, t5, and t7 to calculate the L2 reconstruction loss. For the third level of intensity, t2', t3', t4', t5', t6', t7', and t8' are inserted between t1 and t9 and compared with the real t2, t3, t4, t5, t6, t7, and t8 to calculate the L2 reconstruction loss.

[0050] Algorithms such as DAIN, Super SloMo and RIFE can be used for video interpolation. The SuperSloMo algorithm is used for illustration in this embodiment, but it is not a limitation of this embodiment. For example, for the second-level intensity, three pictures need to be inserted into two pictures. The continuous frames of cell division t1, t2, t3, t4, t5, t6, t7, t8, t9 are taken, and the real frames t1 and t9 taken are input. For the original time points of t3, t5, and t7, one frame is inserted every 3 minutes and 45 seconds to obtain t3', t5', and t7'. The interpolated t3', t5', and t7' are compared with the actual video frames t3, t5, and t7 to calculate the L2 reconstruction loss. Based on the loss function calculation results, the parameters of the model are optimized by back propagation. Continue training until the loss function converges, and it can be determined that the quality of the generated interpolated frames is stable.

[0051] For feature extraction, a VGG or ResNet network can be used. In this embodiment, a ResNet network is used for feature extraction. For feature fusion, direct feature concatenation or addition can be used. In this embodiment, direct concatenation is used. For the remaining primary and tertiary intensity levels, the same training and interpolation methods are used.

[0052] The low-density image is a real image. To ensure that the features extracted from different cell division modes are realistic and reliable, as the convolution proceeds, the features extracted from the low-density image are integrated with the features from the high-density image. The high-density image, on the other hand, extracts the temporal changes of cell division over a short period of time, encompassing more of the cell division process.

[0053] Considering that the high-density image as a whole extracts the temporal changes of cell division in a short period of time and pays less attention to spatial changes, the feature extraction modules in this embodiment use different network settings to extract features from low-density images and high-density images respectively. The spatial information of the low-density image is extracted by the first feature extraction module, and the temporal information of the high-density image is extracted by the second feature extraction module. In this embodiment, the channel capacity of the second feature extraction module is set to be smaller than the channel capacity of the first feature extraction module. In this way, the first feature extraction module pays more attention to the spatial changes of the low-density image, while the second feature extraction module pays more attention to the temporal changes of the high-density image.

[0054] The real low-density video frames are used as the main feature representation, and the high-density video frames that are mixed with the real and generated video frames are used as auxiliary features. They will be fused with the real features after each convolution to supplement the low-density image features. After the low-density and high-density features are fused, convolution will continue. The high-density image is only supplementary and is not the main convolution object. Convolution will continue on the basis of the high-density original features.

[0055] Specifically, this embodiment uses the ResNet network for illustration. To more carefully fuse features, feature fusion is performed after pooling and convolution operations.

[0056] In high-density video frames, because the number of frames is several times that of low-density video frames, direct convolution will generate features of different sizes, causing problems with feature fusion. To solve this problem, temporal pooling or sequential convolution and pooling can be used. This example uses temporal pooling for illustration.

[0057] By performing a pooling operation on the time dimension of each video, such as global average pooling or maximum pooling, this embodiment adopts a global average pooling method to normalize videos with different frame numbers into a feature representation of the same size as the features extracted from the low-density video frame.

[0058] After extracting features from high-density video frames, they are fed into low-density features for direct splicing and fusion. Then the low-density path continues to convolve the fused features, and the high-density path continues to convolve the high-density video frame features.

[0059] When the system of the embodiment of the present invention is used, two steps are required: constructing a data set and detecting the cell division type.

[0060] Construction of the data set: The present invention collects multiple cell division videos, extracts video frames at a certain step, and invites multiple cell experts to annotate each image, and uses the label with the most votes as the final label for the image. In order to further improve the diversity of cell position, morphology, etc., the present invention expands the image data set by flipping, rotating, etc. for the images in the training set to enhance the generalization of the model for cell center prediction. The data set is divided into a test set and a training set. For the labeled training set, a smaller number of photos are extracted at a certain step as a low-density image data set, and the original training set is used as a high-density image data set.

[0061] Cell division type detection is further divided into cell division feature extraction, feature fusion of different dimensions, feature output prediction and test set verification.

[0062] Specifically, we first perform preliminary feature extraction on both high-density and low-density image datasets to generate distinct features. The temporal and spatial features are then pooled within the network, and feature fusion is performed after the convolution phase. The resulting fused features are then fed into a fully connected layer for prediction, ultimately outputting a trained model.

[0063] The technical features of the above embodiments may be combined in any manner. To simplify the description, not all possible combinations of the technical features in the above embodiments are described. Only preferred embodiments of the present invention are presented. While the description is relatively specific and detailed, it should not be construed as limiting the scope of the present invention. As long as there are no contradictions in the combination of these technical features, they should be considered to be within the scope of this specification.

[0064] It should be noted that those skilled in the art may make various modifications and improvements without departing from the scope of the present invention, and these modifications and improvements fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be based on the appended claims.

Claims

1. A classification system for cell division processes based on low-density-high-density dual-branching, characterized by: The system includes: a cell division degree classification module, a video frame insertion module, a feature extraction module and a classification prediction module; The cell division degree classification module is used to analyze two adjacent embryo images in the embryo image sequence to obtain the degree of division change intensity; The video frame insertion module is used to insert frames into the original low-density real embryo image set to obtain a high-density embryo image set with high information density; when inserting frames, different numbers of frames are inserted between two adjacent embryo images according to the degree of cleavage change; The feature extraction module is used to extract features of the low-density real embryo image set and the high-density embryo image set respectively and fuse them; The classification prediction module is used to input the fused features into the fully connected layer for prediction to obtain the classification results; The cell division degree classification module uses two weight-sharing CNN branches to extract feature vectors of two adjacent embryo images respectively; uses a direct splicing method to fuse features; uses a fully connected layer to train and judge the fused feature vectors; judges the effectiveness of the classification network by calculating the cross entropy loss; and repeats the training to obtain a trained cell division degree classification module.

2. A classification system for cell division processes based on low-density-high-density dual-branching according to claim 1, characterized in that: The feature extraction module includes a first feature extraction module and a second feature extraction module; The first feature extraction module is used to extract spatial features of the low-density real embryo image set; The second feature extraction module is used to extract spatial features of the high-density embryo image set.

3. The low-density-high-density dual-branch cell division process classification system according to claim 2, characterized in that: The channel capacity of the second feature extraction module is smaller than the channel capacity of the first feature extraction module.

4. The low-density-high-density dual-branch cell division process classification system according to claim 3, characterized in that: The channel capacity of the second feature extraction module is one quarter of the channel capacity of the first feature extraction module.

5. The low-density-high-density dual-branch cell division process classification system according to claim 1, characterized in that: The cell division degree classification module divides the degree of division change into several levels. The higher the degree of division change, the more frames the video frame insertion module inserts between two adjacent embryo images.

6. The low-density-high-density dual-branch cell division process classification system according to claim 5, characterized in that: The cell division degree classification module obtains a degree of division change severity level that is divided into three levels: a first degree of division severity, a second degree of division severity, and a third degree of division severity.

7. The low-density-high-density dual-branch cell division process classification system according to claim 6, characterized in that: When the severity level of the splitting change between two adjacent embryo images in the original low-density real embryo image set is level one, the video frame insertion module inserts one; When the severity level of the splitting change between two adjacent embryo images in the original low-density real embryo image set is level 2, the video frame insertion module inserts three images; When the severity level of the splitting change between two adjacent embryo images in the original low-density real embryo image set is level three, the video frame insertion module inserts seven images.

Images