Double-branch cell division process classification system based on low density and high density

Abstract

The invention provides a double-branch cell division process classification system based on low density and high density. The system comprises a cell division degree classification module, a video frame insertion module, a feature extraction module and a classification prediction module. The cell division degree classification module is used for obtaining a division change intensity degree grade; the video frame insertion module is used for performing frame insertion on the original low-density real embryo image set to obtain a high-density embryo image set with high information density; when frame insertion is carried out, different frame numbers are inserted between two adjacent embryo images according to fission change intensity levels; the feature extraction module is used for respectively extracting features of the low-density real embryo image set and the high-density embryo image set and fusing the features; and the classification prediction module is used for inputting the fused features into a full connection layer for prediction to obtain a classification result. And laws changing along with time and space are extracted, and the classification accuracy of different splitting modes is improved by using the current laws.

Description

Technical Field

[0001] The present invention relates to the technical field of embryo division recognition, and particularly relates to a dual-branch cell division process classification system based on low density-high density. Background Art

[0002] Cell division refers to the process of live cell proliferation, that is, one cell divides into two cells. The cell before division is called the mother cell, and the newly formed cells after division are called daughter cells. In some cases, cells will undergo abnormal division, that is, the number of daughter cells obtained from the division of one cell is greater than two. Another situation is reverse division, that is, two cells merge into one cell. Detecting the cell division state is beneficial to understanding the laws of cell growth and reproduction.

[0003] Cell division detection is an important step in evaluating whether cell division is normal. The change of the cell wall can often reflect whether the cell division is normal. By comparing the number of cells before and after, the cell division mode can be reflected, so as to realize cell division detection. The incubator can not only provide a stable in vitro culture environment for cells, but also continuously obtain the whole process images of cell division periodically. Combining the cell photographing time recorded in the incubator, cytologists need to discriminate each cell image according to their own cell evaluation experience to obtain the cell division mode, which greatly increases the workload of cytologists. Therefore, how to assist cytologists in quickly detecting the cell division direction by means of computer vision methods has very important research significance.

[0004] In actual shooting detection, for the problem of cell division mode detection, the culture dish generally adopts a way of arranging 8 holes in a row, usually 2 rows of holes in one dish, and a total of 16 embryos are placed. During the photographing process of the time-lapse incubator, the drive shaft will move, move above each hole position, then the control point light source emits light, and then photographing is carried out. Considering the three-dimensionality and size of the embryo, multiple images will be taken from top to bottom. After shooting, the shaft moves to the next hole and takes pictures in turn. When all 16 embryos in the culture dish have been photographed, it will return to the first hole to continue shooting. According to the set photographing time interval of the device, it is ensured that the machine will take pictures again within the specified time.

[0005] If the equipment problem is not considered and photos are directly taken continuously and quickly for the same cell, the shooting will be too frequent, resulting in too long exposure time of the cell, causing damage to the cell and making it impossible to continue shooting.

[0006] Due to the cyclic shooting of cell division photos, the time interval between shots of the same cell is relatively large. In the case of drastic cell changes, motion information may be lost, and the specific division process may be missed. For example, in abnormal division, one cell does not divide while another cell divides into three, and finally the shooting result is the same as that of normal division of two cells into four cells, resulting in problems in judging the cell division method. Even if the intermediate division process is not missed, due to the large time gap, the video frames are not smooth enough.

[0007] Existing intelligent recognition and detection methods process existing pictures without considering the problem of large time intervals between photos taken due to equipment problems, and directly process relatively sparse video frames. It does not consider that the information content of relatively sparse video frames is relatively small and cannot well record the specific situation of cell division over time, which may lead to information loss and inaccuracy. Summary of the Invention

[0008] The present invention proposes a dual-branch cell division process classification system based on low density-high density to solve the technical problem of inaccurate prediction of cell division methods caused by information loss due to sparse video frames.

[0009] To solve the above technical problems, the present invention provides a dual-branch cell division process classification system based on low density-high density. The system includes: a cell division degree classification module, a video frame interpolation module, a feature extraction module, and a classification prediction module; The cell division degree classification module is used to analyze two adjacent embryo images in an embryo image sequence to obtain the level of drastic division change; The video frame interpolation module is used to interpolate the original low-density real embryo image set to obtain a high-density embryo image set with high information density; when interpolating, different numbers of frames are inserted between two adjacent embryo images according to the level of drastic division change; The feature extraction module is used to extract the features of the low-density real embryo image set and the high-density embryo image set respectively and fuse them; The classification prediction module is used to input the fused features into a fully connected layer for prediction to obtain a classification result.

[0010] Preferably, the feature extraction module includes a first feature extraction module and a second feature extraction module; The first feature extraction module is used to extract the spatial features of the low-density real embryo image set; The second feature extraction module is used to extract the spatial features of the high-density embryo image set.

[0011] Preferably, the channel capacity of the second feature extraction module is less than that of the first feature extraction module.

[0012] Preferably, the channel capacity of the second feature extraction module is one-fourth of that of the first feature extraction module.

[0013] Preferably, the severity levels of the division changes obtained by the cell division state classification module are divided into three levels: level one severity, level two severity, and level three severity.

[0014] Preferably, when the severity level of the division change between two adjacent embryo images in the original low-density real embryo image set is level one severity, the video frame interpolation module inserts one; When the severity level of the division change between two adjacent embryo images in the original low-density real embryo image set is level two severity, the video frame interpolation module inserts three; When the severity level of the division change between two adjacent embryo images in the original low-density real embryo image set is level three severity, the video frame interpolation module inserts seven.

[0015] Preferably, the cell division degree classification module respectively extracts the feature vectors of two adjacent embryo images by two CNN branches with shared weights; performs feature fusion in a direct splicing manner; uses a fully connected layer to train and judge the fused feature vectors; calculates the cross-entropy loss to judge the effect of the classification network; repeats the training to obtain a trained cell division degree classification module.

[0016] The beneficial effects of the present invention at least include: The low-density embryo image set is pictures taken by real cameras. The disadvantage is that the time interval of the data set is relatively large, resulting in the loss of cell movement information, the lack of the specific process of cell division, and the possible loss of the division process of the photos of different cell division methods due to the large time interval. The photos of different division methods are relatively similar, resulting in errors in the final judgment of cell division methods. The high-density data set inserts generated video frames into the real data set, which will contain more details of cell division. The disadvantage is that errors may occur in the generated video frames in the data set, and the generated video frames may be incorrect, affecting the results.

[0017] During the process of cell division, not all cell division processes show significant changes between two pictures. Most of the time during cell division, the cells are not in the division phase, and only small-scale movements such as the movement of the original pictures occur. In this case, if multiple pictures are interpolated for all pictures and then convolutional feature extraction is performed, a large amount of computing resources may be wasted. Considering this situation, two consecutive pictures in time are classified in advance, and the number of interpolated frames between the two pictures varies according to different cell division states.

[0018] After the cell division state classification module of the present invention classifies the severity of cell division changes, different numbers of frames are inserted according to the severity of the division changes, and then the features of the low-density real embryo image set and the high-density embryo image set are extracted respectively. The features of cell division in different time and space dimensions are effectively extracted, and the features of the two different dimensions are fused, while ensuring the amount of calculation, improving the classification accuracy of the cell division process. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic diagram of the system structure of an embodiment of the present invention; Figure 2 It is a comparison diagram of abnormal division and normal division; Figure 3 It is a comparison diagram of slight and severe cell division changes; Figure 4 It is a schematic diagram of the three-level classification of different cell division severities; Figure 5 It is a cell division severity classification network model; Figure 6 It is the specific training process of the video interpolation module. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0020] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0021] Such as Figure 2As shown in the figure, during the process of cell division, the cell wall will move or divide over time. Different division methods will have different changes over time, and there will also be significant differences in their spatial manifestations. At the same time, cells may undergo drastic changes due to division and other reasons in a short period of time. For example, a cell divides into three cells in a short period of time. Due to equipment problems, many details of cell changes may be lost in the captured images. For example, the specific process of a cell dividing into three cells is not captured, and only the finally divided three cells are captured. In this case, the images of normal division and abnormal division are highly similar, and direct judgment may lead to misjudgment and affect the results. However, if the method of interpolation is directly and simply used to generate new video frames and then fed into the operation, since the generated images may be inaccurate, it will cause interference of incorrect information and make the prediction results inaccurate.

[0022] And as Figure 3 shown, the process of cell division is not continuous. Most of the time, the cells are not dividing and remain in their original state, only making small-scale displacement movements. On the contrary, the situation where the cells are in the dividing state is relatively rare. In the current situation, if multiple images are to be inserted between every two images in a continuous time state, in most cases, the inserted images do not provide useful information for judging the cell division state. On the contrary, they are just redundant information. Performing convolution after insertion will only increase the computational complexity.

[0023] Therefore, the embodiment of the present invention provides a dual-branch cell division process classification system based on low density-high density, extracts the laws of changes over time and space based on a large number of cell division images, and uses the current laws to improve the accuracy of different division methods. At the same time, it makes up for the possible omission of key process information of cell division due to too long shooting intervals, and prevents the inaccurate generation results caused by directly using the generated video frames, which may generate video frames with incorrect information, thereby making the prediction of cell division more accurate.

[0024] Specifically, as Figure 1 shown, a cell division degree classification module, a video interpolation module, a feature extraction module, and a classification prediction module.

[0025] The cell division degree classification module is used to analyze two adjacent embryo images in an embryo image sequence to obtain the level of the severity of division changes; The video interpolation module is used to perform interpolation on the original low-density real embryo image set to obtain a high-density embryo image set with high information density; when performing interpolation, different numbers of frames are inserted between two adjacent embryo images according to the level of the severity of division changes; The feature extraction module is used to extract the features of the low-density real embryo image set and the high-density embryo image set respectively and fuse them; The classification prediction module is used to input the fused features into the fully connected layer for prediction to obtain the classification results.

[0026] Specifically, the embodiment of the present invention is directed to the cell division video taken, considering that the time interval of the original embryo image set is relatively large, the present embodiment adopts the process of taking a shot every 15 minutes, and the original picture is processed as a low-density picture. For low-density pictures, the extracted features are the features of real cell division, and the shortcoming is that the time interval is relatively large and may cause misjudgment due to missing the specific division process. And in order to supplement the low-density pictures, the video frame insertion module is adopted in the present embodiment to fill the original picture.

[0027] Since cells are not in a division state during most of the cell division process, but in a small range of peristalsis, such as Figure 3 As shown in the figure, the cell changes in the first two video frames are not significant, while the cell divisions in the last two frames are relatively obvious. If the same interpolation strategy is used for all video frames, it will increase the amount of additional calculations without any help in judging the cell division mode. Considering this problem, this embodiment adds a cell division state classification module before interpolating the cell division video frame to judge the degree of cell division between two consecutive video frames.

[0028] In this embodiment, the intensity of cell division between the two images is first manually determined. Figure 4 As shown, the intensity of cell division is divided into three levels. The first level is the lowest level of cell division change. The cell morphology does not change significantly, and there is only a small range of peristalsis. The second level is defined as a certain degree of cell morphological change, such as the separation of two cells, etc., without obvious changes in quantity. The third level is a drastic change in morphology, such as a tendency for one cell to divide into multiple cells. Although they are not completely separated, it can be basically determined that they are about to divide.

[0029] Specifically, the network structure of the cell division degree classification module in this embodiment is as follows: Figure 5 As shown in the figure, first, two consecutive video frames are input into two branches separately, and two corresponding feature vectors are extracted using two CNN branches with shared weights. Then, feature fusion is performed by direct splicing. The fused feature vectors are trained and judged using a fully connected layer. The effect of the classification network is judged by calculating the cross entropy loss. The training is repeated to obtain a trained classification network.

[0030] In this embodiment, the video frame interpolation module adopts common video frame interpolation technologies, including but not limited to video frame interpolation technologies based on deep learning and frame interpolation technologies based on motion estimation. In this embodiment, a video frame interpolation module based on deep learning is used for processing, such as FlowNetS, FlowNetC, PWC-Net, etc. FlowNetS is a classic convolutional neural network for optical flow estimation, adopting a simple encoding-decoding structure. It is mainly used to calculate the optical flow between two frames, and predict the position of the intermediate frame by estimating the optical flow between frames.

[0031] If high-density pictures are sampled, that is, the number of pictures to be processed is several times that of low-density sampled pictures. Considering three different levels of frame interpolation in this embodiment, the number of interpolated frames will be different. To facilitate training as much as possible, different situations of inserting one, three, and seven pictures between two pictures are selected. The extracted features contain more splitting processes. Combining high-density cell division features and low-density cell division features can better make up for the problem of incomplete splitting processes caused by too large a time interval between real pictures, so as to better judge the types of different cell divisions.

[0032] As Figure 6 shown, in order for the video frame interpolation module to better generate pictures close to reality, the video frame interpolation module needs to be pre-trained. For training, the camera is fixed, and a picture is taken every 1 minute, 52 seconds, and 30 microseconds for a single hole in the petri dish. After taking a large number of pictures of cell division, a training set for the video frame interpolation module is formed. The obtained consecutive frames of cell division t1, t2, t3, t4, t5, t6, t7, t8, t9. For the first-level severity, t5' needs to be inserted between t1 and t9, and compared with the real t5 to calculate the L2 reconstruction loss. For the second-level severity, t3', t5', and t7' need to be inserted between t1 and t9, and compared with the real t3, t5, and t7 to calculate the L2 reconstruction loss. For the third-level severity, t2', t3', t4', t5', t6', t7', and t8' need to be inserted between t1 and t9, and compared with the real t2, t3, t4, t5, t6, t7, and t8 to calculate the L2 reconstruction loss.

[0033] Algorithms such as DAIN, Super SloMo, and RIFE can be used for video frame interpolation. In this embodiment, the SuperSloMo algorithm is used for illustration, but it is not a limitation of this embodiment. For example, for the second level of intensity, three pictures need to be inserted between two pictures. The continuously captured frames of cell division t1, t2, t3, t4, t5, t6, t7, t8, t9 are input, and the real frames t1 and t9 are captured. For the original time points of t3, t5, t7, that is, one frame is inserted every 3 minutes and 45 seconds to obtain t3’, t5’, t7’. The interpolated t3’, t5’, t7’ are compared with the actual video frames t3, t5, t7, and the L2 reconstruction loss is calculated. Based on the calculation result of the loss function, the parameters of the model are optimized through backpropagation. Continue training until the loss function converges, and it can be determined that the quality of the generated interpolated frames is stable.

[0034] For the module for feature extraction, VGG, ResNet networks, etc. can be adopted. In this embodiment, the ResNet network is adopted for feature extraction. For feature fusion, direct feature splicing or addition can be used. In this embodiment, direct splicing is adopted. For the remaining first and third levels of intensity, the same training and frame interpolation methods are used.

[0035] The low-density picture is a real picture to ensure that the features extracted from different cell division methods are real and credible. As the convolution progresses, the features extracted from the low-density picture will fuse with the features of the high-density picture. The high-density picture as a whole extracts the temporal changes of cell division in a short period of time and will contain more cell division processes.

[0036] Considering that the high-density picture as a whole extracts the temporal changes of cell division in a short period of time and pays less attention to the spatial changes, in this embodiment, different network settings are adopted for the feature extraction module to extract features from the low-density picture and the high-density picture respectively. The spatial information of the low-density picture is extracted through the first feature extraction module, and the temporal information of the high-density picture is extracted through the second feature extraction module. In this embodiment, the channel capacity of the second feature extraction module is set to be less than that of the first feature extraction module. In this way, the first feature extraction module pays more attention to the spatial changes of the low-density picture, while the second feature extraction module pays more attention to the temporal changes of the high-density picture.

[0037] For the real low-density video frames, they are used as the main feature representation, while the high-density video frames that mix real and generated video frames, as auxiliary features, will be fused with the real features after each convolution to supplement the features of the low-density picture. After the features of the low-density and high-density are fused, convolution continues. Since the high-density picture only plays a supplementary role and is not the main convolution object, convolution continues on the basis of the original high-density features.

[0038] Specifically, in this embodiment, the ResNet network is used for illustration. For more careful feature fusion, feature fusion is performed after pooling and convolution operations.

[0039] In high-density video frames, since the number of frames is several times that of low-density video frames, direct convolution will result in different sizes of generated features, causing problems in feature fusion. To solve the current problem, temporal dimension pooling or temporal convolution and pooling operations can be used. In this embodiment, the temporal dimension pooling operation is used for illustration.

[0040] By performing a pooling operation on the temporal dimension of each video, such as global average pooling or max pooling, this embodiment uses the method of global average pooling to normalize videos with different numbers of frames into a feature representation of the same size as the features extracted from low-density video frames.

[0041] After the features are extracted from the high-density video frames, they are directly concatenated and fused with the low-density features, and then the low-density path continues to perform convolution on the fused features, while the high-density path continues to perform convolution on the high-density video frame features.

[0042] When the system of the embodiment of the present invention is used, it needs to go through two steps: the construction of the data set and the detection of cell division types.

[0043] Construction of the data set: The present invention collects multiple cell division videos, extracts video frames at a certain stride, and for each image, invites multiple cell experts to label it, and uses the label with the most votes as the final label of the image. To further improve the diversity of cell positions, morphologies, etc., the present invention uses methods such as flipping and rotating the images in the training set to expand the image data set, so as to enhance the generalization of the model for cell center prediction. The data set is divided into a test set and a training set. For the training set with already made labels, a smaller number of photos are extracted at a certain stride as the data set of low-density images, and the original training set is used as the data set of high-density images.

[0044] The detection of cell division types is further divided into cell division feature extraction, feature fusion in different dimensions, feature output prediction, and test set verification.

[0045] Specifically, first, the data sets of high-density images and low-density images are respectively subjected to preliminary feature extraction to form different features. For the features in the temporal dimension and spatial dimension, feature fusion is performed after the pooling and convolution stages in the network, and the obtained fused features are input into the fully connected layer for prediction, and a trained model is output.

[0046] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. Only the preferred embodiments of the present invention are expressed. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the patent of the present invention. As long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0047] It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several deformations and improvements can be made, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the appended claims.

Claims

1. A classification system for the process of cell division based on low-density-high-density dual-branching, characterized by: The system comprises: a cell division degree classification module, a video frame insertion module, a feature extraction module and a classification prediction module; The cell division degree classification module is used to analyze two adjacent embryo images in the embryo image sequence to obtain the degree of division change intensity; The video frame insertion module is used to insert frames into the original low-density real embryo image set to obtain a high-density embryo image set with high information density; when inserting frames, different numbers of frames are inserted between two adjacent embryo images according to the degree of cleavage change; The feature extraction module is used to extract the features of the low-density real embryo image set and the high-density embryo image set respectively and fuse them; The classification prediction module is used to input the fused features into the fully connected layer for prediction to obtain the classification result.

2. A classification system for the process of cell division based on low density and high density dual branching according to claim 1, characterized in that: The feature extraction module includes a first feature extraction module and a second feature extraction module; The first feature extraction module is used to extract the spatial features of the low-density real embryo image set; The second feature extraction module is used to extract the spatial features of the high-density embryo image set.

3. A classification system for cell division process based on low density and high density dual branching according to claim 2, characterized in that: The channel capacity of the second feature extraction module is smaller than the channel capacity of the first feature extraction module.

4. A classification system for the process of cell division based on low density and high density dual branching according to claim 3, characterized in that: The channel capacity of the second feature extraction module is one quarter of the channel capacity of the first feature extraction module.

5. A classification system for cell division process based on low density and high density dual branching according to claim 1, characterized in that: The cell division degree classification module classifies the division change intensity into several levels. The higher the division change intensity, the more frames the video frame insertion module inserts between two adjacent embryo images.

6. A classification system for the process of cell division based on low density and high density dual branching according to claim 5, characterized in that: The cell division degree classification module obtains three levels of division change severity: primary severity, secondary severity and tertiary severity.

7. A classification system for the process of cell division based on low density and high density dual branching according to claim 6, characterized in that: When the cleavage change severity level between two adjacent embryo images in the original low-density real embryo image set is level one, the video frame insertion module inserts one; When the cleavage change severity level between two adjacent embryo images in the original low-density real embryo image set is level 2, the video frame insertion module inserts three images; When the cleavage change severity level between two adjacent embryo images in the original low-density real embryo image set is level three, the video interpolation module inserts seven images.

8. The low-density-high-density dual-branch cell division process classification system according to claim 1, characterized in that: The cell division degree classification module extracts feature vectors of two adjacent embryo images respectively by two weight-sharing CNN branches; performs feature fusion by direct splicing; performs training and judgment on the fused feature vectors by using a fully connected layer; judges the effect of the classification network by calculating the cross entropy loss; and repeats the training to obtain a trained cell division degree classification module.

Images