Cell division detection method, electronic device and storage medium

Abstract

The present invention provides a cell division detection method, electronic device and storage medium, constructing an embryo image sequence; dividing the embryo image sequence into a plurality of embryo image groups; dividing the first embryo image in each embryo image group into a plurality of first embryo image blocks of size k×k, where k represents the number of pixels; performing optical flow tracking on each pixel in each first embryo image block, and sequentially obtaining the second embryo image block with the most corresponding pixels in the three subsequent embryo images; the size of the second embryo image block is equal to the size of the first embryo image block; splicing each first embryo image block with the corresponding three second embryo image blocks to obtain a fused embryo image; extracting the features of the fused embryo image to perform cell division mode classification detection, and classifying it as normal division or reverse division. The increase in the amount of calculation caused by sending multiple images into convolution is reduced, while preserving spatial information and time information to the greatest extent.

Description

Technical Field

[0001] The present invention relates to the technical field of embryo image recognition, and in particular to a cell division detection method, electronic equipment and storage medium. Background Art

[0002] Cell division refers to the process of proliferation of living cells, that is, one cell divides into two cells. The cell before division is called the mother cell, and the new cell formed after division is called the daughter cell. In some cases, cells divide abnormally, that is, the number of daughter cells obtained from one cell division is greater than two. Another case is reverse division, that is, two cells merge into one cell. Detecting the state of cell division is helpful to understand the laws of cell growth and reproduction.

[0003] Cell division characteristics are a set of basic but important evaluation indicators for evaluating cell division modes. Cell division characteristics often reflect the differences in different cell division modes. By extracting the characteristics of different cell division modes, different cell division modes can be classified. The incubator can not only stably control the environment for cell tissue growth and provide suitable temperature and humidity. At the same time, it can also monitor cell growth, periodically take pictures of cell tissue growth, and upload them to the server. Cytologists can monitor cell growth modes and determine cell division modes based on periodic images of cell tissue development.

[0004] Under normal division, one cell will split into two cells, for example, a large cell will split into two slightly smaller cells, and the overall number of cells will increase from a small number to a large number. Reverse division is the fusion of two cells into one cell, and the fusion of two slightly smaller cells into a larger cell, and the overall number of cells will decrease from a large number to a small number. Because the two division modes are just different in the order of cells from a small number to a large number and from a large number to a small number, if the overall division mode is judged improperly based on the time information processing method, it may lead to misjudgment of the cell division mode and affect the results.

[0005] Traditional methods generally rely on manual judgment by experts, which consumes a lot of manpower and material resources, resulting in a waste of resources, and ultimately very low efficiency. With the development of artificial intelligence and computer vision technology, how to use artificial intelligence technology to realize automatic discrimination and monitoring of cell division in incubators has become an important issue.

[0006] Existing AI detection methods usually use:

[0007] 1) Identify the region of interest in the culture dish, and then make judgments based on the identified cell region. Use segmentation to process only the region of interest. It is necessary to divide the region of interest on the video frame in advance, which also wastes a lot of manpower and material resources and requires a large amount of initial investment;

[0008] 2) Directly send the entire video frame into the network, directly perform convolution for classification and judgment, and use deep learning methods to directly extract features from multiple video frames for processing. Multiple video frames will make the data dimension sent into the calculation higher, the amount of calculation will be larger, and more computing resources will be consumed.

[0009] In the detection process of the cell division pattern classification problem, a picture only shows the current state of the cell, and judging the cell division mode requires looking at the specific changes in the cell. A single picture cannot directly judge the specific movement process of cell division. Only based on multiple video frames of continuous cell division, the movement changes brought about by the specific division of cells over time can assist in judging the cell division mode. Therefore, in order to obtain the changes in cell division over time, it is necessary to combine the cell division time information and spatial information to perform specific spatiotemporal modeling. To obtain spatiotemporal information, features can be extracted from a single video frame to obtain spatial information, and then the features extracted from each frame can be regarded as time series data, and then the time series information can be extracted along the time dimension through the time series model. Doing so will result in the separation of spatiotemporal information, which will lead to information loss.

[0010] Alternatively, multiple video frames can be directly fed into a three-dimensional convolution to simultaneously calculate the temporal and spatial information of cell division. However, directly feeding multiple video frames into a deep learning network will result in a relatively high dimension of the video frame data that needs to be convolved, which will require a large amount of computation for convolution, wasting both computational resources and time resources. Summary of the invention

[0011] The present invention proposes a cell division detection method, an electronic device and a storage medium to solve the technical problem of how to reduce the increase in calculation amount caused by increasing the dimension while preserving the specific changes of cell division recorded by multiple consecutive embryo images.

[0012] In order to solve the above technical problems, the present invention provides a method for detecting cell division, comprising the following steps:

[0013] Step S1: taking embryo images at time intervals to construct an embryo image sequence;

[0014] Step S2: dividing the embryo image sequence into a plurality of embryo image groups according to the shooting time sequence, each of the embryo image groups including four embryo images;

[0015] Step S3: dividing the first embryo image in each embryo image group into a plurality of first embryo image blocks of size k×k, where k represents the number of pixels;

[0016] Step S4: performing optical flow tracing on each pixel in each of the first embryo image blocks, and sequentially obtaining the second embryo image block with the most corresponding pixels in the subsequent three embryo images; the size of the second embryo image block is equal to the size of the first embryo image block;

[0017] Step S5: splicing each of the first embryo image blocks and the corresponding three second embryo image blocks to obtain a fused embryo image;

[0018] Step S6: extracting features of the fused embryo image to perform cell division pattern classification detection and classify it into normal division or reverse division.

[0019] Preferably, step S4 performs optical flow tracking by using Horn-Schunck optical flow method or Farnebäck optical flow method or PWC-Net.

[0020] Preferably, in step S4, after performing optical flow tracing, remote pixel points that have been tracked are deleted by using a density clustering algorithm.

[0021] Preferably, step S4 comprises:

[0022] Step S41: Perform optical flow tracking to find the minimum boundary coordinates of the tracked pixel set , calculate the width W and height H of the enclosing rectangle:

[0023] ;

[0024] ;

[0025] Take the larger value of the width and height as the side length L of the minimum circumscribed outer package, fill symmetrically on the shorter side, and determine the new boundary coordinates;

[0026] Step S42: scaling the minimum bounding rectangle to restore it to a size of k×k to obtain a second embryo image block.

[0027] Preferably, step S42 includes: calculating the scale factors in the horizontal direction and the vertical direction and For each pixel in the first embryo image block, i.e. (x', y'), the scale factor is used to calculate its floating point coordinate in the minimum bounding rectangle image. and ; Bilinear interpolation uses the weighted average of the four adjacent pixel values ​​in the minimum bounding rectangle image to calculate the specific pixel value of each (x', y'); each pixel calculates its corresponding pixel in the original image through the above steps, and uses the interpolation algorithm to obtain the pixel value, and finally generates the second embryo image block.

[0028] Preferably, step S5 includes: splicing the first embryo image block and the corresponding three second embryo image blocks into a 2x2 small block, wherein the position (1,1) is the first embryo image block, the position (1,2) is the second embryo image block of the second embryo image, the position (2,1) is the second embryo image block of the third embryo image, and the position (2,2) is the second embryo image block of the fourth embryo image.

[0029] Preferably, the method for extracting features of the fused embryo image in step S6 includes: after position encoding the fused embryo image, inputting a self-attention mechanism to extract features including time information of the specific process of cell changes over time and overall cell spatial information.

[0030] Preferably, the method for performing cell division pattern classification detection in step S6 includes: inputting the extracted features into a fully connected layer, and calculating the classification cross entropy loss of the output result to obtain the cell division pattern classification.

[0031] The present invention also provides an electronic device, comprising: a memory, a processor and a computer program, wherein the computer program is stored in the memory and is configured to be executed by the processor to implement the above method.

[0032] The present invention also provides a computer-readable storage medium, in which a computer program is stored. The computer program is executed by a processor to implement the above method.

[0033] The beneficial effects of the present invention include at least: the present invention performs optical flow tracking on the interval-shot embryo images to retain the time information of the embryo image sequence, and then fuses multiple embryo images into one embryo image to retain the spatial information of the embryo image sequence. While being able to accurately extract the time and space information of cell division, it avoids the information loss problem that may be caused by calculating the time and space information separately, and prevents the construction of an overly complex model architecture that increases the complexity of the calculation; it also avoids the problem of directly sending multiple video frames into three-dimensional convolution, resulting in a waste of resources due to increased storage and calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A schematic diagram of the structure of the method flow of an embodiment of the present invention;

[0035] Figure 2 A schematic diagram of image fusion performed by an image fusion module according to an embodiment of the present invention;

[0036] Figure 3 A schematic diagram of a flow chart of an embodiment of the present invention;

[0037] Figure 4Schematic diagram of cell reverse division and normal division. DETAILED DESCRIPTION

[0038] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention.

[0039] The cell division process requires observation of multiple continuous video frames to better observe the dynamic changes of cells. Processing the classification and judgment of cell division patterns requires multiple continuous embryo images, which will contain the specific process of cell division, that is, time information. How to reduce the sudden increase in the amount of calculation caused by feeding high-dimensional data into convolution while correctly preserving time information and improving the accuracy of cell division pattern classification is a key technical problem that the present invention needs to solve.

[0040] like Figure 1 As shown, an embodiment of the present invention provides a method for detecting cell division, including: Step S1: taking embryo images at time intervals to construct an embryo image sequence.

[0041] Specifically, by adjusting the parameters of the time-difference incubator, an embryo image can be taken every 15 minutes to capture the temporal information of cell changes during embryonic development. Optical flow tracking technology is used to track the specific changes of cells over time and extract the temporal changes and trends of cells in video frames.

[0042] Step S2: Divide the embryo image sequence into a number of embryo image groups according to the shooting time sequence, each embryo image group includes four embryo images.

[0043] Specifically, the embryo image sequence captured above is arranged in the order of shooting time, with every four embryo images being regarded as an embryo image group.

[0044] Step S3: Divide the first embryo image in each embryo image group into a plurality of first embryo image blocks of k×k size, where k represents the number of pixels; in the embodiment of the present invention, k is set to 10.

[0045] Step S4: Perform optical flow tracing on each pixel in each first embryo image block to obtain the second embryo image block with the most corresponding pixels in the subsequent three embryo images; the size of the second embryo image block is equal to that of the first embryo image block.

[0046] Specifically, for the optical flow tracking algorithm, the Horn-Schunck optical flow method, the Farnebäck optical flow method, and the PWC-Net method can be used. Considering that the shooting light is illuminated by indoor lighting, the light will not change, and it can be assumed that the light is almost unchanged. Therefore, the embodiment of the present invention uses the Horn-Schunck optical flow method for description to estimate the motion of each pixel in the image.

[0047] Select the three consecutive embryo images corresponding to the first embryo image in chronological order, perform optical flow tracking on all pixels in the first embryo image block after the first embryo image is divided, and calculate the results of the Horn-Schunck optical flow tracking algorithm to obtain the tracked positions of all pixels in each first embryo image block. Since cell division is a dynamic process, the pixels in each small block are definitely not the original small block position and size in the next small block. In order to accurately track, the subsequent three embryo images need to be re-divided into multiple small blocks, and the small block with the most pixels representing all the pixels in the original small block is selected as the second embryo image block.

[0048] For each first embryo image block in the first embryo image, the above method is used to obtain the corresponding second embryo image blocks in the second, third and fourth embryo images.

[0049] In order to further improve the matching speed, the following method is used in the embodiment of the present invention to screen the second embryo image block.

[0050] Find the minimum bounding coordinates of a set of optical flow traced pixels , calculate the width W and height H of the enclosing rectangle.

[0051] ;

[0052] ;

[0053] Take the larger value of the width and height as the side length L of the outer rectangle. Fill symmetrically on the shorter side, for example Front and Fill in the back Units. Ensure that the final enclosing area is a square, determine the new boundary coordinates of the minimum enclosing rectangle, and output the result.

[0054] Then, in order to restore the original k×k size, the minimum enclosing rectangle is directly scaled: the horizontal and vertical scaling factors are calculated and , for each pixel of the target image k×k, that is, (x', y'), use the scale factor to calculate its floating point coordinates in the minimum bounding rectangle image and Bilinear interpolation uses the weighted average of the four adjacent pixel values ​​in the minimum bounding rectangle image to calculate the specific pixel value of each (x', y'). The corresponding pixel in the original image is calculated through the above steps, and the pixel value is obtained using the interpolation algorithm to finally generate the target image. After completing the traversal of all pixel points, the scaled second embryo image is obtained.

[0055] Exemplarily, in order to avoid misjudgment in pixel tracking, the embodiment of the present invention also proposes the following method for processing. Since pixel tracking may have misjudgment, resulting in the tracking of wrong pixels, these erroneous pixels may be far away from the real pixels, resulting in a relatively large minimum bounding rectangle. In order to compensate, an algorithm is used to delete a part of pixels that are too remote to eliminate possible erroneous options. The tracked pixels are divided into different clusters using the density clustering algorithm DBSCAN. DBSCAN can identify areas with denser pixel distribution while ignoring scattered noise points. The clustering results will locate the areas where the tracking pixels are concentrated. Calculate the centroid of the densest area of ​​pixel points in the minimum bounding rectangle of the pixels in the largest cluster. Delete the 5% of pixels farthest from the centroid, and then calculate the minimum bounding rectangle of the remaining 95% of pixels.

[0056] Step S5: each first embryo image block is spliced ​​with the corresponding three second embryo image blocks to obtain a fused embryo image.

[0057] The time-difference incubator periodically captures cell division images to record the changes in the state of cells during division. The changes in the cell division state lead to drastic changes in the cell division state in the image. This change can assist in the problem of cell division judgment, but the direct calculation of multiple embryo images increases the complexity of the calculation. Therefore, using image processing technology to process multiple embryo images to reduce dimensionality and reduce calculation complexity is a key problem that the present invention needs to solve.

[0058] Therefore, in the embodiment of the present invention, for the first embryo image block and the corresponding second embryo image block that have been previously cut and obtained according to optical flow tracking, according to the following Figure 2 In the manner shown, the four blocks are assembled into a 2×2 small block. According to the order of the first embryo image blocks, the new fusion block is obtained by arranging them in sequence. Figure 4As shown in the figure, in normal division, one cell will split into two cells, for example, a large cell will split into two slightly smaller cells, and the overall number of cells will increase from a small number to a large number. Reverse division is the fusion of two cells into one cell, and the fusion of two slightly smaller cells into a larger cell, and the overall number of cells will decrease from a large number to a small number. Because the two division modes are just different in the order of cells from a small number to a large number and from a large number to a small number, if the overall division mode is judged improperly based on the time information processing method, it may lead to misjudgment of the cell division mode and affect the results.

[0059] In order to avoid confusion of time information and problems in judging reverse division and normal division, small blocks are filled in a certain order to retain time information as much as possible. Specifically, for the thirteenth first embryo image block A in the first embryo image, the second embryo image blocks corresponding to the subsequent three embryo images obtained by it are B, C and D, respectively, which are spliced ​​into the thirteenth 2×2 small block of the fused embryo image, with A spliced ​​at the coordinate position (5, 5), B at the coordinate position (5, 6), C at the coordinate position (6, 5), and D at the left position (6, 6).

[0060] Step S6: extracting features of the fused embryo image to perform cell division pattern classification detection.

[0061] Specifically, artificial intelligence technology is used to extract the temporal and spatial information of the fused embryo image, and a subsequent network is needed to better extract relevant information. The universal target detection network of the present invention is used as a network framework, including transformer, VGG16, ResNet, etc. The embodiment of the present invention is described by taking transformer as an example.

[0062] The fused cell division image is input into the transformer network for feature extraction. This can effectively extract the temporal and spatial features of a picture line of sight. The transformer is mainly composed of two network parts, namely the encoder and the decoder. The embodiment of the present invention mainly uses the encoder, which specifically includes two steps: encoder and position encoding.

[0063] 1) Encoder: The encoder is mainly used to extract features. The encoder uses a multi-head self-attention mechanism to divide the input image into small blocks for processing, divide the input sequence into multiple "heads" for parallel processing, each head focuses on a different subspace, and then splices these results together as the final output.

[0064] 2) Position encoding: Since the input is an image, the image processing needs to be position encoded first. Position encoding is achieved by inputting the position information into the network as part of the word vector.

[0065] The classification module is used to classify the embryo image into normal cleavage or reverse cleavage according to the characteristics of the embryo image.

[0066] When the method of the embodiment of the present invention is implemented, Figure 3 shown.

[0067] 1) Construction of standard database

[0068] The present invention collected multiple cell division images, and invited multiple cell experts to annotate each embryo image, and used the image with the most votes as the final label of the image. In order to further improve the diversity of cell position and morphology, the present invention expanded the image data set by flipping and rotating the images in the training set to enhance the generalization of the model for cell center prediction.

[0069] 2) Cell division pattern classification detection

[0070] Divide all the first embryo images in each embryo image group into k×k first embryo image blocks, perform optical flow tracing of the next three embryo images for all pixels in each first embryo image block of the first embryo image, and perform second embryo image blocks corresponding to the second, third, and fourth embryo images of each first embryo image block in the first embryo image according to the method of the largest absolute proportion;

[0071] According to the second embryo image block corresponding to the first embryo image block predicted by the model in the first step, the first embryo image blocks in the first embryo image are sequentially spliced ​​into a new fused embryo image in the order of the first embryo image blocks, and the original k×k embryo image is expanded into a 2k×2k image;

[0072] The fused embryo image is first position-encoded. The encoded image is input into the subsequent self-attention to extract features containing temporal and spatial information;

[0073] The results extracted from the comparisons are input into the fully connected layer, and the output results are used to calculate the classification cross entropy loss to obtain the cell division mode classification detection results, which are classified as normal division or reverse division.

[0074] An embodiment of the present invention further provides an electronic device, comprising: a memory, a processor and a computer program, wherein the computer program is stored in the memory and is configured to be executed by the processor to implement the above method.

[0075] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored. The computer program is executed by a processor to implement the above method.

[0076] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. Only the preferred embodiments of the present invention are expressed. The description is more specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. As long as there is no contradiction in the combination of these technical features, they should be considered as the scope recorded in this specification.

[0077] It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these modifications and improvements all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A method for detecting cell division, characterized in that: The following steps are involved: Step S1: taking embryo images at time intervals to construct an embryo image sequence; Step S2: dividing the embryo image sequence into a plurality of embryo image groups according to the shooting time sequence, each of the embryo image groups including four embryo images; Step S3: dividing the first embryo image in each embryo image group into a plurality of first embryo image blocks of size k×k, where k represents the number of pixels; Step S4: performing optical flow tracing on each pixel in each of the first embryo image blocks, and sequentially obtaining the second embryo image block with the most corresponding pixels in the subsequent three embryo images; the size of the second embryo image block is equal to the size of the first embryo image block; Step S5: splicing each of the first embryo image blocks and the corresponding three second embryo image blocks to obtain a fused embryo image; Step S6: extracting features of the fused embryo image to perform cell division pattern classification detection, and classifying it as normal division or reverse division; The method for extracting features of the fused embryo image in step S6 includes: after position encoding the fused embryo image, inputting the self-attention mechanism to extract features including time information of the specific process of cell changes over time and overall cell spatial information.

2. A cell division detection method according to claim 1, characterized in that: Step S4 performs optical flow tracking by using the Horn-Schunck optical flow method or the Farnebäck optical flow method or the PWC-Net.

3. A cell division detection method according to claim 1, characterized in that: In step S4, after performing optical flow tracing, remote pixel points that have been tracked are deleted by using a density clustering algorithm.

4. A cell division detection method according to claim 1, characterized in that: Step S4 include: Step S41: Perform optical flow tracking to find the minimum boundary coordinates of the tracked pixel set , calculate the width W and height H of the enclosing rectangle: ； ； Take the larger value of the width and height as the side length L of the minimum enclosing rectangle, fill symmetrically on the shorter side, and determine the new boundary coordinates; Step S42: scaling the minimum bounding rectangle to restore it to a size of k×k to obtain a second embryo image block.

5. A cell division detection method according to claim 4, characterized in that: Step S42 includes: calculating the scale factors in the horizontal direction and the vertical direction and For each pixel in the first embryo image block, i.e. (x', y'), the scale factor is used to calculate its floating point coordinate in the minimum bounding rectangle image. and ; Bilinear interpolation uses the weighted average of the four adjacent pixel values ​​in the minimum bounding rectangle image to calculate the specific pixel value of each (x', y'); each pixel calculates its corresponding pixel in the original image through the above steps, and uses the interpolation algorithm to obtain the pixel value, and finally generates the second embryo image block.

6. A cell division detection method according to claim 1, characterized in that: Step S5 includes: splicing the first embryo image block and the corresponding three second embryo image blocks into a 2x2 small block, wherein the position (1,1) is the first embryo image block, the position (1,2) is the second embryo image block of the second embryo image, the position (2,1) is the second embryo image block of the third embryo image, and the position (2,2) is the second embryo image block of the fourth embryo image.

7. A cell division detection method according to claim 1, characterized in that: The method for performing cell division pattern classification detection in step S6 includes: inputting the extracted features into a fully connected layer, and calculating the classification cross entropy loss of the output result to obtain the cell division pattern classification.

8. An electronic device comprising: A memory, a processor and a computer program, characterized in that: the computer program is stored in the memory and is configured to be executed by the processor to implement the method described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 7.

Images