A method for evaluating the number of pronuclei in pronuclear embryos
The number of prokaryotics in embryo images is recognized through convolutional neural network, which solves the problem of difficulty in manual labeling, and realizes automatic high-accuracy recognition of prokaryotic numbers, improving the efficiency of embryo evaluation.
Patent Information
- Application Number
- CN202011251367.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-11-11
- Publication Date
- 2024-12-24
- Estimated Expiration
- 2040-11-11
AI Technical Summary
In the prior art, it is necessary to manually mark the prokaryotic profile, which makes it difficult and time-consuming to judge the number of prokaryotics. Traditional methods consume a lot of time to analyze image data, affecting the quality of embryo evaluation.
The number of prokaryotics in the embryo image was identified by using convolutional neural network (CNN), and the number of prokaryotics was classified into 0PN, 1PN, 2PN and 3PN through the SE-ResNet-18 model, and the time series recognition results of the embryo sequence were used for judgment.
It realizes automatic identification of the number of prokaryotics, reduces the cost of manual annotation, improves accuracy and efficiency, and performs excellently in the accurate judgment of embryos of 3PN and above.
Smart Images
Figure CN112329664B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of embryo culture, and in particular relates to a method for evaluating the number of pronuclei of pronuclear embryos. Background Art
[0002] Embryo quality assessment is an important task of IVF-ET, which directly affects the embryo transfer rate and pregnancy rate. Morphological methods are widely used in embryo quality assessment, that is, in the process of in vitro culture, the key morphological characteristics of the embryo are observed to judge its developmental potential. The fixed-point observation method observes the embryo morphology at several specific time points, evaluates and grades the embryo quality based on morphology, and selects the embryo with the highest grade for transplantation. However, embryo development is a dynamic and continuous process, and observations at several specific time points cannot comprehensively evaluate the embryo quality. The real-time continuous observation method uses a time-difference imaging system, installs a built-in camera device in the incubator, and automatically photographs the embryo at certain intervals, realizing real-time and continuous observation of the embryo in a stable and controllable environment, and completing the dynamic evaluation of embryo quality.
[0003] Based on the morphological assessment of embryonic development potential, the embryonic development process is divided into three stages: the pronuclear stage, the cleavage stage, and the blastocyst stage. The appearance of the pronucleus is a sign of complete fertilization. 2PN indicates normal fertilization, 0PN and 3PN and above embryos are considered abnormal, and 1PN requires confirmation of whether the embryo is diploid. In particular, embryos of 3PN and above, with a probability of 5%-8%, are embryos that should never be transplanted and must be accurately identified. It is also difficult to use manual judgment to determine the number of pronuclei in pronuclear embryos. The pronuclei are small in the image and difficult to distinguish. There will be overlaps between the pronuclei, which is prone to counting errors. The time-difference imaging system generates a large amount of embryo image data and videos. The traditional manual method of analyzing image data consumes a lot of time, and the level and subjectivity of the analyst have different degrees of influence on the quality of embryo evaluation.
[0004] Recently, the application of deep learning algorithms to analyze and judge embryo development potential and clinical outcomes has become one of the research hotspots in the field of assisted reproduction.
[0005] The application of deep learning in prokaryotic period assessment mainly solves the problem of automatic identification of prokaryotic number. Fukunaga et al. used deep learning technology to detect the number of prokaryotic numbers. This method first detected the prokaryotic outline and then identified the number of prokaryotic numbers based on the prokaryotic outline. Both stages used CNN as the outline detector and prokaryotic number classifier. This method requires manual annotation of the prokaryotic outline and the number of prokaryotic numbers, which is very expensive. Summary of the invention
[0006] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a method for evaluating the number of pronuclei in pronuclear embryos, which solves the problem of manually marking the outline of the pronuclei in the prior art.
[0007] The purpose of this disclosure can be achieved through the following technical solutions:
[0008] A method for evaluating the number of pronuclei in pronuclear embryos, comprising the following steps:
[0009] S1. Acquire and / or identify each embryo in the embryo image;
[0010] S2. The number of pronuclei of each embryo in the embryo image is identified by convolutional neural network. The problem of pronuclei identification is treated as a classification problem, and the number of pronuclei of pronuclear embryos is divided into four categories: 0PN, 1PN, 2PN and 3PN, which respectively represent 0 pronuclei, 1 pronucleus, 2 pronuclei, 3 pronuclei and above;
[0011] S3. Determine the number of pronuclei of the embryo based on the result of identifying the number of pronuclei of the embryo corresponding to the pronuclear period time series of the embryo sequence after classification and identification.
[0012] In some disclosures, the acquiring and / or identifying each embryo in the embryo image comprises:
[0013] S1.1, acquiring and / or identifying the blurred embryo region of each embryo in the embryo image, and making a mask of the blurred embryo region for the blurred embryo region of each embryo;
[0014] S1.2. According to the mask of the blurred embryo area, use Gaussian filtering to smooth the image, then use Hough ring transform to find the embryo cell boundary, and based on the embryo cell boundary, make a mask of the effective embryo area to obtain the effective embryo area of each embryo, and use the effective embryo area as the corresponding embryo identifier to obtain and / or identify each embryo in the embryo image.
[0015] In some disclosures, the S1.1 includes the following steps:
[0016] S1.1.1. Perform image binarization and morphological processing of first dilation and then erosion on the embryo image;
[0017] S1.1.2. Obtain all contours in the image after image binarization and calculate the area of the contours, select contours with an area between 80,000 and 100,000 pixels, and use the contours as boundaries to create a mask to blur the embryo area.
[0018] In some disclosures, each embryo in the acquired and / or identified embryo image is scaled to form an image of 224x224 pixels.
[0019] In some disclosures, the embryo presenting a 224x224 pixel image is used as a training set for a convolutional neural network, which identifies the number of pronuclei in the embryo presenting a 224x224 pixel image, and takes the category with the highest probability as the identification result of the embryo presenting a 224x224 pixel image. The convolutional neural network has four categories for the number of pronuclei in the embryo presenting a 224x224 pixel image: 0pn, 1pn, 2pn, and 3pn, representing 0 pronuclei, 1 pronucleus, 2 pronuclei, 3 pronuclei and above, respectively.
[0020] In some disclosures, the convolutional neural network uses a SE-ResNet-18 convolutional neural network.
[0021] In some disclosures, the S3 comprises the following steps:
[0022] 1) According to the classified and identified embryos, the pronuclear stage of the embryo sequence corresponding to the embryo and the number of pronuclear categories appearing are calculated to obtain a set C, and the pronuclear stage of the embryo sequence is set to N frames;
[0023] 2) For each category in set C, calculate the maximum number of continuous frames n of the largest category set c in set C;
[0024] 3) Calculate n / N;
[0025] 4) If n / N>F, it is determined to be of set c; otherwise, the class is removed from set C and return to 2) until the class is found.
[0026] Beneficial effects of the present disclosure:
[0027] The present invention utilizes the SE-ResNet-18 convolutional neural network and a small amount of annotation to realize automatic identification of the number of pronuclei with high accuracy, and also proposes a new method for determining the number of pronuclei of embryos based on the identification results of single-frame images of embryo sequences. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0029] Figure 1 It is an overall framework diagram of an embodiment of the present disclosure;
[0030] Figure 2 is a schematic diagram of a picture preprocessing process according to an embodiment of the present disclosure;
[0031] Figure 3is a schematic diagram of a processed embryo according to an embodiment of the present disclosure;
[0032] Figure 4 It is a schematic diagram of the residual module structure of a ResNet acting on a 56x56 pixel feature map according to an embodiment of the present disclosure;
[0033] Figure 5 is a schematic diagram of the structure of the SE-ResNet-18 module in an embodiment of the present disclosure;
[0034] Figure 6 is the result of determining the number of pronuclei of the 3PN embryo in the system according to the embodiment of the present disclosure;
[0035] Figure 7 This is the result of determining the number of pronuclei of the 2PN embryos in the system according to the embodiment of the present disclosure. DETAILED DESCRIPTION
[0036] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.
[0037] A method for evaluating the number of pronuclei of pronuclear embryos, comprising: acquiring and / or identifying each embryo in an embryo image; identifying the number of pronuclei of each embryo in the embryo image by a convolutional neural network, treating the problem of pronuclei number identification as a classification problem, and classifying the number of pronuclei of pronuclei in the pronuclear embryo into four categories: 0PN, 1PN, 2PN and 3PN, which respectively represent 0 pronuclei, 1 pronuclei, 2 pronuclei, and 3 pronuclei and above; judging the number of pronuclei of the embryo based on the result of pronuclei number identification of the embryo sequence after classification and identification; Figure 1 As shown, Figure 1 (a) Original embryo image (b) Detected embryonic cells (c) Extraction of each embryo image (d) CNN classifier (e) Pronuclear number discrimination results.
[0038] In some disclosures, the evaluation method adopts the following method:
[0039] 1. Acquiring and / or identifying each embryo in the embryo image, comprising the following steps:
[0040] The purpose of image preprocessing is to extract the embryo area in the image. The image preprocessing process is divided into two steps: first, making a mask to blur the embryo area and then extracting the effective embryo area of a single embryo.
[0041] 1.1 Making a mask to blur the embryo area
[0042] In the data, a picture contains multiple embryos. Each embryo is extracted by making a mask for each embryo. The process of making a mask is to first binarize the image and then extract the contour to make a mask. First, the image is converted into a grayscale image, and then the grayscale threshold transformation is used to convert the image into a binary image. There are some noise points in the binarized image, which may interfere with the extraction of the contour boundary of interest. Therefore, the noise points are processed by morphological methods, which adopts the morphological operation of first dilation and then corrosion. When binarizing the image, the threshold is set to 10, and the structural elements of corrosion and expansion use a 3x3 square structure. Then, contour detection is performed, all contours in the binary image after the above processing are calculated and the contour area is calculated. The contour with an area between 80000-100000 pixels is selected, and the contour is used as the boundary to make a mask of the fuzzy embryo area. Then, the mask and the original grayscale image are combined by bitwise operation to extract the circular independent area where each embryo is located, and a mask of the fuzzy embryo area is made. Here, the purpose of making a mask to blur the embryo area is to make it possible to operate on each embryo independently, prevent interference from other parts of the image, and remove noise around each embryo.
[0043] In some disclosures, a micro-hole area is opened in a culture dish, and the embryo is placed in a single micro-hole area. At this time, a mask that blurs the embryo area can be made using the contour boundary of the micro-hole area.
[0044] 1.2 Extraction of effective embryonic region of a single embryo
[0045] After the above steps, the image can be processed to obtain the area where each embryo is located. Then, the effective embryo area of a single embryo is extracted by fitting the cell boundary and making a mask using the Hough ring transform. First, the image is smoothed using Gaussian filtering, and then the Hough ring transform is used to find the embryo cell boundary and use this boundary to make a mask. Finally, each embryo image is obtained by finding the straight boundary rectangle of each ring outline detected by the above Hough ring transform, such as Figure 2 and Figure 3 As shown, Figure 2 (a) Original image; (b) Binarized image: a is binarized and morphologically processed; (c) Mask: The mask is made by extracting the micropore contour according to the contour area of b; (d) c and a are bitwise operated to extract the fuzzy embryo area where each embryo is located; (e) Hough ring transform is performed on d to fit the embryo boundary and use it to extract the effective embryo area;
[0046] Figure 3 In the figure, (a)-(i) are the 9 embryos in the corresponding positions obtained after the original images are preprocessed. These images will be input into the CNN classifier.
[0047] 2. The number of pronuclei of each embryo in the embryo image is identified by a convolutional neural network. The problem of pronuclei identification is treated as a classification problem, and the number of pronuclei of pronuclear stage embryos is divided into four categories: 0PN, 1PN, 2PN and 3PN, representing 0 pronuclei, 1 pronucleus, 2 pronuclei, 3 pronuclei and above, respectively. The following steps are included: After the embryo image is preprocessed, it is directly input into the ensemble convolutional neural network to identify the number of pronuclei.
[0048] The identification of the number of pronuclei is treated as a classification problem. After the preprocessed image is scaled to 224x224 pixels, it can be directly used as training data to input the ensemble convolutional neural network for pronuclei identification. The classification results output by the ensemble convolutional neural network are 0pn, 1pn, 2pn, and 3pn. For each image, the ensemble convolutional neural network will return the probability that the image belongs to a certain category, such as Figure 6 As shown in the figure, the category with the highest probability is taken as the classification result of the image. The SE-ResNet-18 convolutional neural network is used as the classifier.
[0049] SE-ResNet-18 is formed by adding the SE (Squeeze-and-Excitation) module [7] to the ResNet structure. ResNet is known as a deep residual network. It learns residual mapping and adds short connections between some layers of the network to ensure that the accuracy can be continuously improved as the number of network layers increases. Depending on the number of network layers, ResNet has many variants such as ResNet-18, ResNet-34, ResNet-50, ResNet-101, and ResNet-152. This article uses an 18-layer ResNet, namely ResNet-18, whose structure is shown in Table 1. In Table 1, from Conv2 to Conv5 layers, each layer has two residual modules. Figure 4 A residual module acting on a 56X56 pixel feature map is shown. The SE module can perform feature calibration by learning to use global information to selectively emphasize useful features and suppress less useful features. Specifically, the global spatial information is first squeezed into a channel descriptor, and then the excitation operation is used to fully capture the dependencies on the channel. Finally, the feature map is reweighted and input into the subsequent layer. Adding the SE module to the ResNet residual module results in SE-ResNet-18, as shown in Figure 5 As shown, Figure 5 (a) is the ResNet residual module, Figure 5 (b) is the SE module, Figure 5 (c) is the SE-ResNet-18 module obtained by adding the SE module to the ResNet residual module.
[0050] The preprocessed images are scaled to 224X224 pixels and input into SE-ResNet-18-18 for training.
[0051] Table 1: Basic structure of ResNet-18
[0052]
[0053] Based on the pronuclear number identification results, the number of pronuclei in the embryo is determined. The determination method is as follows:
[0054] 1) Calculate the number of pronuclei in each single frame image of the pronuclear period of the time difference sequence (a total of N frames), and the different counting results constitute a set C;
[0055] 2) For the maximum count value c in C, calculate the maximum number of consecutive frames n of c in the time difference sequence;
[0056] 3) If n / N exceeds the threshold F, it is determined to be class c; otherwise, c is removed from C and returns to step 2).
[0057] In the present disclosure, the threshold F is 10%. Of course, in some other disclosures, it may be other parameters.
[0058] The above determination method uses the maximum number of continuous frames as a partial measurement indicator. This can avoid recognition errors caused by the oscillation of the model recognition results of continuous frames between 0pn, 1pn, 2pn, and 3pn due to unstable recognition results, thereby obtaining a robust recognition result.
[0059] like Figure 6 As shown in the figure, the embryo sequence recognition result of an embryo in the complete pronuclear stage is shown. The horizontal axis represents each frame shot according to the time, and the vertical axis represents the probability that the current frame recognized by the trained model belongs to each category. For each frame, the model will select the category with the highest probability as the classification result.
[0060] 4 Experiments
[0061] 4.1 Dataset
[0062] The images were taken based on the time-lapse system. The original image size is 2004x2592 pixels, and each image contains 9 embryos. After the image preprocessing process in Part 2, each embryo is extracted. For each embryo sequence, only the pronuclear stage images are used. Finally, the dataset used contains 111 embryo image sequences, a total of 12,838 images. The dataset is divided into four categories according to the number of pronuclei, namely 0 pronuclei, 1 pronucleus, 2 pronuclei, 3 pronuclei and above, of which there are 2,659 images of 0 pronuclei, 1,813 images of 1 pronucleus, 7,479 images of 2 pronuclei, and 887 images of 3PN. The dataset is randomly divided into training set, validation set and test set in a ratio of 5:2:3, with 6,424, 2,564 and 3,850 images respectively. The distribution of the dataset is shown in Table 2.
[0063] Table 2: Dataset distribution
[0064]
[0065] 4.2 Evaluation Metrics
[0066] For classification problems, samples can be divided into four categories: true positive, false positive, true negative, and false negative according to the combination of their true categories and the categories predicted by the classifier. TP, FP, TN, and FN represent the corresponding number of samples. The confusion matrix of the classification results is shown in Table 3.
[0067] Table 3: Classification result confusion matrix
[0068]
[0069] This disclosure uses precision, recall, and accuracy as model evaluation indicators. The calculation formulas for these evaluation indicators are as follows:
[0070]
[0071]
[0072]
[0073] 4.3 Experimental Setup
[0074] The system implementation and experimental procedures are implemented in Python programming language; OpenCV is used for image processing algorithm; the open source deep learning framework PyTorch is used for CNN construction and training; the loss function uses cross entropy loss, and the model is optimized using Adam algorithm, with the learning rate set to 0.01, BatchSize set to 72, and the number of iterations set to 80.
[0075] The model was trained using 6424 single-frame images in the training set, and the models generated after the 15th iteration of the training process were saved, and the model with the highest accuracy on the validation set was selected. The model will be tested and evaluated on the test set and applied to the embryo pronucleus number discrimination system.
[0076] 4.4.1 Experimental Results
[0077] 4.4.1 Prokaryotic counts in single-frame images
[0078] The SE-ResNet-18-18 model selected from the validation set was tested on 3850 embryo images in the test set. The overall accuracy of the experimental results reached 96.88%, and the detailed results are shown in Tables 4 and 5. The accuracy of each category of 0PN, 1PN, 2PN, and 3PN was 98.10%, 98.21%, 97.69%, and 99.77%, respectively, the recall rate was 95.61%, 91.71%, 98.35%, and 98.86%, respectively, and the precision rate was 95.25%, 95.40%, 97.70%, and 97.74%, respectively. It can be seen from the experimental results that the SE-ResNet-18-18 network model has a strong learning ability and can excellently complete the task of pronucleus counting in a single frame image. Especially for the case of 3PN, the accuracy, recall rate, and precision of the model counting are the highest, which can effectively assist in avoiding 3PN and above embryos.
[0079] Table 4: Confusion matrix of test set classification results
[0080]
[0081] Table 5: Evaluation indicators of test set classification results
[0082]
[0083] 4.4.2 Determination of the number of pronuclei in time-difference images
[0084] The embryo pronuclear number discrimination system will use the trained SE-ResNet-18-18 network and take the time-difference image sequence of the embryo pronuclear period as the system input. The SE-ResNet-18-18 in the system classifies each single-frame image in the sequence and obtains the probability of each single-frame image corresponding to 0PN, 1PN, 2PN, and 3PN. The system comprehensively considers the probability of each frame image in the time sequence and gives the final pronuclear number discrimination result of the embryo in the time-difference image. The threshold used in this experiment is 10%, which enables the system to correctly judge all the embryo sequences in this experiment.
[0085] Figure 6The discrimination results of 3PN embryos in the system are shown. There are 128 frames of time-lapse images of the embryo in the pronuclear stage. The horizontal axis is each frame of the image taken in chronological order, and the vertical axis is the recognition result of the SE-ResNet-18-18 model in the system. The system will select the category with the highest probability as the discrimination result. Figure 7 The figure shows the discrimination results of 2PN embryos in the system. There are 127 frames of time-lapse images of this embryo in the pronuclear stage. As can be seen from the figure, although the 48th frame is identified as 3PN, the system can be identified as 2PN based on the overall result curve. It can be seen that the embryo pronuclear number discrimination system uses the overall evaluation in the time-lapse image and ignores the discrimination error of the SE-ResNet-18-18 model, so that the system can have strong stability and discrimination accuracy.
[0086] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.
[0087] The above shows and describes the basic principles, main features and advantages of the present disclosure. Those skilled in the art should understand that the present disclosure is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present disclosure. Without departing from the spirit and scope of the present disclosure, the present disclosure may have various changes and improvements, and these changes and improvements all fall within the scope of the present disclosure to be protected.
Claims
1. A method for evaluating the number of pronuclear embryos in the pronuclear stage, characterized in that: The following steps are involved: S1. Acquire and / or identify each embryo in the embryo image; S2. The number of pronuclei of each embryo in the embryo image is identified by convolutional neural network. The problem of pronuclei identification is treated as a classification problem, and the number of pronuclei of pronuclear embryos is divided into four categories: 0PN, 1PN, 2PN and 3PN, which respectively represent 0 pronuclei, 1 pronucleus, 2 pronuclei, 3 pronuclei and above; S3, judging the number of pronuclei of the embryo based on the result of identifying the number of pronuclei of the embryo corresponding to the pronuclear period time series of the embryo sequence after classification and identification; The step S1 comprises: S1.1, acquiring and / or identifying a fuzzy embryo region of each embryo in the embryo image, and making a mask of the fuzzy embryo region for the fuzzy embryo region of each embryo; S1.2, according to the mask of the fuzzy embryo area, use Gaussian filtering to smooth the image, then use Hough ring transform to find the embryo cell boundary, and based on the embryo cell boundary, make a mask of the effective embryo area, obtain the effective embryo area of each embryo, and use the effective embryo area as the corresponding embryo identifier, so as to obtain and / or identify each embryo in the embryo image; The S1.1 comprises the following steps: S1.1.
1. Perform image binarization and morphological processing of first dilation and then erosion on the embryo image; S1.1.2, obtain all contours in the image after image binarization and calculate the area of the contour, select contours with an area between 80,000 and 100,000 pixels, and use the contours as boundaries to make a mask to blur the embryo area; The convolutional neural network adopts SE-ResNet-18 convolutional neural network; The S3 comprises the following steps: 1) According to the classified and identified embryos, the pronuclear stage of the embryo sequence corresponding to the embryo and the number of pronuclear categories appearing are calculated to obtain a set C, and the pronuclear stage of the embryo sequence is set to N frames; 2) For each category in set C, calculate the maximum number of continuous frames n of the largest category set c in set C; 3) Calculate n / N; 4) If n / N>F, and the threshold F is 10%, it is determined to be in set c; otherwise, the class is removed from set C and the process returns to step 2) until the class is found.
2. The evaluation method according to claim 1, characterized in that: Each embryo in the acquired and / or identified embryo image is scaled to form an image of 224x224 pixels.
3. The evaluation method according to claim 2, characterized in that: The embryo with a 224x224 pixel image is used as a training set for a convolutional neural network. The convolutional neural network identifies the number of pronuclei in the embryo with a 224x224 pixel image, and takes the category with the highest probability as the recognition result of the embryo with a 224x224 pixel image. The convolutional neural network has four categories for the number of pronuclei in the embryo with a 224x224 pixel image: 0pn, 1pn, 2pn, and 3pn, which respectively represent 0 pronuclei, 1 pronucleus, 2 pronuclei, 3 pronuclei and above.