Blastocyst image instance segmentation method and device, computer equipment and storage medium

Abstract

The invention discloses a blastocyst image instance segmentation method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring a blastocyst image to be segmented; preprocessing the to-be-segmented blastocyst image to obtain a preprocessed to-be-segmented blastocyst image; inputting the preprocessed blastocyst image to be segmented into an instance segmentation model for instance segmentation of internal components to obtain a segmentation result; wherein the instance segmentation model is a model obtained by amplifying a blastocyst image through data enhancement, preprocessing the blastocyst image to form a sample set, and training a deep learning network by using the sample set; and outputting the segmentation result. By implementing the method provided by the invention, the segmentation of the cell mass and the trophoderm in the human blastocyst image can be realized, the morphological characteristics of the cell mass and the trophoderm can be extracted, and the problems existing in blastocyst image analysis can be better solved.

Description

Technical Field

[0001] The present invention relates to a medical image segmentation method, and more specifically to a blastocyst image instance segmentation method, device, computer equipment and storage medium. Background Art

[0002] Infertility is a health challenge faced by many couples around the world, affecting approximately 10%-15% of the population. The development of assisted reproductive technology, especially in vitro fertilization technology, provides an effective way to solve this problem. During the IVF process, sperm and eggs are combined in a laboratory environment to form embryos, and then well-developed embryos are selected and transplanted into the mother's uterus in the hope of a successful pregnancy. In this process, embryo quality assessment is crucial, and it relies on the embryologist's careful observation of the internal structure of the blastocyst, including the inner cell mass and trophectoderm. However, this evaluation method is not only time-consuming, but also depends largely on personal experience and judgment, resulting in large differences in results.

[0003] In order to improve the objectivity and efficiency of the evaluation, researchers began to explore the use of computer vision technology to achieve automated identification and evaluation of the internal structure of the blastocyst. For example, image processing techniques such as level set algorithm, phase consistency and ellipse fitting were used to define the internal boundary of the blastocyst; the TE region was identified through Gabor transform and discrete cosine transform features, and the ICM boundary was determined by combining the region-based level set method. Although these methods have achieved automation to a certain extent, they still require manual adjustment of parameters and have not been able to completely get rid of the influence of human factors.

[0004] With the continuous advancement of deep learning technology, fully convolutional neural networks, U-Net and its improved versions have been widely used in the automatic segmentation of blastocyst images. However, when processing blastocyst images, there are still challenges such as complex cell morphology, similar textures between categories, and blurred boundaries, especially in accurately capturing the morphological characteristics of TE cells around the equatorial plane, which is a key indicator for evaluating blastocyst quality.

[0005] Therefore, it is necessary to design a new method to achieve the segmentation of inner cell mass and trophectoderm in human blastocyst images, extract their morphological features, and better solve the problems existing in blastocyst image analysis. Summary of the invention

[0006] The purpose of the present invention is to overcome the defects of the prior art and provide a method, device, computer equipment and storage medium for blastocyst image instance segmentation.

[0007] To achieve the above object, the present invention adopts the following technical solution: a blastocyst image instance segmentation method, comprising: Acquire an image of a blastocyst to be segmented; Preprocessing the blastocyst image to be segmented to obtain a preprocessed blastocyst image to be segmented; The preprocessed blastocyst image to be segmented is input into the instance segmentation model to perform instance segmentation of the internal components to obtain a segmentation result; wherein the instance segmentation model is a model obtained by amplifying the blastocyst image through data augmentation, forming a sample set after preprocessing, and using the sample set to train a deep learning network; The segmentation result is output.

[0008] A further technical solution is as follows: the instance segmentation model is a model obtained by amplifying the blastocyst image through data enhancement and preprocessing to form a sample set, and using the sample set to train a deep learning network, including: Obtain Hoffman modulation contrast microscopic blastocyst images on the fifth day after in vitro fertilization, and perform data enhancement processing to obtain a blastocyst image sample library; Performing instance segmentation and annotation on the original images in the blastocyst image sample library to obtain a blastocyst image instance segmentation dataset; Preprocessing the blastocyst image instance segmentation dataset to obtain a sample set; Build deep learning networks; The deep learning network is trained using the sample set, and network parameters are adjusted to optimize network performance to obtain an instance segmentation model.

[0009] A further technical solution is: the Hoffman modulation contrast microscopic blastocyst image is obtained on the fifth day after in vitro fertilization, and data enhancement processing is performed to obtain a blastocyst image sample library, including: Acquire Hoffman modulation contrast microscopic blastocyst images on the fifth day after in vitro fertilization, exclude images that meet set conditions, and perform data enhancement processing on the excluded images to obtain a blastocyst image sample library; Among them, the images with set conditions include samples that failed to develop to the blastocyst stage after in vitro fertilization; samples that lacked a clear equatorial cell layer and inner cell mass structure; and samples with incomplete blastocyst cells in the field of view.

[0010] A further technical solution is: performing instance segmentation and annotation on the original images in the blastocyst image sample library to obtain a blastocyst image instance segmentation dataset, including: Selecting some original images from the blastocyst image sample library by random sampling method and dividing them into a training set and a test set; The polygon annotation technology is used to mark the boundaries of each component in the training set and the test set, where the components include the outlines of cell clusters and trophectoderm cells; Save the labeled training and test sets in JSON format to obtain the blastocyst image instance segmentation dataset.

[0011] A further technical solution is: the blastocyst image instance segmentation data set is preprocessed to obtain a sample set, including: The blastocyst image instance segmentation dataset is scaled to obtain a sample set.

[0012] Its further technical solution is: the instance segmentation model includes a backbone feature extraction network, a feature pyramid network, a region proposal network, a region of interest layer, a classification and border branch, and a mask branch; wherein the backbone feature extraction network uses Res2Net and DCN modules; the feature pyramid network is a feature pyramid network with a CARAFE upsampling operator; the region proposal network shares a feature extraction network with the backbone feature extraction network and is connected to a weighted connection network; the mask branch includes a PointRend algorithm.

[0013] A further technical solution is: using the sample set to train the deep learning network, and adjusting network parameters to optimize network performance to obtain an instance segmentation model, including: Using the sample set in an end-to-end manner, fine-tuning the bottom two layers of the backbone feature extraction network with fixed parameters of the ImageNet pre-trained model; During the network optimization process, stochastic gradient descent is used to optimize the multi-task loss function to obtain the instance segmentation model.

[0014] The present invention also provides a blastocyst image instance segmentation device, comprising: An image acquisition unit, used for acquiring an image of the blastocyst to be segmented; A preprocessing unit, used for preprocessing the blastocyst image to be segmented to obtain a preprocessed blastocyst image to be segmented; A segmentation unit, used for inputting the preprocessed blastocyst image to be segmented into an instance segmentation model to perform instance segmentation of internal components to obtain a segmentation result; wherein the instance segmentation model is a model obtained by amplifying the blastocyst image through data augmentation, forming a sample set after preprocessing, and using the sample set to train a deep learning network; An output unit is used to output the segmentation result.

[0015] The present invention further provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor implements the above method when executing the computer program.

[0016] The present invention also provides a storage medium, wherein the storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0017] The beneficial effects of the present invention compared with the prior art are as follows: the present invention obtains the blastocyst image to be segmented and performs necessary preprocessing to improve the image quality and adaptability; the preprocessed blastocyst image is input into a trained instance segmentation model to achieve accurate segmentation of the internal components of the blastocyst such as the inner cell mass and the trophectoderm; the instance segmentation model amplifies the blastocyst image through data enhancement technology, and uses the amplified sample set to train the deep learning network; during the model training process, the annotated images in the sample set help the model learn the features of different structures to improve the accuracy and robustness of the segmentation; finally, the model outputs the segmentation result of the blastocyst image, extracts the morphological features of the inner cell mass and the trophectoderm from it, to support efficient and automated blastocyst quality analysis.

[0018] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0020] Figure 1 A schematic diagram of an application scenario of the blastocyst image instance segmentation method provided by an embodiment of the present invention; Figure 2 A schematic diagram of a flow chart of a blastocyst image instance segmentation method provided by an embodiment of the present invention; Figure 3 A schematic block diagram of a blastocyst image instance segmentation device provided by an embodiment of the present invention; Figure 4 A schematic block diagram of a computer device provided for an embodiment of the present invention; Figure 5 A schematic diagram of the internal components of a human blastocyst image provided by an embodiment of the present invention.

[0021] Figure 6 An architecture diagram of an instance segmentation model provided in an embodiment of the present invention.

[0022] Figure 7 A schematic diagram of a segmentation result provided by an embodiment of the present invention; Figure 8 The performance comparison results of the image instance segmentation model provided in the embodiment of the present invention and various image segmentation algorithms for images inside human blastocysts are shown.

[0023] Fig. 9This is a performance comparison diagram of the image instance segmentation model provided in an embodiment of the present invention and various image segmentation algorithms for images inside a human blastocyst. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0026] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0027] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0028] See also Figure 1 and Figure 2 , Figure 1 A schematic diagram of an application scenario of the blastocyst image instance segmentation method provided in an embodiment of the present invention. Figure 2A schematic flow chart of a blastocyst image instance segmentation method provided in an embodiment of the present invention. The blastocyst image instance segmentation method is applied to a server. The server interacts with the terminal to perform data exchange, first obtains and preprocesses Hoffman modulation contrast microscopic blastocyst images on the fifth day of in vitro fertilization, and establishes a rich sample library through data enhancement. Then, the polygon annotation technology is used to accurately annotate the images in the sample library for instance segmentation, and training and test data sets are generated. The constructed instance segmentation model uses Res2Net and DCN as the backbone network, combined with the feature pyramid network with CARAFE upsampling and the mask branch of the PointRend algorithm, to achieve efficient feature extraction and fine segmentation boundary generation. During the model training process, the ImageNet pre-trained model is used to fine-tune the backbone network, and the overall performance is improved by optimizing the multi-task loss function. Finally, the model can accurately segment the inner cell mass and trophectoderm in human blastocyst images, effectively extract their morphological features, and solve the challenges in blastocyst image analysis.

[0029] Figure 2 FIG. 1 is a flow chart of a method for segmenting a blastocyst image instance provided by an embodiment of the present invention. Figure 2 As shown, the method includes the following steps S110 to S140.

[0030] S110, obtaining an image of the blastocyst to be segmented.

[0031] In this embodiment, the blastocyst images to be segmented are obtained, which are images of the blastocysts taken by Hoffman modulation contrast microscope on the fifth day after in vitro fertilization, for subsequent image processing and analysis. This step ensures that there are high-quality original images for the subsequent instance segmentation process.

[0032] S120 , preprocessing the blastocyst image to be segmented to obtain a preprocessed blastocyst image to be segmented.

[0033] In this embodiment, the blastocyst image to be segmented is scaled to form a pre-processed blastocyst image to be segmented.

[0034] Of course, in other embodiments, other processing methods may be performed on the blastocyst image to be segmented. The main purpose of preprocessing the acquired blastocyst image is to improve the image quality and enhance the features so that the subsequent segmentation operation is more accurate. The preprocessing may include the following aspects: Use filters (such as Gaussian filtering and median filtering) to remove noise from the image and reduce unnecessary interference information.

[0035] If the original image is in color, it is usually converted to grayscale because many image processing algorithms are more suitable for processing single-channel images.

[0036] Enhancing the contrast of the image can make the cell boundaries more distinct, which is helpful for subsequent edge detection and segmentation.

[0037] Normalizing the image so that the pixel values ​​of the image are within a fixed range, such as between 0 and 1, helps improve the consistency and stability of subsequent processing.

[0038] If the sizes of blastocyst images from different sources are inconsistent, they should be scaled to reach a uniform size standard for easy subsequent processing.

[0039] Use morphological operations (such as erosion and dilation) to clean up small structures in the image or fill small holes in the target area.

[0040] The selection and application of each step depends on the specific characteristics of the blastocyst image and the requirements of subsequent segmentation. The ultimate goal of preprocessing is to make the key structures in the blastocyst image more prominent while reducing unnecessary background information, thereby laying a good foundation for the subsequent segmentation step.

[0041] S130. Input the preprocessed blastocyst image to be segmented into an instance segmentation model to perform instance segmentation of internal components to obtain a segmentation result; wherein the instance segmentation model is a model obtained by amplifying the blastocyst image through data enhancement, forming a sample set after preprocessing, and using the sample set to train a deep learning network.

[0042] In this embodiment, the preprocessed blastocyst image to be segmented is input into the instance segmentation model to perform instance segmentation of the internal components, which can bring many benefits: Automated processing can greatly reduce the time and cost of manual operations, especially when a large number of samples need to be processed. The instance segmentation model can complete the task quickly and accurately. Compared with traditional manual observation and manual segmentation, automatic segmentation using a deep learning model can reduce the errors caused by subjective judgment and improve the accuracy of segmentation. This is very important for scientific research and clinical diagnosis. Once the machine learning model is trained, it can produce consistent results under the same conditions, which is very beneficial for the repeatability and standardization of research. Advanced instance segmentation models can identify and segment very subtle structures in the image, which is of great significance for understanding the internal composition and development process of the blastocyst. By accurately segmenting the different components of the blastocyst, researchers and doctors can obtain more information to make more accurate diagnostic and treatment decisions, especially in the field of assisted reproductive technology. Automated segmentation technology can help researchers obtain experimental data faster, accelerate the study of the development mechanism of the blastocyst, and promote scientific progress in related fields.

[0043] In summary, using the instance segmentation model to segment the internal components of blastocyst images not only improves work efficiency, but also enhances the accuracy and reliability of the results, which is of great significance for scientific research and clinical applications.

[0044] In one embodiment, the above-mentioned instance segmentation model is a model obtained by amplifying the blastocyst image through data enhancement and forming a sample set after preprocessing, and using the sample set to train a deep learning network, including steps S131~S135.

[0045] S131. Obtain Hoffman modulation contrast microscopic blastocyst images on the fifth day after in vitro fertilization, and perform data enhancement processing to obtain a blastocyst image sample library.

[0046] In this embodiment, Hoffman modulation contrast microscopic blastocyst images on the fifth day after in vitro fertilization are obtained, images that meet the set conditions are excluded, and the excluded images are subjected to data enhancement processing to obtain a blastocyst image sample library; Among them, the images with set conditions include samples that failed to develop to the blastocyst stage after in vitro fertilization; samples that lacked a clear equatorial cell layer and inner cell mass structure; and samples with incomplete blastocyst cells in the field of view.

[0047] First, images of blastocysts on the fifth day after in vitro fertilization were obtained using Hoffman modulation contrast microscopy. At this stage, blastocysts have developed obvious structural features, such as inner cell mass and peripheral trophoblast cells, which are essential for evaluating the quality of blastocysts. From the acquired images, those that do not meet specific criteria are excluded. The exclusion criteria include, but are not limited to: samples that have failed to develop to the blastocyst stage; samples that lack a clear equatorial cell layer and inner cell mass structure; and samples with incomplete blastocyst cells in the field of view. Through this step, it can be ensured that the selected samples are representative, thereby improving the accuracy of subsequent analysis. Data enhancement processing is performed on the screened images, including but not limited to rotation, flipping, scaling and other operations, to increase the diversity of the sample library. This enhancement not only helps to expand the size of the training set, but also improves the ability of the machine learning model to identify blastocysts of different morphologies, thereby improving the generalization performance of the model.

[0048] Through strict image screening standards, the quality of all images in the sample library is guaranteed, which is crucial for subsequent deep learning algorithm training. The application of data augmentation technology enables the model to be exposed to more diverse images, which helps to improve the robustness and accuracy of the model. The high-quality blastocyst image sample library provides a valuable resource for embryological research, helps to deeply understand the complex mechanisms in the embryonic development process, and also provides strong support for clinical applications.

[0049] S132. Perform instance segmentation and annotation on the original images in the blastocyst image sample library to obtain a blastocyst image instance segmentation dataset.

[0050] In this embodiment, the blastocyst image instance segmentation dataset refers to the blastocyst image after the boundaries of each component are marked by instance segmentation.

[0051] In one embodiment, the above-mentioned step S132 may include steps S1321 - S1323 .

[0052] S1321, selecting some original images from the blastocyst image sample library by using a random sampling method, and dividing them into a training set and a test set; S1322, using polygon annotation technology to mark the boundaries of each component in the training set and the test set, wherein the components include the outlines of cell clusters and trophectoderm cells; S1323. Save the labeled training set and test set in JSON format to obtain a blastocyst image instance segmentation dataset.

[0053] In this embodiment, images used to construct labeled samples are carefully selected by random sampling, and these images need to undergo strict quality control to ensure their representativeness and diversity.

[0054] The selected images are divided into a predetermined ratio (e.g., 70% as a training set and 30% as a test set) to ensure the effectiveness of model training and the accuracy of testing.

[0055] like Figure 5 As shown in the figure, polygonal annotation technology is used to accurately annotate the internal components of the human blastocyst image. The annotation content mainly includes: Inner Cell Mass (ICM): Marks the morphological outline of the inner cell mass, which is the main source of future fetal development.

[0056] Trophectoderm (TE) Cells: Label the morphological outlines of the trophectoderm cells, which will form the placenta and other tissues that support fetal development.

[0057] This step requires a high degree of precision and professionalism to ensure the accuracy of subsequent analysis. The annotator needs to have professional knowledge and rich experience in the relevant field.

[0058] After the annotation is completed, the annotation information of the instance segmentation is stored as a JSON file. In order to ensure the standardization and compatibility of the data, the JSON files of all images will be uniformly converted according to the COCO dataset format.

[0059] The converted data files will be matched with the corresponding human blastocyst images and archived for subsequent data processing and analysis.

[0060] Polygonal annotation technology can accurately depict the contours of the internal structure of the blastocyst, ensuring the accuracy of the annotation information, thereby improving the reliability of subsequent analysis. The annotation information is uniformly converted according to the COCO dataset format to ensure data standardization and compatibility, making it easier to share and collaborate with other research teams. High-quality annotated data helps train more accurate and robust machine learning models, thereby improving the performance of the models in practical applications. Through systematic methods to manage and utilize high-quality blastocyst image sample libraries, the progress of embryological research can be accelerated, providing strong support for the development of assisted reproductive technology.

[0061] S133. Preprocess the blastocyst image instance segmentation dataset to obtain a sample set.

[0062] In this embodiment, the blastocyst image instance segmentation dataset is scaled to obtain a sample set.

[0063] Specifically, different deep learning models may require the input image to have a specific size. By scaling the original image, you can ensure that the image size meets the model's requirements. During the scaling process, you need to adjust the annotation information (such as polygon coordinates) to ensure that the annotation information is consistent with the scaled image.

[0064] First, determine the image size after scaling based on the input size requirements of the target model. Common target sizes include 256x256, 512x512, etc. Select an appropriate interpolation algorithm (such as bilinear interpolation, nearest neighbor interpolation, etc.) to scale the image. Bilinear interpolation usually provides better image quality, but the computational cost is high; nearest neighbor interpolation is fast, but may result in image quality degradation. Read the annotation information of each image from the JSON file, including the vertex coordinates of the polygon. Calculate the scaling ratio based on the original image size and the target image size. For example, if the original image size is 1024x1024 and the target size is 512x512, the scaling ratio is 0.5. Adjust the coordinates of the polygon vertices based on the scaling ratio. The new coordinates can be obtained by multiplying the original coordinates by the scaling ratio. For example, if the original coordinates of a vertex are (100, 200) and the scaling ratio is 0.5, the new coordinates are (50, 100). Write the adjusted annotation information back to the JSON file and save it to the corresponding directory.

[0065] Through scaling, images can be adapted to the input size requirements of different models, improving the versatility and flexibility of the model. Synchronously adjust the annotation information to ensure that the annotation information is consistent with the scaled image to avoid annotation errors caused by size changes. The scaled images and annotation information can be used for more experiments and model training, improving the utilization and value of data. S134, build a deep learning network; S135. Use the sample set to train the deep learning network, and adjust network parameters to optimize network performance to obtain an instance segmentation model.

[0066] In this embodiment, if Figure 6 As shown, the instance segmentation model includes a backbone feature extraction network, a feature pyramid network, a region proposal network, an area of ​​interest layer, a classification and border branch, and a mask branch; wherein the backbone feature extraction network uses Res2Net and DCN modules; the feature pyramid network is a feature pyramid network with a CARAFE upsampling operator; the region proposal network shares a feature extraction network with the backbone feature extraction network and is connected to a weighted connection network; the mask branch includes a PointRend algorithm.

[0067] In this embodiment, in the constructed blastocyst image instance segmentation model, a multi-branch neural network structure is adopted, and each component is optimized for a specific task to improve the overall performance. Specifically, the backbone network selects the Res2Net and DCN modules, which are designed to enhance the receptive field of the model and the fine-grained representation of the blastocyst image. Res2Net improves the traditional ResNet structure through grouping and residual operations. Specifically, it divides the input features into multiple groups, each group is processed by a small convolution kernel, and then the output of the previous group is combined with the next part of the input features and passed to the next convolution kernel group. This process is repeated several times until all input features are processed. Finally, the feature maps of all groups are fused through a 1x1 convolution layer to achieve complete integration of information. In each 3x3 deformable convolution layer, the effective increase of the receptive field can obtain a variety of equivalent feature scales through the combination effect without significantly increasing the computational burden. DCN (Deformable Convolutional Networks) further enhances this capability by adding an offset to the standard convolution kernel, allowing the convolution kernel to adaptively focus on any position in the feature map, which is particularly suitable for processing irregularly shaped cell structures.

[0068] In order to better capture features at different scales, the model uses a feature pyramid network (FPN) that integrates the CARAFE (Content-Aware ReAssembly of Features) operator. FPN builds a powerful multi-scale feature pyramid through top-down paths and lateral connections, effectively solving the problem of insufficient contextual semantic information transmission between feature maps of different scales. The CARAFE operator upsamples the feature map in a content-aware manner, ensuring the smooth fusion of features at different levels, especially improving the recognition of small cells and cell edge details.

[0069] The region proposal network shares the feature extraction layer with the backbone network and is responsible for generating candidate regions. RPN infers the location information of the candidate regions and the foreground and background classification by adding a fully connected layer, and then uses the non-maximum suppression (NMS) technique to remove redundant candidate regions and retain the regions most likely to contain the target.

[0070] The RoI layer obtains the features of the candidate region from the backbone network and RPN, and converts them into fixed-size feature maps through the RoI Align technology. These feature maps are used for classification and bounding box regression (7x7) and mask generation (14x14), respectively.

[0071] The classification and bounding box regression branch is composed of a convolutional neural network, and its main task is to classify candidate regions and regress accurate bounding boxes. This branch receives the 7x7 feature map from the RoI layer, and outputs the category probability and accurate bounding box coordinates of each candidate region after a series of convolution operations.

[0072] The mask generation branch is also based on the convolutional neural network design, but focuses on generating high-precision segmentation masks. It receives a 14x14 feature map as input and uses a fully convolutional network (FCN) to classify candidate regions at the pixel level. In order to further improve the refinement of the segmentation boundary, this branch uses the PointRend algorithm. PointRend selects uncertain points (usually points with a predicted probability close to 0.5), extracts features from high-resolution feature maps, and uses a multi-layer perceptron for refined prediction, gradually improving the resolution of the segmentation result, and finally generating a segmentation mask with clear boundaries.

[0073] In summary, this model combines advanced deep learning techniques and optimization algorithms to provide an efficient blastocyst image instance segmentation solution.

[0074] In one embodiment, the above-mentioned step S135 may include steps S1351 - S1352 .

[0075] S1351, using the sample set in an end-to-end manner, fine-tuning the bottom two layers of the backbone feature extraction network by fixing the parameters of the ImageNet pre-trained model; S1352. During the network optimization process, stochastic gradient descent is used to optimize the multi-task loss function to obtain an instance segmentation model.

[0076] Specifically, in the human blastocyst image instance segmentation task, an end-to-end training method was used, which was trained based on a dataset constructed from a sample library to achieve accurate segmentation of individual cells in the blastocyst image. The design of the entire network architecture takes into account the generalization ability and segmentation accuracy of the model.

[0077] Human blastocyst image instance segmentation datasets carefully selected and annotated from the sample library. These datasets not only contain rich cell morphological changes, but also have detailed annotation information, which can effectively support the precise positioning of cell boundaries during model training.

[0078] The backbone feature extraction network was pre-trained on a large-scale dataset on ImageNet to obtain a wide range of general feature representation capabilities. On this basis, the parameters of the bottom two layers of the model were fixed to avoid overfitting due to the relatively small amount of training data, while other layers were fine-tuned to adapt to the task requirements specific to blastocyst images. This strategy not only takes advantage of the powerful feature extraction capabilities of the pre-trained model, but also improves the model's understanding and performance of specific tasks through fine-tuning.

[0079] In order to simultaneously optimize the model’s classification, bounding box regression, and mask generation capabilities, a multi-task loss function L is defined, which is expressed as: ; Represents the classification loss, and cross entropy loss is usually used to measure the difference between the model prediction category and the true label. It is the border loss, and smooth L1 loss is generally used to evaluate the deviation between the predicted bounding box and the true bounding box. Represents mask loss, which is used to evaluate the quality of segmentation mask, usually using binary cross entropy loss.

[0080] During the training process, in order to make the model converge faster and avoid falling into the local optimum too early, a two-stage learning rate adjustment strategy is adopted. In the initial stage, a higher learning rate is set to speed up the update of model parameters. As the training progresses, the learning rate is gradually reduced to fine-tune the model parameters and improve the accuracy of the model. This strategy helps to balance the rapid convergence in the early stage of training and the improvement of accuracy in the later stage.

[0081] In summary, this study aims to build an efficient and accurate human blastocyst image instance segmentation model through end-to-end training methods, reasonable backbone network pre-training and fine-tuning strategies, carefully designed multi-task loss functions, and dynamically adjusted learning rate strategies. This method can not only effectively improve the model's segmentation accuracy for blastocyst cells, but also reduce the time and resource costs required for model training to a certain extent.

[0082] S140: Output the segmentation result.

[0083] In this embodiment, the segmentation result is output to the terminal for display, such as Figure 7 shown.

[0084] The method of this embodiment trains a deep learning model by constructing a labeled sample library, thereby realizing automatic instance segmentation of the internal components of human blastocyst images. Through this process, the morphological information of the blastocyst can be extracted quickly and accurately, improving the efficiency of blastocyst quality assessment. Figure 8 and Fig. 9 As shown, compared with the traditional evaluation method, this method only needs to perform instance segmentation and annotation on part of the blastocyst images to complete the accurate segmentation of the internal components of the blastocyst, thereby significantly reducing labor costs and material consumption, and effectively reducing the errors that may be caused by human factors.

[0085] The above-mentioned blastocyst image instance segmentation method obtains the blastocyst image to be segmented and performs necessary preprocessing to improve the image quality and adaptability; the preprocessed blastocyst image is input into the trained instance segmentation model to achieve accurate segmentation of the internal components of the blastocyst such as the inner cell mass and the trophectoderm; the instance segmentation model amplifies the blastocyst image through data augmentation technology, and uses the amplified sample set to train the deep learning network; during the model training process, the annotated images in the sample set help the model learn the characteristics of different structures to improve the accuracy and robustness of the segmentation; finally, the model outputs the segmentation result of the blastocyst image, extracts the morphological characteristics of the inner cell mass and the trophectoderm, so as to support efficient and automated blastocyst quality analysis.

[0086] Figure 3 FIG. 3 is a schematic block diagram of a blastocyst image instance segmentation device 300 provided in an embodiment of the present invention. Figure 3 As shown, corresponding to the above blastocyst image instance segmentation method, the present invention also provides a blastocyst image instance segmentation device 300. The blastocyst image instance segmentation device 300 includes a unit for executing the above blastocyst image instance segmentation method, and the device can be configured in a server. Figure 3 The blastocyst image instance segmentation device 300 includes an image acquisition unit 301, a preprocessing unit 302, a segmentation unit 303 and an output unit 304.

[0087] An image acquisition unit 301 is used to acquire a blastocyst image to be segmented; a preprocessing unit 302 is used to preprocess the blastocyst image to be segmented to obtain a preprocessed blastocyst image to be segmented; a segmentation unit 303 is used to input the preprocessed blastocyst image to be segmented into an instance segmentation model to perform instance segmentation of internal components to obtain a segmentation result; wherein the instance segmentation model is a model obtained by amplifying the blastocyst image through data enhancement, forming a sample set after preprocessing, and using the sample set to train a deep learning network; an output unit 304 is used to output the segmentation result.

[0088] In one embodiment, the instance segmentation model training process includes: Hoffman modulation contrast microscopic blastocyst images on the fifth day after in vitro fertilization are obtained, and data enhancement processing is performed to obtain a blastocyst image sample library; specifically, Hoffman modulation contrast microscopic blastocyst images on the fifth day after in vitro fertilization are obtained, images that meet set conditions are excluded, and data enhancement processing is performed on the excluded images to obtain a blastocyst image sample library; wherein the images with set conditions include samples that fail to develop to the blastocyst stage after in vitro fertilization; samples that lack a clear equatorial plane surrounding cell layer and inner cell mass structure; and samples with incomplete blastocyst cells in the field of view.

[0089] The original images in the blastocyst image sample library are instance segmented and annotated to obtain a blastocyst image instance segmentation dataset; specifically, some original images are selected from the blastocyst image sample library using a random sampling method and divided into a training set and a test set; the boundaries of each component in the training set and the test set are marked using polygonal annotation technology, wherein the components include the outlines of cell clusters and trophectoderm cells; the marked training set and test set are saved in JSON format to obtain a blastocyst image instance segmentation dataset.

[0090] The blastocyst image instance segmentation dataset is preprocessed to obtain a sample set; specifically, the blastocyst image instance segmentation dataset is scaled to obtain a sample set.

[0091] Build deep learning networks; The deep learning network is trained using the sample set, and the network parameters are adjusted to optimize the performance of the network to obtain an instance segmentation model. The instance segmentation model includes a backbone feature extraction network, a feature pyramid network, a region proposal network, a region of interest layer, a classification and bounding box branch, and a mask branch; wherein the backbone feature extraction network uses Res2Net and DCN modules; the feature pyramid network is a feature pyramid network with a CARAFE upsampling operator; the region proposal network shares a feature extraction network with the backbone feature extraction network and is connected to a weighted connection network; the mask branch includes a PointRend algorithm. Specifically, the sample set is used in an end-to-end manner, and the backbone feature extraction network is fine-tuned using the ImageNet pre-trained model parameters to fix the bottom two layers; in the network optimization process, stochastic gradient descent is used to optimize the multi-task loss function to obtain an instance segmentation model.

[0092] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned blastocyst image instance segmentation device 300 and each unit can refer to the corresponding description in the aforementioned method embodiment, and for the convenience and brevity of description, it will not be repeated here.

[0093] The blastocyst image instance segmentation device 300 can be implemented in the form of a computer program. The computer program can be used in Figure 4 Runs on the computer device shown.

[0094] See also Figure 4 , Figure 4 5 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 500 may be a server, wherein the server may be an independent server or a server cluster composed of multiple servers.

[0095] See also Figure 4 The computer device 500 includes a processor 502 , a memory and a network interface 505 connected via a system bus 501 , wherein the memory may include a non-volatile storage medium 503 and an internal memory 504 .

[0096] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions, and when the program instructions are executed, the processor 502 can execute a blastocyst image instance segmentation method.

[0097] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500 .

[0098] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a blastocyst image instance segmentation method.

[0099] The network interface 505 is used to communicate with other devices over the network. Figure 4 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device 500 to which the solution of the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0100] The processor 502 is used to run the computer program 5032 stored in the memory to implement the following steps: Acquire a blastocyst image to be segmented; preprocess the blastocyst image to be segmented to obtain a preprocessed blastocyst image to be segmented; input the preprocessed blastocyst image to be segmented into an instance segmentation model to perform instance segmentation of internal components to obtain a segmentation result; wherein the instance segmentation model is a model obtained by amplifying the blastocyst image through data enhancement, forming a sample set after preprocessing, and using the sample set to train a deep learning network; and output the segmentation result.

[0101] In one embodiment, when the processor 502 implements the example segmentation model by amplifying the blastocyst image through data enhancement, forming a sample set after preprocessing, and using the sample set to train the deep learning network to obtain the model step, the following steps are specifically implemented: The method comprises the following steps: obtaining Hoffman modulation contrast microscopic blastocyst images on the fifth day after in vitro fertilization, performing data enhancement processing, and obtaining a blastocyst image sample library; performing instance segmentation and annotation on the original images in the blastocyst image sample library, and obtaining a blastocyst image instance segmentation dataset; preprocessing the blastocyst image instance segmentation dataset, and obtaining a sample set; constructing a deep learning network; training the deep learning network using the sample set, and adjusting network parameters to optimize network performance, and obtaining an instance segmentation model.

[0102] Among them, the instance segmentation model includes a backbone feature extraction network, a feature pyramid network, a region proposal network, an area of ​​interest layer, a classification and border branch, and a mask branch; wherein the backbone feature extraction network uses Res2Net and DCN modules; the feature pyramid network is a feature pyramid network with a CARAFE upsampling operator; the region proposal network shares a feature extraction network with the backbone feature extraction network and is connected to a weighted connection network; the mask branch includes a PointRend algorithm.

[0103] In one embodiment, when the processor 502 implements the step of obtaining a Hoffman modulation contrast microscopic blastocyst image on the fifth day after in vitro fertilization and performing data enhancement processing to obtain a blastocyst image sample library, the processor 502 specifically implements the following steps: Acquire Hoffman modulation contrast microscopic blastocyst images on the fifth day after in vitro fertilization, exclude images that meet set conditions, and perform data enhancement processing on the excluded images to obtain a blastocyst image sample library; Among them, the images with set conditions include samples that failed to develop to the blastocyst stage after in vitro fertilization; samples that lacked a clear equatorial cell layer and inner cell mass structure; and samples with incomplete blastocyst cells in the field of view.

[0104] In one embodiment, when the processor 502 implements the step of performing instance segmentation and annotation on the original images in the blastocyst image sample library to obtain a blastocyst image instance segmentation dataset, the processor 502 specifically implements the following steps: A portion of original images are selected from the blastocyst image sample library using a random sampling method and divided into a training set and a test set; the boundaries of each component in the training set and the test set are marked using polygonal annotation technology, wherein the components include the outlines of cell clusters and trophectoderm cells; the marked training set and test set are saved in JSON format to obtain a blastocyst image instance segmentation dataset.

[0105] In one embodiment, when the processor 502 implements the step of preprocessing the blastocyst image instance segmentation dataset to obtain a sample set, the processor 502 specifically implements the following steps: The blastocyst image instance segmentation dataset is scaled to obtain a sample set.

[0106] In one embodiment, when the processor 502 implements the step of training the deep learning network using the sample set and adjusting the network parameters to optimize the performance of the network to obtain the instance segmentation model, the processor 502 specifically implements the following steps: The sample set is used in an end-to-end manner to fine-tune the backbone feature extraction network by fixing the bottom two layers of the ImageNet pre-trained model parameters; during the network optimization process, stochastic gradient descent is used to optimize the multi-task loss function to obtain an instance segmentation model.

[0107] It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0108] It can be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.

[0109] Therefore, the present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein when the computer program is executed by a processor, the processor executes the following steps: Acquire a blastocyst image to be segmented; preprocess the blastocyst image to be segmented to obtain a preprocessed blastocyst image to be segmented; input the preprocessed blastocyst image to be segmented into an instance segmentation model to perform instance segmentation of internal components to obtain a segmentation result; wherein the instance segmentation model is a model obtained by amplifying the blastocyst image through data enhancement, forming a sample set after preprocessing, and using the sample set to train a deep learning network; and output the segmentation result.

[0110] In one embodiment, when the processor executes the computer program to implement the instance segmentation model, the blastocyst image is amplified by data enhancement, and a sample set is formed after preprocessing, and the model step of using the sample set to train the deep learning network is specifically implemented as follows: The method comprises the following steps: obtaining Hoffman modulation contrast microscopic blastocyst images on the fifth day after in vitro fertilization, performing data enhancement processing, and obtaining a blastocyst image sample library; performing instance segmentation and annotation on the original images in the blastocyst image sample library, and obtaining a blastocyst image instance segmentation dataset; preprocessing the blastocyst image instance segmentation dataset, and obtaining a sample set; constructing a deep learning network; training the deep learning network using the sample set, and adjusting network parameters to optimize network performance, and obtaining an instance segmentation model.

[0111] Among them, the instance segmentation model includes a backbone feature extraction network, a feature pyramid network, a region proposal network, an area of ​​interest layer, a classification and border branch, and a mask branch; wherein the backbone feature extraction network uses Res2Net and DCN modules; the feature pyramid network is a feature pyramid network with a CARAFE upsampling operator; the region proposal network shares a feature extraction network with the backbone feature extraction network and is connected to a weighted connection network; the mask branch includes a PointRend algorithm.

[0112] In one embodiment, when the processor executes the computer program to implement the step of obtaining a Hoffman modulation contrast microscopic blastocyst image on the fifth day after in vitro fertilization and performing data enhancement processing to obtain a blastocyst image sample library, the processor specifically implements the following steps: Acquire Hoffman modulation contrast microscopic blastocyst images on the fifth day after in vitro fertilization, exclude images that meet set conditions, and perform data enhancement processing on the excluded images to obtain a blastocyst image sample library; Among them, the images with set conditions include samples that failed to develop to the blastocyst stage after in vitro fertilization; samples that lacked a clear equatorial cell layer and inner cell mass structure; and samples with incomplete blastocyst cells in the field of view.

[0113] In one embodiment, when the processor executes the computer program to implement the step of performing instance segmentation and annotation on the original images in the blastocyst image sample library to obtain a blastocyst image instance segmentation dataset, the processor specifically implements the following steps: A portion of original images are selected from the blastocyst image sample library using a random sampling method and divided into a training set and a test set; the boundaries of each component in the training set and the test set are marked using polygonal annotation technology, wherein the components include the outlines of cell clusters and trophectoderm cells; the marked training set and test set are saved in JSON format to obtain a blastocyst image instance segmentation dataset.

[0114] In one embodiment, when the processor executes the computer program to implement the step of preprocessing the blastocyst image instance segmentation dataset to obtain a sample set, the processor specifically implements the following steps: The blastocyst image instance segmentation dataset is scaled to obtain a sample set.

[0115] In one embodiment, when the processor executes the computer program to implement the step of training the deep learning network using the sample set and adjusting the network parameters to optimize the performance of the network to obtain the instance segmentation model, the processor specifically implements the following steps: The sample set is used in an end-to-end manner to fine-tune the backbone feature extraction network by fixing the bottom two layers of the ImageNet pre-trained model parameters; during the network optimization process, stochastic gradient descent is used to optimize the multi-task loss function to obtain an instance segmentation model.

[0116] The storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, etc., which are computer-readable storage media that can store program codes.

[0117] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0118] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0119] The steps in the method of the embodiment of the present invention can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present invention can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0120] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0121] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for segmenting blastocyst images, characterized in that: include: Acquire an image of a blastocyst to be segmented; Preprocessing the blastocyst image to be segmented to obtain a preprocessed blastocyst image to be segmented; The preprocessed blastocyst image to be segmented is input into the instance segmentation model to perform instance segmentation of the internal components to obtain a segmentation result; wherein the instance segmentation model is a model obtained by amplifying the blastocyst image through data augmentation, forming a sample set after preprocessing, and using the sample set to train a deep learning network; The segmentation result is output.

2. The blastocyst image instance segmentation method according to claim 1, characterized in that: The instance segmentation model is a model obtained by amplifying the blastocyst image by data enhancement, forming a sample set after preprocessing, and using the sample set to train the deep learning network, including: Obtain Hoffman modulation contrast microscopic blastocyst images on the fifth day after in vitro fertilization, and perform data enhancement processing to obtain a blastocyst image sample library; Performing instance segmentation and annotation on the original images in the blastocyst image sample library to obtain a blastocyst image instance segmentation dataset; Preprocessing the blastocyst image instance segmentation dataset to obtain a sample set; Build deep learning networks; The deep learning network is trained using the sample set, and network parameters are adjusted to optimize network performance to obtain an instance segmentation model.

3. The blastocyst image instance segmentation method according to claim 2, characterized in that: The method of obtaining a Hoffman modulation contrast microscopic blastocyst image on the fifth day after in vitro fertilization and performing data enhancement processing to obtain a blastocyst image sample library includes: Acquire Hoffman modulation contrast microscopic blastocyst images on the fifth day after in vitro fertilization, exclude images that meet set conditions, and perform data enhancement processing on the excluded images to obtain a blastocyst image sample library; Among them, the images with set conditions include samples that failed to develop to the blastocyst stage after in vitro fertilization; samples that lacked a clear equatorial cell layer and inner cell mass structure; and samples with incomplete blastocyst cells in the field of view.

4. The blastocyst image instance segmentation method according to claim 3, characterized in that: The performing instance segmentation and annotation on the original images in the blastocyst image sample library to obtain a blastocyst image instance segmentation dataset includes: Selecting some original images from the blastocyst image sample library by random sampling method and dividing them into a training set and a test set; The polygon annotation technology is used to mark the boundaries of each component in the training set and the test set, where the components include the outlines of cell clusters and trophectoderm cells; Save the labeled training and test sets in JSON format to obtain the blastocyst image instance segmentation dataset.

5. The blastocyst image instance segmentation method according to claim 2, characterized in that: The preprocessing of the blastocyst image instance segmentation dataset to obtain a sample set includes: The blastocyst image instance segmentation dataset is scaled to obtain a sample set.

6. The blastocyst image instance segmentation method according to claim 2, characterized in that: The instance segmentation model includes a backbone feature extraction network, a feature pyramid network, a region proposal network, an area of ​​interest layer, a classification and border branch, and a mask branch; wherein the backbone feature extraction network uses Res2Net and DCN modules; the feature pyramid network is a feature pyramid network with a CARAFE upsampling operator; the region proposal network shares a feature extraction network with the backbone feature extraction network and is connected to a weighted connection network; the mask branch includes a PointRend algorithm.

7. The blastocyst image instance segmentation method according to claim 6, characterized in that: The method of training the deep learning network using the sample set and adjusting network parameters to optimize network performance to obtain an instance segmentation model includes: Using the sample set in an end-to-end manner, fine-tuning the backbone feature extraction network with the ImageNet pre-trained model parameters fixed at the bottom two layers; During the network optimization process, stochastic gradient descent is used to optimize the multi-task loss function to obtain the instance segmentation model.

8. A blastocyst image instance segmentation device, characterized in that: include: An image acquisition unit, used for acquiring an image of the blastocyst to be segmented; A preprocessing unit, used for preprocessing the blastocyst image to be segmented to obtain a preprocessed blastocyst image to be segmented; A segmentation unit, used for inputting the preprocessed blastocyst image to be segmented into an instance segmentation model to perform instance segmentation of internal components to obtain a segmentation result; wherein the instance segmentation model is a model obtained by amplifying the blastocyst image through data augmentation, forming a sample set after preprocessing, and using the sample set to train a deep learning network; An output unit is used to output the segmentation result.

9. A computer device, characterized in that: The computer device comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images