Embryo development detection device and training platform thereof

Abstract

The present application relates to a kind of embryo development detection device and its training platform, including embryo image enhancement module, embryo image training set construction module, model total loss judging module, parameter adjustment module, wherein corresponding to the embryo image of two different embryo development stages is fused by weighted addition method after image, and is used to construct the embryo feature training set corresponding to embryo feature;According to the total loss calculated after sending embryo feature training set into embryo target detection initial model, at least one parameter in embryo target detection initial model is modified, to generate the embryo target detection model corresponding to embryo feature, so that multiple predetermined features in embryo image can be accurately and quickly identified.The present application also relates to the embryo development detection device generated using the above training platform.

Description

TECHNICAL FIELD

[0001] The present application relates to embryo development detection based on time-lapse photography, in particular to an embryo development detection device and a training platform thereof. BACKGROUND

[0002] It has become one of the effective ways to reduce costs and increase efficiency in the medical field to use AI artificial intelligence technology to identify image data. Some people have applied artificial intelligence algorithms to replace embryologists to analyze time-lapse data of embryo culture to achieve automatic embryo growth and development state evaluation.

[0003] Such time-lapse photography equipment is known in the prior art, which combines a high-resolution camera with an embryo incubator to continuously and dynamically monitor the embryo development process at a specific interval, frequency, angle, etc. and take pictures, without frequent interference with the environment in the embryo incubator, so as to perform morphological observation and analysis on the embryo development process.

[0004] For this purpose, it is necessary to accurately identify and measure specific features and parameters indicating the embryo development state in the time sequence images of the embryo taken by the time-lapse photography equipment, such as identifying pronucleus, nucleolus, cell number, cell shape and area, etc.

[0005] In the prior art, in order to realize specific target detection in images, a large number of bounding boxes that may contain target objects are generated in candidate regions, and then a classifier is used to determine whether each bounding box contains a target object and the probability or confidence of the target object belonging to a certain category. Through post-processing, the bounding box is improved and the repeated detection target is eliminated. For example, CN111539308A discloses a target detection neural network using Faster RCNN to locate the bounding box of the blastomere region in the embryo image. Faster RCNN first extracts the feature map of the embryo image using a convolutional neural network, obtains multiple proposal regions through a region proposal network, uses an ROI-Align layer to pool each proposal region, and finally performs bounding box regression and logistic regression through a fully connected layer to obtain the bounding box of the blastomere region.

[0006] In the above-mentioned target detection system using sliding window or candidate region, the classifier can only obtain local information of the image, so it cannot well utilize context information when detecting objects, which easily leads to predicting wrong object information on the background, resulting in unreliable embryo image recognition results that require accuracy. SUMMARY

[0007] The purpose of the present application is to provide a target detection model for fast and accurate identification of small targets in embryo images, in particular, and to provide a training platform for quickly training multiple different target detection models from the same data set.

[0008] The first aspect of the present application relates to a training platform of an embryo development detection device, comprising an embryo image enhancement module, an embryo image training set construction module, a model total loss judgment module, and a parameter adjustment module, wherein the embryo image enhancement module receives two image groups of an embryo, the two image groups respectively comprising a plurality of embryo images obtained by time-lapse photography on the embryo, and the two image groups respectively correspond to two different embryo development stages; and the embryo image enhancement module performs image fusion on the plurality of embryo images of the two image groups by weighted addition, and takes the plurality of embryo images as input images of the embryo image training set construction module; the embryo image training set construction module labels the input images according to selected embryo features, so as to construct an embryo feature training set corresponding to the embryo features; after the embryo feature training set is sent into an embryo target detection initial model, the model total loss judgment module calculates the total loss of the embryo target detection initial model according to a loss function; and the parameter adjustment module modifies at least one parameter in the embryo target detection initial model according to the total loss, so as to generate an embryo target detection model M corresponding to the embryo features.

[0009] Preferably, the model total loss judgment module calculates the total loss by using a BCEWithLogitsLoss loss function and a CIOU loss function.

[0010] Preferably, the parameter adjustment module calculates the gradient of at least one parameter in the embryo target detection initial model M by back propagation, and optimizes the parameter by using an Adam optimization algorithm.

[0011] Preferably, the embryo target detection initial model is a path aggregation network, which comprises slice layer, first down-sampling layer, second down-sampling layer, third down-sampling layer, spatial pyramid layer, first up-sampling layer, second up-sampling layer, first output layer, second output layer, third output layer and detection layer connected in sequence; wherein the slice layer receives the embryo feature training set; and the first up-sampling layer comprises first residual convolution module and first up-sampling module connected in sequence, and the output image of the first up-sampling module is connected with the output of the third down-sampling layer as the input image of the second sampling layer; the second up-sampling layer comprises second residual convolution module and second up-sampling module connected in sequence, and the output image of the second up-sampling module is the input image of the first output layer; the first output layer comprises first output module, and the output image of the second up-sampling module is connected with the output of the second down-sampling module as the input image of the first output module; the second output layer comprises second output module, and the output image of the first output module is connected with the output of the second residual convolution module as the input image of the second output module after convolution operation; the third output layer comprises third output module, and the output image of the second output module is connected with the output of the first residual convolution module as the input image of the third output module after convolution operation; the detection layer comprises detection module, and the first output layer, second output layer and third output layer are respectively the input image of the detection layer.

[0012] The second aspect of the present application relates to an embryo development detection device, which receives an embryo image obtained by time-lapse photography on an embryo, and outputs feature classification information and positioning information of the embryo image by using a pre-trained embryo target detection model, the embryo target detection model comprising a slice layer, a first down-sampling layer, a second down-sampling layer, a third down-sampling layer, a spatial pyramid layer, a first up-sampling layer, a second up-sampling layer, a first output layer, a second output layer and a third output layer connected in sequence; wherein: the slice layer receives the embryo feature training set; and the first up-sampling layer comprises a first residual convolution module and a first up-sampling module connected in sequence, and the output image of the first up-sampling module is connected with the output of the third down-sampling layer as the input image of the second sampling layer; the second up-sampling layer comprises a second residual convolution module and a second up-sampling module connected in sequence, and the output image of the second up-sampling module is used as the input image of the first output layer; the first output layer comprises a first output module, and the output image of the second up-sampling module is connected with the output of the second down-sampling module as the input image of the first output module; the second output layer comprises a second output module, and the output image of the first output module is connected with the output of the second residual convolution module after convolution operation as the input image of the second output module; the third output layer comprises a third output module, and the output image of the second output module is connected with the output of the first residual convolution module after convolution operation as the input image of the third output module; and the detection layer comprises a detection module, and the first output layer, the second output layer and the third output layer are used as the input image of the detection layer respectively.

[0013] By the present application, efficient extraction of embryo in vitro culture stage image data can be realized, the workload of embryo experts in judging data acquisition process is reduced, and work efficiency is improved.

[0014] Compared with the prior art, the target detection network model of the present application can directly predict the coordinates of the bounding box, the confidence of the object contained in the bounding box and the class probability from a whole image. Since the target detection network model of the present application can always see the information of a whole image during training and testing, the context information can be well utilized when detecting objects, so that it is not easy to predict false object information on the background. In addition, since the object detection process is completed in a neural network, the object detection performance can be optimized in an end-to-end manner. In addition, compared with Faster RCNN, the initial model of the target detection network of the present application can reduce background errors and is less likely to predict non-existing objects on the background. BRIEF DESCRIPTION OF DRAWINGS

[0015] The above summary of the application and the following detailed description of the application will be better understood when read in conjunction with the accompanying drawings, in which like reference numerals refer to identical or similar elements in the various figures. It should be understood that the drawings are not necessarily to scale. The drawings are merely schematic representations, not intended to portray specific structural details of the application. In the drawings:

[0016] Figure 1 A schematic diagram showing the composition of the training platform 100 of the embryo development detection device of the application.

[0017] Figure 2 A schematic diagram showing the overall structure of the embryo target detection initial model M0 and the embryo target detection model M1 of the application.

[0018] Figure 3 A schematic diagram showing the slice layer structure of the embryo target detection initial model M0 and the embryo target detection model M1 of the application.

[0019] Figure 4 A schematic diagram showing the first down-sampling layer structure of the embryo target detection initial model M0 and the embryo target detection model M1 of the application.

[0020] Figure 5 A schematic diagram showing the second down-sampling layer and the third down-sampling layer structure of the embryo target detection initial model M0 and the embryo target detection model M1 of the application.

[0021] Figure 6 A schematic diagram showing the spatial pyramid layer structure of the embryo target detection initial model M0 and the embryo target detection model M1 of the application.

[0022] Figure 7 A schematic diagram showing the first residual convolution module and the second residual convolution module structure of the embryo target detection initial model M0 and the embryo target detection model M1 of the application.

[0023] Figure 8 A schematic diagram showing the first output module, the second output module, the third output module structure of the embryo target detection initial model M0 and the embryo target detection model M1 of the application.

[0024] Figure 9 A schematic diagram showing the detection module D structure of the embryo target detection initial model M0 and the embryo target detection model M1 of the application.

[0025] Figure 10 An original input image and an output image after detection of the embryo target detection of the application are shown. DETAILED DESCRIPTION

[0026] The specific features and advantages of the present application are described in detail in the detailed description of the embodiments below, which are sufficient to enable any person skilled in the art to understand the technical content of the present application and to implement it, and the person skilled in the art can easily understand the related purposes and advantages of the present application according to the description, claims and drawings disclosed in the specification.

[0027] In the prior art, by performing time-lapse photography on the embryo to cover multiple embryo stages, a raw image set PP composed of multiple images of the embryo is obtained. Such a time-lapse photography device is known in the prior art, which combines a high-resolution camera with an embryo incubator to continuously and dynamically monitor the development process of the embryo at a specific interval, frequency, angle, etc. and take images, without frequent interference with the environment in the embryo incubator, so as to enable morphological observation and analysis of the development process of the embryo.

[0028] The first aspect of the present application relates to a training platform 100 of an embryo development detection device. As shown in the figure, the training platform 100 comprises an embryo image enhancement module 1, an embryo image training set construction module 2, a model total loss judgment module 3, and a parameter adjustment module 4. Figure 1

[0029]

Embryo image enhancement module

[0030] The embryo image training set construction module 2 is used to label the input image according to the selected embryo features and construct an embryo feature training set corresponding to the embryo features.

[0031] The embryo image enhancement module 1 receives at least two image groups Ga, Gb of the embryo, the two image groups Ga, Gb respectively comprising a plurality of embryo images obtained by time-lapse photography on the embryo, and the two image groups Ga, Gb respectively corresponding to two different embryo development stages; and the embryo image enhancement module 1 performs image fusion on the plurality of embryo images of the two image groups Ga, Gb by weighted addition, and takes the image fusion result as the input image of the embryo image training set construction module 2.

[0032] ​Any two embryo images from different embryo stages can be fused, and the fusion ratio is lam: 0.7. When adding, the pixel values corresponding to the two embryo images are directly added. That is, inputs = 0.7 * images + 0.3 * images_random, wherein images can be images from image group Ga, corresponding to the first stage of embryo development, and images_random can be images from image group Gb, corresponding to images of the remaining stages of embryo development different from the first stage, for example, images of the fourth stage of embryo development. In this way, the blurring of images after the fourth stage of embryo development is improved, and the accuracy of embryo image recognition after the fourth stage of embryo development is increased. On this basis, in the subsequent training stage, the label information of any two embryo images of different embryo stages is also fused to construct a new data set for training, thereby improving the generalization ability of the model.

[0033] In addition, the embryo image enhancement module 1 can also perform various enhancement processing on the embryo images, including but not limited to image generation, image fusion, image size change, rotation, folding, etc.

[0034] In an embodiment of the present application, the embryo image enhancement module can normalize the pixel data of each embryo image to [-1, 1]. Specifically, the mean and variance of the image data can be calculated, and then the original data is subtracted from the mean divided by the variance, so that the data is normalized to [-1, 1], so that the model is easier to converge; on this basis, the normalized data can also be data enhanced, including: specifying a probability of 0.5, flipping the image vertically or horizontally.

[0035] In another embodiment of the present application, the embryo image enhancement module 1 can automatically adapt the embryo image with a pixel size ranging from 320 to 960 pixels to a 640*640 pixel image matching the embryo target detection initial model M0 adopted by the present application, which makes it possible to train the model for embryo original images of different pixel sizes, avoids information loss caused by filling or scaling of the original image, and improves the detection accuracy of the embryo target detection model M1 generated by the present application for small targets in the embryo.

[0036]

Embryo image training set construction module

[0037] The embryo image training set construction module 2 performs various annotations on the time-lapse video data to generate a data set.

[0038] According to different features of interest, the embryo image data processed by the embryo image enhancement module is labeled, and the predetermined selected labeling content may include, for example, the time point of cell division, including the first cleavage time, the second cleavage time, the third cleavage time, or the 1-cell to 2-cell, 3-cell, 4-cell, 5-cell time; cell area (cell area at the time of division); pronucleus area (female pronucleus and male pronucleus), pronucleus generation time, pronucleus disappearance time, 8 hours before pronucleus disappearance, area of female and male pronuclei; nucleolus number, nucleolus distribution three hours before pronucleus disappearance, etc.

[0039] The labeled data generates a corresponding data set, including: a division time point data set, a cell area data set, a female and male pronucleus change time point data set, a pronucleus area data set, a nucleolus number data set, and a nucleolus distribution mode data set.

[0040] In the present application, the image standard can be completed manually by a physician or trained staff, or the embryo image can be labeled by using an automatic labeling module.

[0041]

Target detection network initial model

[0042] The overall structure of the target detection network initial model M0 in the present application is as shown in Figure 2

[0043] After the embryo feature training set is sent into the embryo target detection initial model M0, the model total loss judgment module 3 calculates the total loss of the embryo target detection initial model M0 according to the loss function, and the parameter adjustment module 4 modifies at least one parameter in the embryo target detection initial model M0 according to the total loss, to generate an embryo target detection model M1 corresponding to the embryo feature. Therefore, the target detection initial model M0 and the embryo target detection model M1 generated by the training platform involved in the first aspect of the present application have the same structure, but the parameter settings of each component part are different.

[0044] The target detection network initial model M0 uses the PANET sampling method for information sampling, including: increasing the bottom-up path expansion method, fully utilizing the embryo image information, enhancing the image features, and avoiding information loss.

[0045] Referring to Figure 2 , the embryo target detection initial model M0 is a path aggregation network, which includes, in sequence, a slice layer, a first down-sampling layer, a second down-sampling layer, a third down-sampling layer, a spatial pyramid layer, a first up-sampling layer, a second up-sampling layer, a first output layer, a second output layer, a third output layer, and a detection layer, i.e., the output of each layer is taken as the input of the next layer.

[0046] ​The slice layer receives an embryo feature training set; and the first up-sampling layer includes a first residual convolution module RES1 and a first up-sampling module LOS1 connected in sequence, and the output image of the first up-sampling module LOS is connected with the output of the third down-sampling layer to serve as the input image of the second sampling layer; the second up-sampling layer includes a second residual convolution module RES2 and a second up-sampling module LOS2 connected in sequence, and the output image of the second up-sampling module LOS serves as the input image of the first output layer;

[0047] The first output layer includes a first output module LO1, and the output image of the second up-sampling module LOS2 is connected with the output of the second down-sampling module LUS2 to serve as the input image of the first output module LO1;

[0048] The second output layer includes a second output module LO2, and the output image of the first output module LO1 after a convolution operation conv is connected with the output of the second residual convolution module RES2 to serve as the input image of the second output module LO2;

[0049] The third output layer includes a third output module LO3, and the output image of the second output module LO2 after a convolution operation conv is connected with the output of the first residual convolution module RES1 to serve as the input image of the third output module LO3;

[0050] The detection layer includes a detection module D, and the output images of the first output layer, the second output layer and the third output layer serve as the input images of the detection module.

[0051] Each layer and its composition will be described in detail below.

[0052]

Slice Layer

[0053] The slice layer includes a slice module F, which includes a slice operation Slice, a connection operation Concat, and a slice layer CBS operation, wherein:

[0054] The slice operation: first receives an original input image with a size of 640*640 and a channel number of 3, and performs a three-channel slicing operation on the original input image to generate four slices with a size of 320*320;

[0055] In the connection operation concat, the above four slices with a size of 320*320 are connected, that is, the same size images are expanded in the channel to obtain a 320*320 image with 12 channels;

[0056] Then, a CBS operation with a setting of {32*12*3*3} is performed. The CBS operation includes three steps: convolution (C), normalization (B), and SiLu (S). Here, 32 represents the 32 convolution kernels used in the convolution operation, 12 represents the 12 input channels, and 3*3 represents the kernel size. Normalization (B) and SiLu (S) are known in the prior art. Finally, a 320*320 image with 32 channels is obtained.

[0057] [First downsampling layer]

[0058] The first downsampling layer includes the first downsampling module LUS1, see [link / reference] Figure 4 The first downsampling module LUS1 includes a convolution operation (conv), a single residual network composed of multiple CBS operations, pixel stacking (add), a connection operation (concat), and an additional CBS operation, wherein:

[0059] In the convolution operation conv, a 320*320 image with 32 channels from the slice layer is received and passed through a convolution operation with 64 convolution kernels, 32 channels, and a 3*3 kernel size to obtain a 160*160 image with 64 channels.

[0060] A single residual network is composed of four CBS operations, such as Figure 4 As shown:

[0061] The first CBS operation C1 of the downsampling layer receives a 64-channel 160*160 image output from the convolution operation conv, which is set to {32*64*1*1}, and its output is a 32-channel 160*160 image.

[0062] The second CBS operation C2 of the downsampling layer receives the 32-channel 160*160 image output by the first CBS operation C2 of the downsampling layer, which is set to {32*32*1*1}, and its output is a 32-channel 160*160 image.

[0063] The third CBS operation C3 of the downsampling layer receives a 32-channel 160*160 image output from the second CBS operation C2 of the downsampling layer, which is set to {32*32*3*3}, and its output is a 32-channel 160*160 image.

[0064] The fourth CBS operation C4 of the downsampling layer, like the first CBS operation C1 of the downsampling layer, receives a 64-channel 160*160 image output by the convolution operation conv of the downsampling layer and is set to {32*64*1*1}, and its output is a 32-channel 160*160 image.

[0065] In the pixel superposition add step, the 32-channel 160*160 image output by the third CBS operation C3 of the down-sampling layer is pixel superposed with the 32-channel 160*160 image output by the first CBS operation C1 of the down-sampling layer, and a 32-channel 160*160 image is output;

[0066] The connection operation contact: the 32-channel 160*160 image after pixel superposition is connected with the 32-channel 160*160 image output by the fourth CBS operation C4 of the down-sampling layer, and a 64-channel 160*160 image is output;

[0067] Then, the CBS operation is performed again, and the 64-channel 160*160 image output by the connection operation is subjected to the CBS operation with the setting of {64*64*1*1}, and a 64-channel 160*160 image is output as the output of the down-sampling layer 1.

[0068]

Second down-sampling layer

[0069] The second down-sampling layer includes a second down-sampling module LUS2, as shown in Figure 5 .

[0070] For the second down-sampling module LUS2, 20-29 are respectively set as: {64*128*3*3}, {64*128*1*1}, {64*64*1*1}, {64*64*3*3}, {64*64*1*1}, {64*64*3*3},

[0071] {64*64*1*1}, {64*64*3*3}, {128*128*1*1}, so that the 64-channel 160*160 image from the first down-sampling layer is subjected to a 128-convolution kernel 3*3 convolution operation to obtain a 128-channel 80*80 image, and then a three-layer residual network structure is used to perform a CBS operation on the 128-channel 80*80 image, and finally an 80*80 image with a channel of 128 is output.

[0072]

Third down-sampling layer

[0073] The third down-sampling layer includes a third down-sampling module LUS3, which has the same structure as the second down-sampling module LU2, as shown in Figure 5 .

[0074] For the third down-sampling module LUS3, its settings 30-39 are respectively: {128*256*3*3}, {128*256*1*1}, {128*128*1*1}, {128*128*3*3}, {128*128*1*1}, {128*128*3*3}, {128*128*1*1}, {128*128*3*3}, {245*256*1*1}, so as to realize that the 128-channel 80*80 image from the second down-sampling module LU2 is subjected to a 256-kernel 3*3 convolution operation, a 256-channel 80*80 image is obtained, and then the 256-channel 80*80 image is subjected to a CBS operation once by using a three-layer residual network structure, and finally a 256-channel 40*40 image is output.

[0075]

Spatial pyramid layer

[0076] The spatial pyramid layer includes a spatial pyramid module LSPP, as shown in Figure 6 .

[0077] The spatial pyramid module LSPP receives the 256-channel 40*40 image from the third down-sampling layer, and is subjected to a convolution operation conv with a kernel size of 3*3 and a kernel number of 512, so as to obtain a 512-channel 20*20 image;

[0078] Then, a CBS operation with a setting of {256*512*1*1} is adopted, so as to obtain a 256-channel 20*20 image;

[0079] Then, a maximum pooling operation and a deep concatenation are respectively performed, so as to obtain a 1024-channel 20*20 image; finally, a CBS operation with a setting of {512*1024*1*1} is performed, so as to output a 512-channel 20*20 image.

[0080]

First up-sampling layer

[0081] The first up-sampling layer includes a first residual convolution module RES1 and a first up-sampling module LOS1 in sequence.

[0082] For the first residual convolution module RES1, referring to Figure 7 , its settings 40-45 are respectively: {256*512*1*1}, {256*256*1*1}, {256*256*3*3}, {512*512*1*1}, {256*512*1*1}, {256*512*1*1}; so as to receive the 512-channel 20*20 image output by the spatial pyramid layer and output a 256-channel 20*20 image.

[0083] In the first up-sampling module LOS1, the 256-channel 20*20 image is further up-sampled by a convolution kernel of 2*2 to output a 256-channel 40*40 image.

[0084] Thereafter, the 256-channel 40*40 image output by the third down-sampling layer LUS3 is connected with the 256-channel 40*40 image output by the first up-sampling module LOS1 through a connection operation concat to output a 512-channel 40*40 image.

[0085]

Second up-sampling layer

[0086] The second up-sampling layer includes a second residual convolution module RES2 and a second up-sampling module LOS2.

[0087] The second residual convolution module RES2 has the same structure as the first residual convolution module RES1, as shown in Figure 7 For the second residual convolution module RES2, the parameters 50~50 are set as {128*512*1*1}, {128*128*1*1}, {128*128*3*3}, {256*256*1*1}, {128*256*1*1}, and {256*512*1*1}, so that the second residual convolution module RES2 receives the 512-channel 40*40 image output by the first up-sampling layer and outputs a 128-channel 40*40 image. In the second up-sampling operation LOS2, the 128-channel 40*40 image is further up-sampled by a convolution kernel of 2*2 to convert the 128-channel 40*40 image into a 128-channel 80*80 image.

[0088]

First output layer

[0089] The first output layer includes a connection operation concat and a first output module LO1.

[0090] First, the first output module LO1 receives the 128-channel 80*80 image output by the second up-sampling layer. In the connection operation concat, the 128-channel 80*80 image output by the second down-sampling layer is connected with the 128-channel 80*80 image output by the second up-sampling module LOS2 to obtain a 256-channel 80*80 image.

[0091] For the first output module LO1, refer to Figure 8 The parameters 60~64 are set as {64*256*1*1}, {64*64*1*1}, {64*64*3*3}, {128*128*1*1}, and {64*256*1*1}, so that in the first output module LO1, the 256-channel 80*80 image is subjected to multiple CBS operations and combinations thereof, as shown in Figure 8 to output a 128-channel 80*80 image.

[0092] SECOND OUTPUT LAYER

[0093] The second output layer comprises a first convolution operation conv, a first connection operation concat and a second output module LO2 in sequence.

[0094] In the convolution operation conv, the 128-channel 80*80 image from the first output module LO1 is subjected to a convolution operation with 128 convolution kernels of size 3*3 and step 2, to obtain a 128-channel 40*40 image.

[0095] In the connection operation concat, the above-mentioned 128-channel 40*40 image is connected with the 128-channel 40*40 image of the second residual convolution module RES2 of the second up-sampling layer, to obtain a 256-channel 40*40 image.

[0096] The second output module LO2 has the same structure as the first output module LO1, as shown in Figure 8 For the second output module LO2, the settings 70-74 are respectively {128*256*1*1}, {128*128*1*1}, {128*128*3*3}, {256*256*1*1}, {128*128*1*1}, {128*256*1*1}, so that in the second output module LO2, the above-mentioned 256-channel 40*40 image is subjected to multiple CBS operations and combinations thereof as shown in Figure 8 to output a 256-channel 40*40 image.

[0097] THIRD OUTPUT LAYER

[0098] The third output layer comprises a first convolution operation conv, a first connection operation concat and a third output module LO3 in sequence.

[0099] In the convolution operation conv, the 256-channel 40*40 image from the second output module LO2 is subjected to a convolution operation with 256 convolution kernels of size 3*3 and step 2, to obtain a 256-channel 20*20 image.

[0100] In the connection operation concat, the above-mentioned 256-channel 20*20 image is connected with the 256-channel 20*20 image of the first residual convolution module RES1 of the first up-sampling layer, to obtain a 512-channel 20*20 image.

[0101] The third output module LO3 has the same structure as the first output module LO1 and the second output module LO2, as shown in Figure 8 ​As shown, for the third output module LO2, its settings 80-84 are {256*512*1*1}, {256*256*1*1}, {256*256*3*3}, {512*512*1*1}, and {256*512*1*1}, respectively; thus, in the third output module LO2, the above 256-channel 40*40 image is processed as follows... Figure 8 After multiple CBS operations and their combinations shown, a 512-channel 20*20 image is output.

[0102] [Detection Layer]

[0103] like Figure 9 As shown, the detection layer includes a detection layer module D, which includes three anchor frame layers DML1, DML2, and DML3. The receptive fields of the three anchor frame layers are 8*8, 16*16, and 32*32, respectively, corresponding to the 80*80 feature map grids of the first output layer, the 40*40 feature map grids of the second output layer, and the 20*20 feature map grids of the third output layer.

[0104] Each anchor frame layer receives output images from the first output layer, the second output layer, and the third output layer, and each anchor frame layer DML1, DML2, and DML3 has three prediction boxes.

[0105] The following description uses the example of the first anchor frame layer DML1 receiving the output image from the first output layer. The 128-channel 80*80 image output by the first output layer is sent to the first anchor frame layer DML1. The first anchor frame layer DML1 uses its three prediction boxes F11, F12, and F13 to predict the 128-channel 80*80 image output by the first output layer. That is, each grid in the 80*80 feature map grid will be convolved and recognized by the three prediction boxes F11, F12, and F13 of the first anchor frame layer DML1. Therefore, the output image of the first output layer will obtain 3*80*80 prediction results after passing through the first anchor frame layer DML1. Each prediction result includes the target category, the target center point and its width and height coordinates, and the confidence score.

[0106] That is, the first anchor frame layer DML1 obtains 80*80 feature map grids of 128-channel 80*80 image of the first output layer. The receptive field of each feature map grid is 640 / 80=8*8, which is responsible for detecting small targets and outputs information of 3*80*80 small-sized detected targets.

[0107] Similarly, the second anchor frame layer DML2 obtains a 40*40 feature map grid of 128 channels of 80*80 images of the first output layer, and each feature map grid has a receptive field of 640 / 40 = 16*16 size, responsible for detecting medium targets; the 40*40 feature map grid is combined with three prediction boxes F21, F22, F23 in the second layer anchor frame layer DML2 to perform convolution operation, and information of 3*40*40 medium size detection targets is output.

[0108] Similarly, the third anchor frame layer DML3 obtains a 20*20 feature map grid of 128 channels of 80*80 images of the first output layer, and each feature map grid has a receptive field of 640 / 20 = 32*32 size, responsible for detecting large targets; the 20*20 feature map grid is combined with three prediction boxes F31, F32, F33 in the third layer anchor frame layer DML3 to perform convolution operation, and information of 3*20*20 large size detection targets is output.

[0109] Finally, according to the confidence obj of each grid output by the detection layer, it is judged whether the grid contains an embryo target. Specifically, the threshold of the confidence obj is set to 0.5, that is, the target with a confidence ob not exceeding 0.5 is removed; according to the non-maximum suppression algorithm, the prediction box in the target information is screened and removed, the target that repeatedly detects the rectangular frame of the same embryo target is removed, and finally according to the classification probability value in the target information, the target with the largest embryo target probability value is reserved information, and the corresponding embryo class cls, confidence obj, center point and width and height coordinates of the embryo target are output to generate a rectangular frame D1.

[0110] As Figure 10 (a) shows the original image sent into the embryo development detection device of the present application, Figure 10 (c) shows a detection result of the embryo development detection device of the present application, for example, a rectangular frame D1 generated based on the first output layer represents the size and accurate position of the embryo. Specifically, the upper left corner of the rectangular frame D1 is labeled as "t3", which is the embryo classification cls corresponding to the embryo image detected by the embryo development detection device of the present application, "0.92" is the confidence obj representing the classification result, and the rectangular frame D1 is generated according to the center point coordinates and width and height output by the detection model.

[0111]

Model total loss judgment module

[0112] The model total loss judgment module 3 uses the BCEWithLogitsLoss loss function and the CIOU loss function to calculate the total loss loss of the embryo.

[0113] Specifically, in the present application, the model total loss is composed of three loss modules, which are embryo rectangular frame loss box_loss, confidence loss obj_loss and embryo classification probability loss cls_loss, and the embryo total loss Loss is the weighted sum of the above three losses.

[0114] The rectangular frame represents the size and accurate position of the embryo, and the rectangular frame loss box_loss is used to calculate the distance error between the predicted embryo rectangular frame D1 output by the detection layer and the rectangular frame coordinates in the embryo label in the actual picture.

[0115] The rectangular frame loss box_loss adopts a CIOU loss function, and its formula is as follows:

[0116]

[0117] As shown in Figure 10 (b), wherein:

[0118] Distance_2: the Euclidean distance between the center point of the embryo predicted frame RG and the center point of the embryo real frame RR;

[0119] Distance_C: the diagonal distance of C (the diagonal distance between the upper left corner and the lower right corner of the frame RY);

[0120] C: the minimum circumscribed matrix of the embryo predicted frame RG and the embryo real frame RR, as shown in the frame RY in the figure;

[0121] v: (wherein w is the width, h is the height, gt represents the embryo real frame RR, and p represents the embryo predicted frame RG) aspect ratio influence factor

[0122] IOU refers to the intersection over union of the rectangular frame output by the embryo detection model and the rectangular frame of the original picture label, as shown in Figure 10 (b). First, the area of the intersection of the two rectangular frames is calculated:

[0123] S1=(xp2-xl1)*(yp2-yl1)

[0124] The area of the intersection is calculated:

[0125] S2=(xp2-xp1)*(yp2-yp1)+(yl2-yl1)*(xl2-xl1)-S1

[0126] Therefore, the calculation of IOU is:

[0127] IOU=S1 / S2

[0128] CIOU adds the predicted rectangular frame overlap area, center point distance and aspect ratio to the calculation on the basis of IOU.

[0129] The confidence degree represents the confidence degree of the predicted frame, and the value range is 0-1. The greater the value, the more likely the embryo exists in the rectangular frame. The confidence loss obj_loss calculates the confidence degree of the network.

[0130] The classification probability loss represents the category of the embryo. The classification probability loss cls_loss calculates whether the embryo category output by the detection layer is correct or not.

[0131] The embryo classification probability loss cls_loss and the confidence loss obj_loss adopt the BCEWithLogitsLoss loss function, and the formula is as follows:

[0132]

[0133] In the calculation of the embryo classification probability loss cls_loss, yn represents the category of the embryo in the label, and xn represents the predicted embryo category value output by the embryo detection model. In the calculation of the confidence loss obj_loss, yn represents the CIOU of the predicted frame output by the embryo detection model and the original picture label target frame. The CIOU is used as the confidence label of the predicted frame, and xn represents the random gradient value obtained at the moment t when the embryo detection model outputs the predicted confidence value t.

[0134] The total loss Loss of the embryo is the weighted sum of the above three losses. In the present application, the embryo confidence loss obtains the maximum weight, and the weights of the embryo rectangular frame loss and the embryo classification loss are second, so a=0.4, b=0.3, and c=0.3:

[0135] Loss=a*obj_loss+b*loss_box+c*clc_loss

[0136]

Parameter adjustment module

[0137] The parameter adjustment module 4 calculates the gradient of at least one parameter in the initial embryo target detection model M1 through back propagation, and optimizes the parameter through the Adam optimization algorithm.

[0138] Specifically, the Adam optimization function dynamically adjusts the learning rate of each parameter in the embryo training process by using the first moment estimate and the second moment estimate of the gradient. Compared with other optimization functions, the main advantage of Adam is that after the embryo is corrected by bias, the learning rate of each iteration has a certain range, so that the embryo model parameters are more stable. The Adam optimization algorithm adopted in the present application is updated as follows:

[0139] t←t+1

[0140] Calculate the gradient:

[0141]

[0142] Update the biased first moment estimate:

[0143] m t ← β1 · m t-1 + (1 - β1) g t

[0144] Update the biased second moment estimate:

[0145]

[0146] Calculate the bias-corrected first moment estimate:

[0147]

[0148] Calculate the bias-corrected first moment estimate:

[0149]

[0150] Update the parameters:

[0151]

[0152] where the exponential moving average of the gradient m t is calculated according to formula (1), m0 is initialized to 0. Refer to the Momentum algorithm, integrate the momentum of the gradient at previous time steps. The β1 coefficient is the exponential decay rate, which controls the weight distribution (momentum and current gradient), usually takes a value close to 1, the default is 0.9. gt is the random gradient value obtained at time t.

[0153] m t ← β1 · m t-1 + (1 - β1) g t … (1)

[0154] The exponential moving average of the gradient square v t is calculated according to formula (2), v0 is initialized to 0. The coefficient is the exponential decay rate, which controls the influence of the previous gradient square. Refer to the RMSProp algorithm, weight the mean of the gradient square, the default is 0.999.

[0155]

[0156] Since m0 is initialized to 0, it will cause m t biased to 0, especially in the initial stage of embryo model training. Therefore, refer to formula (3) m t correct the bias, reduce the influence of bias on the initial stage of embryo training.

[0157]

[0158] Similar to m0, because v0 is initialized to 0, the embryo training initial stage v t Bias 0, correct it according to formula (4)

[0159]

[0160] The parameters can be updated by formula (5), the initial learning rate alpha multiplied by the ratio of the average gradient and the square root of the gradient variance. Where the default learning rate alpha = 0.001; Set = 10 -8 , avoid the divisor becomes 0.

[0161]

[0162] It can be seen that the step length of the updated embryo model training can be adaptively adjusted from the two aspects of the average gradient and the square of the gradient, instead of being directly determined by the current gradient.

[0163] Adam has the following advantages relative to other optimization functions:

[0164] ① Automatically initialize the learning rate during embryo training;

[0165] ② Automatically adjust the learning rate during embryo training;

[0166] ③ Suitable for large-scale data and parameter scenarios of embryo;

[0167] ④ Suitable for detecting various unstable objective functions combined scenarios of embryo;

[0168] Although the present application has been described with reference to the current specific embodiments, those skilled in the art should recognize that the above embodiments are only used to illustrate the present application, and various equivalent changes or replacements can be made without departing from the spirit of the present application, therefore, any changes or modifications of the above embodiments within the scope of the spirit of the present application will fall within the scope of the claims of the present application.

Claims

1. A training platform of an embryo development detection device, comprising an embryo image enhancement module (1), an embryo image training set construction module (2), a model total loss judgment module (3), and a parameter adjustment module (4), wherein: the embryo image enhancement module (1) receives at least two image groups (Ga, Gb) of an embryo, the two image groups (Ga, Gb) each include a plurality of embryo images obtained by time-lapse photography on the embryo, and the two image groups (Ga, Gb) correspond to two different embryo development stages respectively; and the embryo image enhancement module (1) image fuses the plurality of embryo images of the two image groups (Ga, Gb) by weighted addition, and takes the image fusion result as an input image of the embryo image training set construction module (2); the embryo image training set construction module (2) labels the input image according to a selected embryo feature, to construct an embryo feature training set corresponding to the embryo feature; after the embryo feature training set is input into an embryo target detection initial model (M0), the model total loss judgment module (3) calculates a total loss of the embryo target detection initial model (M0) according to a loss function; and the parameter adjustment module (4) modifies at least one parameter in the embryo target detection initial model (M0) according to the total loss, to generate an embryo target detection model (M1) corresponding to the embryo feature.

2. The training platform according to claim 1, wherein: the model total loss judgment module (3) calculates the total loss by using a BCEWithLogitsLoss loss function and a CIOU loss function.

3. The training platform according to claim 1, wherein: the parameter adjustment module (4) calculates a gradient of at least one parameter in the embryo target detection initial model (M1) by back propagation, and optimizes the parameter by using an Adam optimization algorithm.

4. The training platform according to claim 1, wherein: the embryo target detection initial model (M0) is a path aggregation network, which includes a slice layer, a first down-sampling layer, a second down-sampling layer, a third down-sampling layer, a spatial pyramid layer, a first up-sampling layer, a second up-sampling layer, a first output layer, a second output layer, a third output layer, and a detection layer connected in sequence; wherein: the slice layer receives the embryo feature training set; the first up-sampling layer includes a first residual convolution module (RES1) and a first up-sampling module (LOS1) connected in sequence, and connects an output image of the first up-sampling module (LOS1) with an output of the third down-sampling layer, to serve as an input image of the second up-sampling layer; the second up-sampling layer includes a second residual convolution module (RES2) and a second up-sampling module (LOS2) connected in sequence, and takes an output image of the second up-sampling module (LOS2) as an input image of the first output layer. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ The first output layer comprises a first output module (LO1), and the output image of the second up-sampling module (LOS2) is connected with the output of the second down-sampling module (LUS2) to serve as the input image of the first output module (LO1); The second output layer comprises a second output module (LO2), and the output image of the first output module (LO1) is connected with the output of the second residual convolution module (RES2) after a convolution operation (conv) to serve as the input image of the second output module (LO2); The third output layer comprises a third output module (LO3), and the output image of the second output module (LO2) is connected with the output of the first residual convolution module (RES1) after a convolution operation (conv) to serve as the input image of the third output module (LO3); The detection layer comprises a detection module (D), and the output images of the first output module (LO1), the second output module (LO2) and the third output module (LO3) are respectively taken as the input images of the detection module (D).

5. An embryo development detection device, which receives an embryo image acquired by time-lapse photography on an embryo, and outputs target category information, a center point, width and height coordinates and a confidence of the embryo image by using a pre-trained embryo target detection model; The embryo target detection model comprises a slice layer, a first down-sampling layer, a second down-sampling layer, a third down-sampling layer, a spatial pyramid layer, a first up-sampling layer, a second up-sampling layer, a first output layer, a second output layer, a third output layer and a detection layer connected in sequence; wherein: The slice layer receives the embryo feature training set; and The first up-sampling layer comprises a first residual convolution module (RES1) and a first up-sampling module (LOS1) connected in sequence, and the output image of the first up-sampling module (LOS1) is connected with the output of the third down-sampling layer to serve as the input image of the second up-sampling layer; The second up-sampling layer comprises a second residual convolution module (RES2) and a second up-sampling module (LOS2) connected in sequence, and the output image of the second up-sampling module (LOS2) serves as the input image of the first output layer; The first output layer comprises a first output module (LO1), and the output image of the second up-sampling module (LOS2) is connected with the output of the second down-sampling module (LUS2) to serve as the input image of the first output module (LO1); The second output layer comprises a second output module (LO2), and the output image of the first output module (LO1) is connected with the output of the second residual convolution module (RES2) after a convolution operation (conv) to serve as the input image of the second output module (LO2); The third output layer comprises a third output module (LO3), and the output image of the second output module (LO2) is connected with the output of the first residual convolution module (RES1) after a convolution operation (conv) to serve as an input image of the third output module (LO3); The detection layer comprises a detection module (D), and the output image of the first output module (LO1), the output image of the second output module (LO2) and the output image of the third output module (LO3) are taken as input images of the detection module (D) respectively.

Images