Autofocusing methods, electronic devices, and storage media based on embryo image sequences

Abstract

The present invention provides an automatic focusing method, electronic device and storage medium based on embryo image sequence, wherein several groups of embryo image sequences are input into an automatic focusing network for training; the automatic focusing network includes a Transformer module, a channel attention mechanism module and a feature fusion module; the Transformer module captures the global information of the embryo image sequence through a self-attention mechanism and outputs a feature vector; the channel attention mechanism module is used to adjust the weight of each embryo image in the embryo image sequence and output a feature vector; the feature fusion module is used to fuse the outputs of the Transformer module and the channel attention mechanism module; the embryo image sequence to be focused is input into the trained automatic focusing network to obtain the clearest embryo picture in the embryo image sequence. Combined with time series analysis, the morphological changes during embryonic development are tracked to provide more comprehensive data support for embryo selection.

Description

Technical Field

[0001] The present invention relates to the technical field of embryo image processing, and in particular to an automatic focusing method based on embryo image sequence, electronic equipment and storage medium. Background Art

[0002] With the continuous advancement of assisted reproductive technology, especially in the field of in vitro fertilization and embryo culture, embryo quality assessment has become a key factor in improving pregnancy success rates. During the embryo evaluation process, clear, high-contrast embryo images are essential for accurately judging the embryo development stage and selecting the best embryos. However, due to differences in operator skills, equipment limitations, or the characteristics of the sample itself, obtaining high-quality embryo images is often challenging. Therefore, the development and application of embryo image autofocus technology has become an important research direction to improve the accuracy and efficiency of embryo evaluation.

[0003] Traditionally, embryologists need to rely on their naked eyes to screen out the appropriately focused embryo images from a large number of embryo images taken with a microscope. This process is not only time-consuming and labor-intensive, but also easily affected by subjective factors, resulting in inconsistency and poor repeatability of the evaluation results.

[0004] In summary, the research on automatic focusing evaluation technology based on embryo image sequences has practical significance for improving the success rate of assisted reproductive technology and reducing labor intensity. Summary of the invention

[0005] The present invention proposes an automatic focusing method based on embryo image sequence to solve the technical problems that the traditional method of screening suitable embryo images is time-consuming, labor-intensive and inaccurate.

[0006] To solve the above technical problems, the present invention provides an automatic focusing method based on an embryo image sequence, comprising the following steps:

[0007] Step S1: inputting a plurality of sets of embryo image sequences into the autofocus network for training; wherein the embryo image sequences are a plurality of embryo images taken by a microscope in a time sequence within a zoom cycle;

[0008] The automatic focusing network includes a Transformer module, a channel attention mechanism module, and a feature fusion module;

[0009] The Transformer module extracts the temporal features of the embryo image sequence and the morphological features of a single embryo image in the embryo image sequence through a CNN encoder, inputs a self-attention mechanism to capture the global information of the embryo image sequence, and outputs a feature vector;

[0010] The channel attention mechanism module is used to adjust the weight of each embryo image in the embryo image sequence and output a feature vector;

[0011] The feature fusion module is used to fuse the outputs of the Transformer module and the channel attention mechanism module;

[0012] Step S2: Inputting the embryo image sequence to be focused into the trained automatic focusing network to obtain the clearest embryo image in the embryo image sequence.

[0013] Preferably, the CNN encoder consists of two convolutional layers.

[0014] Preferably, the channel attention mechanism module adopts the following channel weight loss when adjusting the weight of each embryo image in the embryo image sequence: To train:

[0015] ;

[0016] In the formula, n represents the dimension of the feature vector, is the one-hot encoding of the true label of the channel weight The jth element in is the weight of the j-th channel generated by the channel attention mechanism branch with loss function.

[0017] Preferably, when the automatic focusing network is trained, a weighted loss function based on cross entropy is added to the feature vectors output by the Transformer module and the channel attention mechanism module and the fused output, and the weighted loss function is added to the channel weight loss function. Add them together to form the final loss function.

[0018] Preferably, the expression of the cross entropy-based weighted loss function is:

[0019] ;

[0020] ;

[0021] In the formula, represents the weight of the jth category in the embryo image sequence; m represents the number of categories; or Indicates category or Category The number of samples; represents the unique hot encoding of the true label; when i=1, represents the final output of the autofocus network; when i=3, Represents the output of the Transformer module; when i=4, Represents the output of the channel attention mechanism module.

[0022] Preferably, the feature fusion module uses a maximum fusion method to perform feature fusion.

[0023] The present invention also provides an electronic device, comprising: a memory, a processor and a computer program, wherein the computer program is stored in the memory and is configured to be executed by the processor to implement the above method.

[0024] The present invention also provides a computer-readable storage medium, in which a computer program is stored. The computer program is executed by a processor to implement the above method.

[0025] The beneficial effects of the present invention include at least:

[0026] 1) Improve the speed and consistency of image acquisition and reduce errors caused by manual focusing.

[0027] 2) Reduce the reliance on operator skills and make embryo assessment more objective and standardized.

[0028] 3) Combined with time series analysis, morphological changes during embryonic development can be tracked to provide more comprehensive data support for embryo selection. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a schematic diagram of the structure of an automatic focusing network model according to Embodiment 1 of the present invention;

[0030] Figure 2 A schematic diagram of a 5-fold cross method according to an embodiment of the present invention;

[0031] Figure 3 Schematic diagram of the results of the ablation experiment;

[0032] Figure 4 This is a schematic diagram of the results predicted by EISAFNet;

[0033] Figure 5 A schematic diagram reflecting the internal temporal relationship of an embryo image sequence according to an embodiment of the present invention;

[0034] Figure 6 This is a schematic diagram of processing data of a Transformer with a CNN encoder in Example 2 of the present invention;

[0035] Figure 7 This is a schematic diagram of the effect of the channel attention mechanism with loss function in Example 3 of the present invention;

[0036] Figure 8This is a schematic diagram of the distribution of the number of categories in the data set used by the present invention to train EISAFNet;

[0037] Fig. 9 Schematic diagram of the results for verifying the weighted loss function;

[0038] Fig.10 This is a schematic diagram of the network structure after integrating Embodiment 1 to Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0039] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention.

[0040] Example 1

[0041] The embodiment of the present invention provides an automatic focusing method based on an embryo image sequence, comprising the following steps:

[0042] Step S1: inputting a plurality of sets of embryo image sequences into the autofocus network for training; wherein the embryo image sequences are a plurality of embryo images taken by a microscope in a time sequence within a zoom cycle;

[0043] The automatic focusing network includes a Transformer module, a channel attention mechanism module, and a feature fusion module;

[0044] The Transformer module captures the global information of the embryo image sequence through a self-attention mechanism and outputs a feature vector;

[0045] The channel attention mechanism module is used to adjust the weight of each embryo image in the embryo image sequence and output a feature vector;

[0046] The feature fusion module is used to fuse the outputs of the Transformer module and the channel attention mechanism module;

[0047] Step S2: Input the embryo image sequence to be focused into the trained autofocus network to obtain the clearest embryo image in the embryo image sequence.

[0048] The embryo image sequence adopted in the present embodiment is collected by the time difference incubator (TLS301) developed by Huchuang United Technology Co., Ltd. The 6,080 groups of embryo image sequences used in the experiment are from this equipment, and each group comprises 7 embryo images. In these embryo image sequences, the best focus position of the microscope mainly concentrates on the 4th, 5th, 6th and 7th pictures of the embryo image sequence taken by each group. Therefore, the present invention converts the auto-focus task of the embryo image sequence into a 4 classification task, and each focus position corresponds to a category.

[0049] In the research of automated embryo image focusing technology, color information is often not a decisive factor, and grayscale images are sufficient to provide the main visual clues required for automatic focusing. In order to reduce the complexity of processing and give play to the characteristics of grayscale images, the present invention adopts the cv2.cvtColor() function in the OpenCV library to achieve the conversion of colored embryo images to grayscale images. This step not only effectively eliminates the unnecessary interference caused by color, but also can greatly improve the accuracy and efficiency of the network model to the best focus picture judgment, providing a reliable basis for subsequent analysis and research.

[0050] The network structure in this embodiment is as follows Figure 1 As shown, in order to achieve automatic focusing on an embryo image sequence and thus locate the best embryo image in a set of embryo image sequences, the present invention proposes an automatic focusing network named EISAFNet based on deep learning.

[0051] The autofocus network mainly consists of two branches. Branch 1 is the combination of the Transformer module, and branch 2 is the channel attention mechanism module. Through the combination of the two branches, EISAFNet can better focus on the best focused embryo image while extracting the morphological and temporal features of the image in the embryo image sequence.

[0052] For branch 1, Transformer mainly achieves efficient modeling and processing of sequence data through the self-attention mechanism, thereby effectively capturing global information. The self-attention mechanism is the core component of Transformer, which allows the model to interact between each position in the input sequence and calculate the attention weight for each position relative to all other positions. The following is the mathematical expression of the self-attention mechanism:

[0053] (1)

[0054] (2)

[0055] (3)

[0056] (4)

[0057] in, is the input sequence, , , Yes Query ,key ,value The weight matrix of . Parameters yes The dimension size of . Attention is the required attention, and the softmax function is an activation function that converts the input vector into a probability distribution.

[0058] For branch 2, the channel attention mechanism is a method of focusing on the importance of different channels of the input data. The main idea is to automatically adjust the importance of each channel by setting the weight of the corresponding channel, so that the model can pay more attention to channels that have an important impact on the task, while suppressing unimportant channels. In our experimental samples, a set of embryo image sequences consists of seven microscope embryo images taken in chronological order, forming a sequence. The sequence shows the developmental changes of the embryo in different focal planes. Considering the completeness of a set of embryo image sequences, the input of our model is not a single image. Instead, such as Figure 1 As shown, the image sequence in this embodiment is composed of 7 stacked images, which can be regarded as data with 7 channels, each channel corresponding to an embryo image in the sequence. Therefore, this embodiment selects the best focused embryo image in a set of embryo image sequences, and adjusts the weights of the 7 channels through the channel attention mechanism module, so that the grid pays more attention to the best focused image.

[0059] For the feature fusion module, this embodiment fuses the two 4-dimensional feature vectors from branch 1 and branch 2 using the maximum fusion method. Maximum fusion is a simple and effective feature fusion technique that generates a new feature vector by selecting the maximum value in each dimension, which can retain the most important features and enhance the performance of the proposed model. Assume that the feature vectors output by the two branches are and Then the fused vector The calculation formula is:

[0060] (5)

[0061] in, is the eigenvector and In Dimension Maximum value on.

[0062] After the network training is completed, the input embryo image sequence can be classified to obtain the final focused image. The 4 classification tasks in this embodiment are only for the situation that the best focus position of the data set of this embodiment is mainly concentrated on the 4th, 5th, 6th and 7th pictures, and are not used as a limitation of the present invention.

[0063] In the following embodiment, during the model training process, the epoch is set to 60, the batch size is set to 20, and the learning rate is set to 0.0001 for training.

[0064] Then, the weighted average accuracy, weighted average precision, weighted average recall and weighted average F1 score are used as evaluation indicators of this embodiment. The formulas of these evaluation indicators are as follows:

[0065] (6)

[0066] (7)

[0067] in, It is The number of true positive examples of a class, It is The number of misclassified samples of the class, It is The number of instances in the class that are actually positive but are incorrectly predicted to be negative. It is The accuracy of the class, Represents the weighted average accuracy.

[0068] (8)

[0069] (9)

[0070] in, It is The precision of the class, Stands for weighted average precision. (10)

[0071] (11)

[0072] in, It is The recall rate of the class, Represents the weighted average recall.

[0073] (12)

[0074] (13)

[0075] in, It is Class , Represents weighted average . Weight It is obtained by calculating the proportion of samples in each category to the total samples. The weight ratio of each category is category 1: category 2: category 3: category 4 = 0.36: 0.40: 0.18: 0.06.

[0076] Then the 5-fold cross validation method is used. Figure 2 As shown in Figure 1, it shows the basic idea of ​​5-fold cross validation. The dataset is divided into five non-overlapping subsets. Each subset is used as a test set in turn, while the remaining four subsets are used for training. This approach allows us to conduct five independent experiments to fully evaluate the performance of the network. Table 1 shows the accuracy obtained using 5-fold cross validation.

[0077] Table 1

[0078]

[0079] Then, 4 sets of ablation experiments were conducted using 6,080 sets of samples, and each set was validated using a 5-fold cross-validation method. The results of the ablation experiments are shown in Figure 2. Figure 3 As shown in FIG. 1 , the weighted average accuracy, precision, recall rate and F1 score are compared. In addition, four groups of effect diagrams of the model prediction results of the embodiments of the present invention are selected, as shown in FIG. Figure 4 As shown, the embryo image with a border is the image with the best focus.

[0080] Experiment 2 and Experiment 3 were conducted using the separate Transformer module and channel attention mechanism module of this embodiment, respectively. Obviously, the performance of any single branch is not as good as EISAFNet.

[0081] The reasons can be roughly summarized as follows. On the one hand, branch 1 uses Transformer to extract temporal features, but this is an unfocused operation that prevents Transformer from better focusing on the best focused embryo image. On the other hand, the single branch 2 emphasizes the best focused image in the embryo image sequence, ignoring the temporal correlation within the entire sequence. However, EISAFNet uses a combination of two branches, which extracts the morphological features of each image and the temporal features of the embryo image sequence while directing attention to the best focused embryo image, resulting in better classification results than the two separate branches.

[0082] Example 2

[0083] In the data of this embodiment, the images in each set of embryo image sequences are taken by the microscope at each focus adjustment. During the imaging process, the focus is adjusted continuously over time. Therefore, there is a time correlation between the embryo images in the sequence. These embryo images show the changes and development of the embryo at different time points. Figure 5 The figure shows a possible scenario in which the distance between the microscope lens and the embryo changes over time during the focusing and imaging process, indicating the temporal correlation that exists in the embryo image sequence.

[0084] Traditional 2DCNN performs convolution operations on a two-dimensional plane, which is very effective for processing single-frame images or two-dimensional data. However, when processing time series data or three-dimensional spatial data, 2DCNN cannot capture information in the third dimension (time or depth), which may lead to information loss. Inspired by the Video Transformer Network (VTN), this embodiment introduces a Transformer with a CNN encoder in branch 1 of the network to process embryonic image sequences. Specifically, Figure 6 As shown, 7 images in a set of embryo image sequences are regarded as 7 frames of a short video. Therefore, the Transformer with a CNN encoder can extract the morphological features of a single embryo image and the temporal features from the embryo image sequence, thereby improving the overall classification performance. It should be noted that the CNN in this embodiment consists of two ordinary convolutional layers.

[0085] Example 3

[0086] Since the present invention is to select the best focused embryo image in a set of embryo image sequences, this indicates that more attention needs to be paid to the best focused image. Therefore, this embodiment adopts a channel attention mechanism with a loss function.

[0087] Specifically, the channel attention mechanism can adjust the weight of each embryo image in the sequence, and the introduction of the loss function can maximize the weight of the best focused embryo image, forcing the network to increase its attention to the best focused embryo image. In this way, the network's attention can be directed to the best focused embryo image. The expression of the channel weight loss used in this part is as follows:

[0088] (14)

[0089] In the formula, 7 represents the dimension of the feature vector. Since the embodiment of the present invention has 7 pictures, 7 is selected here. is the one-hot encoding of the true label of the channel weight The jth element in is the weight of the j-th channel generated by the channel attention mechanism branch with loss function.

[0090] Figure 3 Experiment 4 in is used to verify the effectiveness of the channel weight loss. Experiment 4 is training without using the channel weight loss of this embodiment. As can be seen from the figure, the performance of Experiment 4 is not as good as the experiment conducted by EISAFNet. The performance of Experiment 4 is not as good as the experiment conducted by EISAFNet. A simple channel attention mechanism usually adaptively adjusts the channel weights without using label information for supervised optimization. However, the channel attention mechanism combined with the loss function can optimize towards the global optimal direction during the training process. Specifically, the channel attention mechanism with the loss function can combine label information to make the adjustment of channel weights more targeted, while maximizing the weight of the best focused embryo image in the embryo sequence, which makes the network pay more attention to the best focused embryo image, thereby further improving the performance of the entire network. Figure 7 As shown in Figure 1, we selected a set of embryo image sequences and plotted heat maps of each image with and without channel weight loss. Obviously, the channel attention mechanism with loss function does enhance the network's focus on the best focused embryo image, which can be illustrated by the weight of each embryo image. In contrast, without using channel weight loss, the model assigns relatively poor weights to each image and also produces incorrect prediction results.

[0091] Example 4

[0092] The quantity distribution of each category in the data set of the present invention is as follows Figure 8 As shown. Obviously, there is an imbalance problem in the number of various types of data in the data set. Specifically, the microscope takes embryo images 7 times in a row within a specific time, and obtains 7 embryo images with different degrees of focus. The goal of the present invention is to find the best focused embryo image in these images. In the present invention, a classification strategy is adopted to solve this problem, and each focus position is regarded as a separate data category. In theory, this strategy will divide the original embryo image data set into 7 categories, corresponding to 7 focus positions. However, when checking 6080 groups of embryo image sequences, it is found that the optimal focus position is mainly distributed in positions 4, 5, 6 and 7, which means that in all embryo image sequences, there are only the 4th, 5th, 6th and 7th images that are the best focus images, and the 1st, 2nd and 3rd embryo images are 0 as the best focus images. Therefore, in fact, the number of categories is set to 4, corresponding to focus positions 4, 5, 6 and 7.

[0093] 6080 sets of high-quality embryo image sequences constitute the training set and test set. It is worth noting that the categories of embryo image sequences in the dataset are mainly concentrated in the case of the 4th and 5th embryo images, with 2161 and 2410 sets respectively. In contrast, the number of categories where the best focused embryo images are the 6th and 7th images is relatively small, with 1107 and 402 sets respectively. Obviously, the number distribution of each category in the dataset is unbalanced.

[0094] In order to solve this problem, this embodiment uses a weighted loss function based on cross entropy in the network training process on the basis of Embodiment 3, and its expression is:

[0095] (15)

[0096] Among them is category The weight of is the one-hot encoding of the true label, is the final output of EISAFNet or the output of branch 1 or branch 2. It indicates the final output when When , it indicates the output of branch 1, when , it indicates the output of branch 2. They are the losses of the final output of the network and the two feature vectors output by branch 1 and branch 2 respectively.

[0097] The weight of each category Calculated according to the following formula:

[0098] (16)

[0099] In the formula, 4 represents the number of categories, or Yes Category or Category According to the above weight formula, the weight ratio of each category is category 1: category 2: category 3: category 4 = 0.70: 0.63: 1.37: 3.78.

[0100] In this embodiment, in the automatic focusing network EISAFNet, there are four parts that use loss functions. The first part is the loss of the final output of the network ( ), and the second part is the loss of channel weights ( ), and the third part is the loss of branch 1 ( ), the fourth part is the loss of branch 2 ( ). The total loss function can be expressed as follows:

[0101] (17)

[0102] in , , and is the regularization coefficient. According to experience, this embodiment sets It should be noted that not all of these four loss functions use a weighted loss function based on cross entropy. , Since the dimension of the feature vector involved in the loss calculation is 7, the channel weight loss in Example 3 is used.

[0103] Figure 3 Experiment 1 in is used to verify the role of the cross-entropy based weighted loss function. First, in Experiment 1, all cross-entropy based weighted losses (L1, L3, and L4) used in EISAFNet are replaced with ordinary cross-entropy losses. Fig. 9 As shown, taking accuracy as an example, although overall, the accuracy of using weighted loss is only slightly increased (+0.0005) compared to the result of using ordinary cross entropy. However, the purpose of using weighted loss is to emphasize the relatively small number of categories 3 and 4. The accuracy of the experiment conducted by EISAFNet on categories 3 and 4 is higher than that of experiment 1. In particular, on category 4, the accuracy of the experiment conducted by EISAFNet is 0.0173 higher than that of experiment 1. Secondly, in category 2, although the results of the experiment conducted by EISAFNet are not as good as those of experiment 2 in terms of accuracy. However, this just shows the rationality and correctness of using weighted loss from the side. Since the samples of category 2 account for the largest proportion of the total samples, the ordinary cross entropy loss cannot evenly calculate the loss of each category, causing the network to pay more attention to category 2, thereby improving its accuracy. However, for categories 3 and 4 with fewer samples, their accuracy is reduced compared to the experiment of EISAFNet. Therefore, in summary, Experiment 1 fully verifies the role of the weighted loss function based on cross entropy.

[0104] In order to further integrate the embodiments 1 to 4, the structure of the automatic focusing network after integrating the embodiments 1 to 4 is as follows: Fig.10 shown.

[0105] The present invention also provides an electronic device, comprising: a memory, a processor and a computer program, wherein the computer program is stored in the memory and is configured to be executed by the processor to implement the above method.

[0106] The present invention also provides a computer-readable storage medium, in which a computer program is stored. The computer program is executed by a processor to implement the above method.

[0107] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. Only the preferred embodiments of the present invention are expressed. The description is more specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. As long as there is no contradiction in the combination of these technical features, they should be considered as the scope recorded in this specification.

[0108] It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these modifications and improvements all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. An automatic focusing method based on embryo image sequence, characterized in that: The following steps are involved: Step S1: inputting a plurality of sets of embryo image sequences into the autofocus network for training; wherein the embryo image sequences are a plurality of embryo images taken by a microscope in a time sequence within a zoom cycle; The automatic focusing network includes a Transformer module, a channel attention mechanism module, and a feature fusion module; The Transformer module extracts the temporal features of the embryo image sequence and the morphological features of a single embryo image in the embryo image sequence through a CNN encoder, inputs a self-attention mechanism to capture the global information of the embryo image sequence, and outputs a feature vector; The channel attention mechanism module is used to adjust the weight of each embryo image in the embryo image sequence and output a feature vector; The feature fusion module is used to fuse the outputs of the Transformer module and the channel attention mechanism module; Step S2: inputting the embryo image sequence to be focused into the trained autofocus network to obtain the clearest embryo image in the embryo image sequence; The channel attention mechanism module uses the following channel weight loss when adjusting the weight of each embryo image in the embryo image sequence: To train: ； In the formula, n represents the dimension of the feature vector, is the one-hot encoding of the true label of the channel weight The jth element in is the weight of the jth channel generated by the channel attention mechanism branch with loss function; When the automatic focusing network is trained, a weighted loss function based on cross entropy is added to the feature vectors output by the Transformer module and the channel attention mechanism module and the fused output, and the weighted loss function is added to the channel weight loss function. Add them together to form the final loss function; The expression of the weighted loss function based on cross entropy is: ； ； In the formula, represents the weight of the jth category in the embryo image sequence; m represents the number of categories; or Indicates category or Category The number of samples; represents the unique hot encoding of the true label; when i=1, represents the final output of the autofocus network; when i=3, Represents the output of the Transformer module; when i=4, Represents the output of the channel attention mechanism module.

2. The automatic focusing method based on embryo image sequence according to claim 1, characterized in that: The CNN encoder consists of two convolutional layers.

3. The automatic focusing method based on embryo image sequence according to claim 1, characterized in that: The feature fusion module uses a maximum fusion method to perform feature fusion.

4. An electronic device, comprising: A memory, a processor and a computer program, characterized in that: the computer program is stored in the memory and is configured to be executed by the processor to implement the method described in any one of claims 1 to 3.

5. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 3.

Images