Embryo video cystogenesis prediction device based on adaptive key frame selection

Abstract

The present invention discloses an embryo cyst formation prediction device based on adaptive key frame selection. The device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. An embryo cyst formation prediction model is stored in the memory. When the processor executes the computer program, the following steps are implemented: obtaining a cleavage-stage video obtained by monitoring using a time-delay monitoring technology, and performing video frame sampling on the cleavage-stage video to obtain a video frame sequence and sampling position information of the video frames; inputting the video frame sequence and the sampling position information into the embryo cyst formation prediction model, and obtaining a prediction result of the embryo cyst formation through calculation; wherein the embryo cyst formation prediction model comprises a feature extraction unit, a position information conversion unit, an LSTM unit, a key frame decision unit and a cyst formation prediction unit. The device can realize efficient and accurate prediction of embryo cyst formation.

Description

Technical Field

[0001] The present invention belongs to the field of medical image processing, and in particular relates to an embryo video cystogenesis prediction device based on adaptive key frame selection. Background Art

[0002] In recent years, with the advancement of computing power, the application of deep learning technology in engineering has rapidly developed. Medical artificial intelligence, as a subfield of deep learning, has leveraged deep learning to help experts avoid redundant and complex tasks. Furthermore, deep learning, through the depth of its networks, can uncover previously unnoticed features. Therefore, deep learning-based medical assistance has become a crucial area of ​​computer science. In vitro fertilization (IVF) is the most commonly used medical treatment for infertility. To improve the success rate of embryo transfer, selecting well-developed embryos is a key step in embryo transfer (ET). However, long-term in vitro culture can reduce embryonic developmental potential, leading to the risk of embryo failure at the blastocyst stage. While multiple embryo transfer can ensure successful pregnancy, it can also increase the burden on pregnant women. Therefore, early embryo assessment, selecting embryos with good developmental potential, and appropriately extending the culture period are crucial steps in IVF.

[0003] Embryologists incubate embryos in an incubator equipped with a built-in microscope and use TLM to regularly record videos of the embryo's development. This provides a stable recording environment and eliminates the need to remove the embryo from the incubator, ensuring the quality of embryonic development. Experts score the embryos based on the TLM videos of their development. Embryos rated as high-quality have a greater likelihood of cyst formation, while embryos rated as poor-quality have a lower likelihood of cyst formation.

[0004] The evaluation indicators during the embryonic cleavage stage development mainly include stage, symmetry, and fragmentation. The embryo during the cleavage stage will go through the pronuclei appearance (PNA), pronuclei fading (PNF), 2-5 cell and 8-cell and above stages. During the cleavage stage, the embryo will show a variety of abnormal division phenomena, including asymmetry: the size of the cells after cleavage is quite different; fragmentation: a large number of cell fragments appear after division instead of two complete cells, but less than 10% of the fragments can be absorbed during subsequent development; multiple nuclei: the presence of multiple nuclei in the blastomere; direct cleavage: skipping the 2-cell stage and directly dividing into 3 cells; developmental arrest: a cell stops dividing during the division. Embryologists usually judge whether the embryo can form a cyst based on whether there are any of the above abnormal developmental behaviors during embryonic development.

[0005] However, embryo scoring still relies on embryologists manually analyzing time-lapse monitoring (TLM) videos to determine embryo grades. Manually analyzing large numbers of embryos is tedious for cytologists, and manual scoring is subjective, leading to significant variability among embryologists. With the advancement of artificial intelligence (AI) and medical image analysis, numerous machine learning and deep learning methods have been applied to assist in the diagnosis of medical images, helping experts make more accurate diagnoses. Video image classification is a research area within the field of AI computer vision. Its task is to extend image classification, transforming the input from a single 2D image into a sequence of 2D images. The ResNet network, proposed by Kaiming He et al. in their 2015 CVPR paper "Deep Residual Learning for Image Recognition," utilizes a residual structure to improve network generalization, making it one of the most commonly used feature extraction networks. The LSTM network, proposed by Hochreiter et al. in their 1997 paper "Long Short-Term Memory," is a variant of the RNN designed to capture longer sequences of information. LSTM records both long-term and short-term information through the input gate, output gate, and forget gate. Therefore, LSTM has become one of the most common feature analysis networks in temporal networks.

[0006] During embryonic cyst classification, embryologists grade and count the cells in each frame to assess the overall quality of embryonic development and select the best-performing embryos for ET. However, due to the excessive number of redundant images during embryonic development, selection efficiency is low, reducing accuracy. Summary of the Invention

[0007] In view of the above, an object of the present invention is to provide an embryo video cyst prediction device based on adaptive key frame selection to achieve efficient and accurate prediction of embryo cyst formation.

[0008] To achieve the above-mentioned purpose of the invention, the embodiment provides an embryo video cyst prediction device based on adaptive key frame selection, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the memory stores an embryo cyst prediction model, and when the processor executes the computer program, the following steps are implemented:

[0009] Acquire a cleavage-stage video obtained by time-lapse monitoring technology, and perform video frame sampling on the cleavage-stage video to obtain a video frame sequence and sampling position information of the video frames;

[0010] The video frame sequence and sampling position information are input into the embryo cyst formation prediction model, and the embryo cyst formation prediction result is obtained through calculation;

[0011] The embryo cyst formation prediction model includes a feature extraction unit, a position information conversion unit, an LSTM unit, a key frame decision unit and a cyst formation prediction unit, wherein a video frame is input into the feature extraction unit to obtain a feature vector through feature extraction, sampling position information is input into the position information conversion unit to obtain a position vector through conversion, the feature vector and the position vector are spliced ​​and input into the LSTM unit for processing to output a hidden layer state, the hidden layer state is input into the key frame decision unit to judge whether it is a key frame, the hidden layer state judged as the key frame is input into the cyst formation prediction unit to calculate the embryo cyst formation prediction result; the hidden layer state judged as a non-key frame is discarded and does not participate in the cyst formation prediction, and the hidden layer state of the previous key frame is used to update the hidden layer state of the current video frame, the updated hidden layer state of the current video frame is input into the LSTM unit to participate in the hidden layer state calculation of the next video frame, and the embryo cyst formation prediction result of the last key frame is used as the final embryo cyst formation prediction result.

[0012] In one embodiment, before the embryo cyst formation prediction model is applied, model parameter optimization is required. The model parameter optimization process includes:

[0013] Construct a sample set, using the video frame sequence and the corresponding sampling position information sequence obtained by sampling a single cleavage-stage video as input data, and whether the embryo has formed a cyst as the true value label to form a single sample;

[0014] Construct the loss function, loss function L=L cls +βL rwd , where L clsis the classification loss constructed based on the predicted result of embryo cyst formation and the true value label, β represents the weight, with a value of 0-10, preferably 1, L rwd is the reward loss of the prediction result corresponding to the key frame, specifically:

[0015]

[0016] r t =max(0,d t -max(d t ′,t′∈(0~t-1)))

[0017]

[0018] Among them, t represents the current time, corresponding to the index of the input video frame at the current time, T is the total time, corresponding to the total number of sampled video frames of a single cleavage stage video, r t The prediction result reward corresponding to the input video frame at time t as the key frame, d t Represents the predicted boundary of the input video frame at time t, that is, the maximum value of the true label prediction value minus the prediction value of other categories, Indicates the predicted value of the true label, c indicates the predicted category, and gt indicates the true label (ground truth);

[0019] The samples are used as the input of the embryo cyst formation prediction model, and the loss function is used to perform supervised learning on the embryo cyst formation prediction model under the supervision of the true value label to optimize the model parameters.

[0020] In one embodiment, when constructing a sample set, invalid filtering is performed on the acquired cleavage stage videos, including: first filtering out empty culture dishes and overexposed video frames, then filtering out cleavage stage videos whose number of video frames does not meet a preset video frame threshold, and the remaining cleavage stage videos participate in video frame sampling to construct the sample.

[0021] In one embodiment, the feature extraction unit uses at least the first 5 layers of the ResNet50 network.

[0022] In one embodiment, the position information conversion unit performs vector conversion on the input position information in the following manner:

[0023] First, use the formula Calculate the first transformation vector PE (pos,2i) , where pos represents the input sampling position information, i represents the vector dimension, and d model is a preset constant, with a value of 720-900, preferably 850;

[0024] Then, using the formula Calculate the second transformation vector PE (pos,2i+1) ;

[0025] Finally, concatenate the first transformation vector PE (pos,2i) and the second transformation vector PE (pos,2i+1) To get the position vector.

[0026] In one embodiment, the key frame decision unit adopts a fully connected neural network.

[0027] In one embodiment, the key frame decision unit calculates the input hidden layer state to output a judgment result. When the judgment result is greater than or equal to 0.5, the video frame corresponding to the input hidden layer state is considered to be a key frame. When the judgment result is less than 0.5, the video frame corresponding to the input hidden layer state is considered to be a non-key frame.

[0028] In one embodiment, the result of the key frame decision unit determining that the frame is a key frame is recorded as (0, 1), and the result of the key frame decision unit determining that the frame is a non-key frame is recorded as (1, 0). Based on this, the hidden layer state of the current video frame that is not a key frame is updated according to the non-key frame judgment result (1, 0). The update formula is:

[0029] h t ′=h t″ ×1+h t ×0

[0030] Among them, h t Indicates the hidden state of the current video frame that is not a key frame, h t″ Indicates the hidden state of the previous key frame, h t ′ represents the updated hidden layer state of the current video frame, and 1 and 0 represent the judgment results of the current video frame being judged as a non-key frame.

[0031] In one embodiment, after video frame sampling is performed on the cleavage stage video, data enhancement is performed on the sampled video frame images, and the enhanced video frame images form a video frame image sequence, wherein the data enhancement includes scaling and cropping, random horizontal flipping, random vertical flipping, random brightness change, and random angle rotation.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] When constructing the sample, the ground truth label for embryonic development D3 only requires labeling whether the cleavage-stage video captured by the overall time-lapse monitoring technology is cyst-formed or not, without requiring embryonic stage labeling. This makes the labeling process simple and efficient.

[0034] Keyframes are adaptively selected based on the keyframe decision unit. The selected keyframes participate in the LSTM calculation of the LSTM unit and the capsule prediction of the capsule prediction unit, avoiding poor prediction results caused by redundancy of video frame images. At the same time, the memory length of the LSTM is extended through the keyframe adaptive selection operation.

[0035] The parameter-optimized embryo cyst prediction model can realize automatic prediction of embryo cyst formation, improve the accuracy and efficiency of embryo cyst formation prediction, and help embryologists select excellent embryos more accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0037] Figure 1 This is a flow chart of using the embryo cyst formation prediction model provided in the embodiment to predict embryo cyst formation;

[0038] Figure 2 Schematic diagram of the structure and calculation process of the embryo cyst formation prediction model provided in the embodiment. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0040] In order to solve the problem that the embryo cyst formation prediction is not accurate enough due to the large redundancy of the embryo cleavage stage video monitored by TLM in actual applications, an embodiment provides an embryo video cyst formation prediction device based on adaptive key frame selection, which includes a memory, a processor and a computer program stored in the memory and executable on the processor, wherein an embryo cyst formation prediction model is stored in the memory, and the embryo video cyst formation prediction step is implemented when the processor executes the computer program.

[0041] The embryo cyst formation prediction model stored in the memory has been pre-optimized for model parameters. The specific parameter optimization process includes:

[0042] Step 1: Construction of the sample set, i.e. recording the embryo development images and marking whether the embryo has formed a cyst.

[0043] In the embodiment, TLM technology is used to monitor and obtain cleavage stage video. Specifically, recording is started from in vitro fertilization, and images are taken at fixed time intervals (for example, 5 minutes) until the third day of embryonic development (abbreviated as D3) is stopped, forming a 10-frame-per-second cleavage stage video, where each frame image is a single-channel grayscale image of 224*224 size. The main body of the embryo is generally in the middle of the image, but the embryo will be offset in a small number of embryo images.

[0044] Since the video frames of empty culture dishes and overexposed video frames appear at the end of the cleavage-stage video, it is necessary to filter out these invalid video frames. Specifically, the video frames of empty culture dishes and overexposed video frames can be removed by an edge detection algorithm. Then, it is necessary to filter out invalid videos. Specifically, the cleavage-stage videos whose number of video frames does not meet the preset video frame threshold (e.g., 720 frames) are filtered out, and the remaining cleavage-stage videos participate in video frame sampling to construct the sample.

[0045] After invalid video frame images and invalid videos are filtered out, the remaining cleavage-stage videos are marked to determine whether the embryos have formed cysts. Embryologists use a two-category true value labeling of the embryos recorded in each cleavage-stage video according to whether they can develop into the blastocyst stage based on clinical results, where label 1 indicates that they cannot form cysts and 0 indicates that they can form cysts. During specific labeling, the embryos may have asymmetry, fragmentation, direct cleavage and other abnormal developmental conditions during development, and embryologists usually use this to determine whether embryonic cells can form cysts. It should be noted that the true value labeling is for the entire cleavage-stage video, and the video frame images obtained by sampling the entire cleavage-stage video have the same true value labels as the cleavage-stage video to which they belong.

[0046] After labeling, the cleavage-stage videos were divided into a validation set (10%), a test set (10%), and a training set (80%). The data was randomly selected to avoid similar data being in the same set.

[0047] In the embodiment, each cleavage stage video is also uniformly sampled. For example, 32 frames of video frame images are sampled. After sampling, the sampled video frame images are subjected to batch data enhancement within a certain range, including: scaling and cropping, random horizontal flipping, random vertical flipping, random brightness change, and random angle rotation. Because it is necessary to ensure that the data of the same sample is at the same angle, the above-mentioned data enhancement needs to be uniformly processed for all sampled video frame images of a single video, that is, the same operation needs to be performed on all sampled video frame images in a certain sample, thereby ensuring the uniformity of the video data and ensuring that the relative positions of the features in the image are fixed in the same sample. In the embodiment, only batch normalization and batch center cropping are performed on the validation set and the test set.

[0048] After sampling and data augmentation, the video frame sequence and the corresponding sampling position information sequence obtained from sampling a single cleavage-stage video are used as input data, and whether the embryo has formed a cyst is used as the ground truth label to form a single sample. The sampling position information refers to the temporal position information of the video frame image in the cleavage-stage video.

[0049] Step 2: Construct the network structure of the embryo cyst formation prediction model.

[0050] like Figure 2 As shown, the embryo cyst prediction model constructed in the embodiment includes a feature extraction unit, a position information conversion unit, an LSTM unit, a key frame decision unit and a cyst prediction unit. Among them, the feature extraction unit is used to extract the feature vector of the input video frame image. Specifically, at least the first 5 layers of the ResNet50 network can be used. The experimental example uses the first 5 layers, wherein the first 5 layers are 5 convolution modules, the first convolution module is a single-layer convolution layer (Convolutional Layer), and then 2-4 convolution modules are connected in series to form a residual module with an increasing number of channels by connecting multiple Bottleneck modules, and each convolution module is activated by a ReLU function.

[0051] The position information conversion unit is used to perform vector conversion on the input sampling position information to obtain a position vector. Specifically, the following method is used to convert the input position information into a vector:

[0052] First, use the formula Calculate the first transformation vector PE (pos,2i) , where pos represents the input sampling position information, i represents the vector dimension, in the experimental example, i is selected as 1024, d model It is a preset constant with a value of 850;

[0053] Then, using the formula Calculate the second transformation vector PE (pos,2i+1) ;

[0054] Finally, concatenate the first transformation vector PE (pos,2i) and the second transformation vector PE (pos,2i+1) To get the position vector.

[0055] Because embryonic division requires consideration not only of morphological parameters, i.e., abnormalities in division, but also of dynamic parameters, i.e., the rate of embryonic development, which reflects its vigor. Faster development indicates greater activity, to a certain extent. Therefore, the 2048-dimensional feature vector extracted by the first five convolutional modules of ResNet50 is horizontally concatenated with the 2048-dimensional position vector obtained by converting the sampling position information of the video frame image, resulting in a 4096-dimensional joint vector corresponding to each video frame image.

[0056] The LSTM unit is used to output the corresponding hidden state (hiddenstate, h t ) and cell state (cell state,c t ), where h t Contains the feature information of the current period, c t Integrates long-term feature information.

[0057] The key frame decision unit is used to input the hidden state h t A binary classification judgment is performed to determine whether the current video frame image is a key frame and a judgment result is output. This judgment result is used for the adaptive key frame selection result. When the judgment result is greater than or equal to 0.5, the video frame corresponding to the input hidden layer state is considered a key frame. When the judgment result is less than 0.5, the video frame corresponding to the input hidden layer state is considered a non-key frame. In this embodiment, the key frame decision unit can use a fully connected neural network to perform key frame judgment.

[0058] The capsule prediction unit is used to predict the hidden state h according to the input t The embryo cyst prediction result is calculated. In the embodiment, the cyst prediction unit can use a fully connected neural network to predict embryo cyst formation. When a key frame is determined to be a key frame, the hidden state of the key frame is input into the cyst prediction unit to calculate the embryo cyst prediction result. When a non-key frame is determined to be a non-key frame, the hidden state of the non-key frame is discarded and does not participate in the cyst prediction. The hidden state of the previous key frame is used to update the hidden state of the current video frame. The updated hidden state of the current video frame is input into the LSTM unit to participate in the calculation of the hidden state of the next video frame. The selection of the transferred state achieves the selection of the video key frame adaptively.

[0059] In this embodiment, the result of the key frame decision unit determining that the frame is a key frame is recorded as (0, 1), and the result of the key frame decision unit determining that the frame is a non-key frame is recorded as (1, 0). Based on this, the hidden layer state of the current video frame that is not a key frame is updated according to the non-key frame judgment result (1, 0). The update formula is:

[0060] h t ′=h t″ ×1+h t ×0

[0061] Among them, h t Indicates the hidden state of the current video frame that is not a key frame, h t″ Indicates the hidden state of the previous key frame, h t ′ represents the updated hidden layer state of the current video frame, and 1 and 0 represent the judgment results of the current video frame being judged as a non-key frame.

[0062] During the training phase, the embryo cyst prediction result of each key frame is used to calculate the loss function to update the model parameters. In the application function phase, since the embryo cyst prediction result of each key frame is calculated based on the hidden state, and the hidden state records the hidden state information of the historical key frames, the embryo cyst prediction result of the last key frame is used as the embryo cyst prediction result of the entire cleavage period video.

[0063] Step 3: Construct the loss function of the embryo cyst formation prediction model.

[0064] In the embodiment, the constructed loss function L=L cls +βL rwd , where L cls The classification loss is constructed based on the predicted results of embryo cyst formation and the true value label. The cross entropy loss can be used. β represents the weight, which ranges from 0 to 10. L rwd is the reward loss of the prediction result corresponding to the key frame. The reward loss is the difference in the correct classification scores between the current key frame and all previous key frames, and the score differences of all key frames are summed up. Specifically:

[0065]

[0066] r t =max(0,d t -max(d t′ ,t′∈(0~t-1)))

[0067]

[0068] Among them, t represents the current time, corresponding to the index of the input video frame at the current time, T is the total time, corresponding to the total number of sampled video frames of a single cleavage stage video, r t The prediction result reward corresponding to the input video frame at time t as the key frame, d t represents the predicted boundary of the input video frame at time t (i.e., the maximum value of the true label prediction value minus the prediction value of other categories), Indicates the predicted value of the true label, c indicates the predicted category, and gt indicates the true label (ground truth);

[0069] Step 4: Use samples and loss functions to optimize the model parameters of the embryo cyst formation prediction model.

[0070] In the embodiment, the sample is used as the input of the embryo cyst prediction model, and the loss function is used to perform supervised learning on the embryo cyst prediction model under the supervision of the true value label. Specifically, a single sample is first sampled at equal intervals of 32 frames, first undergoes data preprocessing operations, and then is unified into the input model. Feature extraction is performed through the first 5 layers of the pre-trained model, and then the absolute position of the sampled image is converted into a continuous vector and spliced ​​with the previously extracted feature vector to obtain the final joint vector. The joint vector is input into the LSTM, and the fully connected layer adaptively determines whether the current frame is the required key frame. If it is, the current prediction result is calculated; if not, the current frame is skipped and the next frame is input. The prediction result of the last frame is used as the final prediction result of the sample, and the loss result of the classification is calculated. For key frames, it is necessary to calculate the improvement of the key frame compared to all previous key frames, and calculate the reward loss. Back propagation is used to help the network select key frames more effectively to optimize the model parameters.

[0071] In this embodiment, the number of iterations is set to 75. In each round, data is input according to a batch size of 3, the aforementioned loss is calculated and back-propagated, and the model parameters are updated until training is completed. The model with the best validation performance is saved during each training round. By modifying hyperparameters, including the learning rate (LR) and the LR decrease rate (K% decrease every N rounds), the loss value, accuracy, recall rate, and other indicators on the validation set are optimized, resulting in better generalization performance of the model.

[0072] In the embodiment, manual labeling of more than 700 frames of normally recorded cleavage stage video is very time-consuming and has a very large redundancy. In the embryo cyst formation prediction model provided in the embodiment, the key frame decision unit is used to adaptively learn and extract key frames of different stages and abnormal cleavage, which can improve the classification effect of the key frame decision unit on whether the embryo has formed a cyst.

[0073] In the embodiment, more than 700 frames of data are still a very large load for the embryo cyst formation prediction model, and the memory length of LSTM cannot cope with such a long sequence. Therefore, 32 frames of cleavage period video are uniformly sampled to eliminate a large amount of redundant data during the cleavage process, and then adaptive frame selection of the key frame decision unit is performed on the basis of 32 frames, which further improves the prediction efficiency.

[0074] In the embodiment, Figure 1 As shown, the embryo video cyst formation prediction device uses the embryo cyst formation prediction model to perform embryo video cyst formation prediction, including the following steps:

[0075] Step 1: Acquire a cleavage stage video monitored by TLM technology. The cleavage stage video is also a video composed of TLM video images from embryo fertilization to embryo development D3.

[0076] Step 2: performing invalid image processing and video frame sampling on the cleavage stage video to obtain a video frame sequence and sampling position information of the video frames.

[0077] In the embodiment, the data is first pre-processed to remove over-exposed and invalid images, and after uniform sampling, batch normalization and batch center cropping are performed to obtain a video frame sequence and sampling position information of the video frames.

[0078] Step 3: Input the video frame sequence and sampling position information into the embryo cyst formation prediction model to obtain the embryo cyst formation prediction result through calculation;

[0079] In one embodiment, multiple groups (generally a batch of 3) of uniformly sampled video frame images can be input into an embryo cyst formation prediction model with adaptive keyframe selection. After calculation, an embryo cyst formation prediction result is obtained. This prediction result is a binary classification result, indicating whether the embryo has cysted or not. In one embodiment, the position with the maximum score in the output result is the data category. The output of the keyframe decision unit can also be used to obtain the adaptively selected keyframes to verify whether the current judgment is reliable.

[0080] In practical applications, the memory can be a local volatile memory such as RAM, or a non-volatile memory such as ROM, FLASH, floppy disk, mechanical hard disk, etc., or a remote storage cloud. The processor can be a central processing unit (CPU), microprocessor (MPU), digital signal processor (DSP), or field programmable gate array (FPGA), which can be used to implement the embryo video cystogenesis prediction step based on adaptive key frame selection.

[0081] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An embryo video cyst prediction device based on adaptive key frame selection, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: The memory stores an embryo cyst formation prediction model, and the processor implements the following steps when executing the computer program: Acquire a cleavage-stage video obtained by time-lapse monitoring technology, and perform video frame sampling on the cleavage-stage video to obtain a video frame sequence and sampling position information of the video frames; The video frame sequence and sampling position information are input into the embryo cyst formation prediction model, and the embryo cyst formation prediction result is obtained through calculation; The embryo cyst prediction model includes a feature extraction unit, a position information conversion unit, an LSTM unit, a key frame decision unit and a cyst prediction unit, wherein a video frame is input into the feature extraction unit to obtain a feature vector through feature extraction, the sampling position information is input into the position information conversion unit to obtain a position vector through conversion, the feature vector and the position vector are spliced ​​and input into the LSTM unit to output a hidden layer state after processing, the hidden layer state is input into the key frame decision unit to determine whether the output is a key frame, the hidden layer state determined to be a key frame is input into the cyst prediction unit to calculate the embryo cyst prediction result; the hidden layer state determined to be a non-key frame is discarded and does not participate in the cyst prediction, and the hidden layer state of the current video frame is updated using the hidden layer state of the previous key frame, the updated hidden layer state of the current video frame is input into the LSTM unit to participate in the hidden layer state calculation of the next video frame, and the embryo cyst prediction result of the last key frame is used as the final embryo cyst prediction result; The position information conversion unit performs vector conversion on the input position information in the following manner: First, the formula Calculate the first transformation vector PE (pos,2i) , where pos represents the input sampling position information, 2i represents the vector dimension, and d model is a preset constant with a value of 720-900; then, use the formula Calculate the second transformation vector PE (pos,2i+1) ; Finally, concatenate the first transformation vector PE (pos,2i) and the second transformation vector PE (pos,2i+1) To get the position vector; Among them, the key frame decision unit adopts a fully connected neural network.

2. The embryo video cyst prediction device based on adaptive key frame selection according to claim 1, wherein Before the embryo cyst formation prediction model is applied, model parameter optimization is required. The model parameter optimization process includes: Construct a sample set, using the video frame sequence and the corresponding sampling position information sequence obtained by sampling a single cleavage-stage video as input data, and whether the embryo has formed a cyst as the true value label to form a single sample; Construct the loss function, loss function L=L cls +βL rwd , where L cls is the classification loss constructed based on the predicted results of embryo cyst formation and the true value label, β represents the weight, ranging from 0 to 10, L rwd is the reward loss of the prediction result corresponding to the key frame, specifically: r t =max(0,d t -max(d t′ ,t′∈(0~t-1))) Among them, t represents the current time, corresponding to the index of the input video frame at the current time, T is the total time, corresponding to the total number of sampled video frames of a single cleavage stage video, r t The prediction result reward corresponding to the input video frame at time t as the key frame, d t Represents the predicted boundary of the input video frame at time t, that is, the maximum value of the true label prediction value minus the prediction value of other categories, Represents the predicted value of the true label, c represents the predicted category, and gt represents the true label; The samples are used as the input of the embryo cyst formation prediction model, and the loss function is used to perform supervised learning on the embryo cyst formation prediction model under the supervision of the true value label to optimize the model parameters.

3. The embryo video cyst prediction device based on adaptive key frame selection according to claim 2, characterized in that: When constructing the sample set, invalid filtering is performed on the acquired cleavage-stage videos, including: first filtering out empty culture dishes and overexposed video frames, then filtering out cleavage-stage videos whose number of video frames does not meet the preset video frame threshold, and the remaining cleavage-stage videos participate in video frame sampling to construct the sample.

4. The embryo video cyst prediction device based on adaptive key frame selection according to claim 1, characterized in that: The feature extraction unit uses at least the first 5 layers of the ResNet50 network.

5. The embryo video cyst prediction device based on adaptive key frame selection according to claim 1, characterized in that: The key frame decision unit calculates the input hidden layer state to output a judgment result. When the judgment result is greater than or equal to 0.5, the video frame corresponding to the input hidden layer state is considered to be a key frame. When the judgment result is less than 0.5, the video frame corresponding to the input hidden layer state is considered to be a non-key frame.

6. The embryo video cyst prediction device based on adaptive key frame selection according to claim 1, characterized in that: The result of the key frame decision unit judging it as a key frame is recorded as (0,1), and the result of the key frame decision unit judging it as a non-key frame is recorded as (1,0). Based on this, the hidden layer state of the current video frame that is not a key frame is updated according to the judgment result (1,0) of the non-key frame. The update formula is: h t ′=h t ″×1+h t ×0 Among them, h t Indicates the hidden state of the current video frame that is not a key frame, h t″ Indicates the hidden state of the previous key frame, h t ′ represents the updated hidden layer state of the current video frame, and 1 and 0 represent the judgment results of the current video frame being judged as a non-key frame.

7. The embryo cyst prediction device based on adaptive key frame selection according to claim 1 or 2, characterized in that: After sampling the video frames of the cleavage stage video, data enhancement is performed on the sampled video frame images to form a video frame image sequence. The data enhancement includes scaling and cropping, random horizontal flipping, random vertical flipping, random brightness change, and random angle rotation.

Images