An embryo future development state generation method based on a space-time flow attention driven diffusion model

Abstract

The application discloses an embryo future development state generation method based on a space-time flow attention driving diffusion model, and belongs to the technical field of artificial intelligence and medical images. Existing embryo evaluation technologies mostly adopt discriminative models, output single classification or scores, lose visual details and have poor interpretability, and cannot provide prospective insight. The application obtains an embryo time sequence microscopic image sequence, extracts space-time condition features, inputs random noise images, time step parameters and the features into a U-Net-based denoising network, fuses the features through a cross attention module to guide an iterative denoising process, generates a visual image of a future development state, and outputs after post-processing. The method generates high-fidelity images, improves interpretability, provides prospective visual assistance for clinicians, and promotes the application of precision assisted reproduction.

Description

Technical Field

[0001] This invention belongs to the field of interdisciplinary technology of artificial intelligence and medical imaging, and in particular relates to a method for generating the future developmental state of an embryo based on a spatiotemporal flow attention-driven diffusion model. Background Technology

[0002] In vitro fertilization (IVF) is a key technology for treating infertility, its core being the selection of embryos with the highest implantation potential by assessing their dynamic development. Temporal microscopy can non-invasively capture this continuous dynamic process, providing high-dimensional data rich in spatiotemporal information for accurately assessing embryo health and developmental potential. However, interpreting the future developmental trends of the embryo from this complex video data and translating it into intuitive auxiliary information that clinicians can understand has become a major bottleneck limiting the clinical effectiveness of this technology.

[0003] Currently, most existing AI-based embryo assessment technologies are discriminative models, which typically compress the complex process of embryonic development into a single, low-dimensional classification or score. This approach reveals significant shortcomings in actual clinical decision-making. First, the process loses a large amount of visual detail, resulting in assessment results lacking intuitive morphological evidence, making it difficult for clinicians to conduct in-depth morphological analysis. Second, these models are black-box in nature; their internal decision-making logic is opaque, making the output scores or classifications difficult to interpret and trust. This greatly limits the depth and credibility of such technologies in critical clinical decisions. Furthermore, existing technologies are essentially retrospective ratings of past embryonic developmental states, failing to provide any forward-looking visual insights into their future developmental potential, which is precisely the most pressing need in clinical screening for high-quality embryos.

[0004] Therefore, generating visualizations of future critical states based on the embryo's past developmental sequence has significant theoretical and practical value for assisting clinical decision-making and improving in-vitro fertilization success rates. However, achieving this goal faces technical challenges, the core of which lies in how to effectively extract features representing developmental dynamics from high-dimensional time-series data and use these features to generate high-fidelity images of future states in an interpretable manner, which exceeds the capabilities of traditional discriminative models. Summary of the Invention

[0005] To address the aforementioned technical issues, this invention proposes a method for generating future embryonic developmental states based on a spatiotemporal flow attention-driven diffusion model. This method generates high-fidelity images, enhances interpretability, provides prospective visual assistance to clinicians, and promotes the application of precision assisted reproduction.

[0006] This invention provides a method for generating future embryonic developmental states based on a spatiotemporal flow attention-driven diffusion model, comprising the following steps:

[0007] S1. Obtain a time-series microscopic image sequence of embryos during in vitro culture;

[0008] S2. Spatiotemporal condition features are extracted from the time-series microscopic image sequence using a video encoder; the video encoder integrates a spatiotemporal flow attention module for weighted fusion of spatial information, temporal information, and frequency domain information of the feature map;

[0009] S3. Based on the spatiotemporal condition features, a visual image of the future developmental state of the embryo is generated by using a temporal conditional diffusion model in an iterative denoising manner;

[0010] S4. Construct a loss function that includes optical flow consistency loss to train the model and constrain the motion consistency of the generated visual image sequence.

[0011] As a further improvement of the present invention, in S2, the video encoder includes a three-dimensional convolution module, a spatiotemporal flow attention module, and a bidirectional long short-term memory network module connected in sequence; the spatiotemporal flow attention module includes a spatial feature extraction branch, a temporal feature extraction branch, and a frequency domain feature extraction branch set in parallel; the frequency domain feature extraction branch is used to perform fast Fourier transform on the feature map, and recover it by inverse fast Fourier transform after frequency domain convolution, so as to enhance the capture of high-frequency details.

[0012] As a further improvement of the present invention, the spatial feature extraction branch utilizes Convolutional kernels extract local spatial features, utilizing and and The convolutional kernel extracts horizontal and vertical spatial features respectively, and then combines the three as the output spatial features.

[0013] As a further improvement of the present invention, the video encoder also includes a temporal attention mechanism for calculating the attention weight of each time step in the Bi-LSTM output sequence and performing a weighted summation of the output to obtain the final spatiotemporal condition features.

[0014] As a further improvement of the present invention, in S3, the temporal conditional diffusion model includes a modulator network; the modulator network receives the concatenated vector of the spatiotemporal conditional features and the time step embedding vector to generate adjustment parameters; the adjustment parameters are applied to the output of the convolutional layer of the diffusion model through feature affine transformation to achieve conditional injection, as shown in the formula:

[0015] .

[0016] As a further improvement of the present invention, in S4, the optical flow consistency loss is obtained by calculating the difference in optical flow field between the predicted image frame and the real image frame, and the calculation formula is as follows:

[0017] .

[0018] As a further improvement of the present invention, the loss function further includes a weighted combination of L1 pixel loss, structural similarity loss (SSIM), and perceptual loss.

[0019] As a further improvement of the present invention, the specific execution steps of the spatiotemporal flow attention module include: spatial feature extraction branch: extracting local spatial features, horizontal spatial features and vertical spatial features of the feature map using convolution kernels of different sizes, and superimposing the three to obtain spatial feature output; frequency domain feature extraction branch: performing fast Fourier transform on the input feature map to obtain frequency domain representation, performing convolution operation in the frequency domain, and restoring the frequency domain features to the spatiotemporal domain through inverse fast Fourier transform to obtain frequency domain feature output; feature fusion: concatenating the spatial feature output, temporal feature output and frequency domain feature output, calculating an adaptive weighting factor through a fusion gate mechanism, and performing weighted fusion of the features of each branch.

[0020] As a further improvement of the present invention, in S3, the iterative execution of the denoising operation at multiple time steps includes: iterating over a preset total number of time steps; at each time step, inputting the current noisy image, time step parameters, and spatiotemporal conditional features into a conditional denoising network; the conditional denoising network fuses the spatiotemporal conditional features through a cross-attention module to predict the noise added at the current time step; using a scheduler algorithm to calculate the image of the previous time step with less noise based on the predicted noise and the current image; repeating this process until the time step ends to obtain the final denoised image.

[0021] As a further improvement of the present invention, the conditional denoising network is based on the U-Net architecture and includes multiple downsampling modules, upsampling modules, and cross-layer connections. Some of the modules integrate cross-attention layers to receive the spatiotemporal conditional features. The cross-attention modules are used to deeply fuse the spatiotemporal conditional features to guide the conditional denoising network to accurately predict noise.

[0022] As a further improvement of the present invention, in S4, post-processing of the generated visualization image includes: performing inverse normalization on the generated visualization image tensor, restoring the numerical range of the inverse normalized image from the generated range to the image pixel value range; and converting the processed tensor into a standard image file format and saving or displaying it.

[0023] Technical Advantages of this Invention: This invention discloses a method for generating future embryonic developmental states based on a spatiotemporal flow attention-driven diffusion model, transforming embryo assessment from a single discriminative output to the generation of high-fidelity visual images. The method outputs images of future developmental states containing rich morphological details, rather than low-dimensional classification results or scores, thus greatly improving the interpretability of the results and enabling clinical experts to intuitively understand the potential developmental trends of embryos. This invention provides, for the first time, a forward-looking visual insight into the developmental potential of embryos, rather than merely a retrospective rating of past states, offering clinicians a new, intuitive, and reliable visual aid for selecting embryos with the highest implantation potential. By fully utilizing the deep information in time-series data, this invention provides a more reliable decision-making tool for the in-depth application of artificial intelligence technology in the field of precision assisted reproduction. Attached Figure Description

[0024] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0025] Figure 1 This is a flowchart illustrating a method for generating a visualization image of the future developmental state of an embryo based on a diffusion model, according to an embodiment of the present invention.

[0026] Figure 2 This is a structural diagram of the spatiotemporal flow attention-driven diffusion model according to an embodiment of the present invention;

[0027] Figure 3 This is a schematic diagram of the video encoder network structure according to an embodiment of the present invention;

[0028] Figure 4 This is a schematic diagram of the Spatiotemporal Flow Attention (STFA) network structure according to an embodiment of the present invention;

[0029] Figure 5 This is a schematic diagram of the network structure of the temporal conditional diffusion model in an embodiment of the present invention. Detailed Implementation

[0030] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0031] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0032] like Figure 1As shown, this embodiment provides a method for generating the future developmental state of an embryo based on a spatiotemporal flow attention-driven diffusion model, specifically including the following steps:

[0033] S1. Obtain a time-series microscopic image sequence of embryos during in vitro culture;

[0034] In this embodiment, the data acquisition and preprocessing process is as follows:

[0035] Data Reading: Read consecutive image frames from the embryo culture dataset. Based on the code settings, read the first N frames (e.g., 20 or 80 frames) as the conditional input sequence, and use the (N+1)th frame as the target image to be predicted.

[0036] The processed N-frame conditional images are stacked along the channel dimension or the time dimension to form a conditional video tensor with dimensions [C,T,H,W], where C is the number of channels, T is the number of frames, and H and W are the height and width.

[0037] S2. Extract spatiotemporal condition features from the time-series microscopic image sequence using a video encoder;

[0038] This step utilizes a video encoder to extract features. This process integrates a spatiotemporal flow attention module (STFA), the specific structure of which is as follows: Figure 2 As shown:

[0039] Spatiotemporal features characterizing the dynamic development of the embryo are extracted from the time-series microscopic image sequence. The input tensor undergoes feature extraction in three stages, each stage including: a 3D convolutional layer (Conv3d), a batch normalization layer (BatchNorm3d), a ReLU activation function, a spatiotemporal flow attention module (STFA), and a max pooling layer (MaxPool3d). As the number of layers increases, the number of channels increases sequentially (e.g., 64->128->256), and the spatial resolution gradually decreases.

[0040] The specific implementation of the SpatioTemporalAwareBlock (STFA) module: This module processes feature maps through three parallel branches, such as... Figure 4 As shown:

[0041] Spatial feature extraction branch: Employs multi-scale convolutional kernels to extract spatial information. Specifically, this includes: using... Convolutional kernels extract local spatial features; using Convolutional kernels extract long-range features in the horizontal direction; using Convolutional kernels extract long-range features in the vertical direction. The outputs of the three types of convolutions are summed to obtain the spatial feature output.

[0042] Temporal feature extraction branch: using differential convolution kernels with fixed weights;

[0043] Frequency domain feature extraction branch: Performs a Fast Fourier Transform on the input feature map to transform the features from the spatiotemporal domain to the frequency domain; concatenates the real and imaginary parts of the frequency domain features, and then... The convolutional layer performs frequency domain feature interaction; the inverse fast Fourier transform is used to restore the processed features back to the spatiotemporal domain, resulting in frequency domain feature output.

[0044] The original input, spatial feature output, temporal motion edge, and frequency domain feature output are concatenated along the channel dimension. An adaptive weighting factor is calculated through a fusion gate consisting of global average pooling, convolution, ReLU, and Sigmoid. Finally, the fused features are output through a projection layer.

[0045] Temporal modeling: such as Figure 3 As shown, after passing through the backbone network, the feature maps are compressed in spatial dimension using an adaptive average pooling layer (AdaptiveAvgPool3d). A bidirectional long short-term memory (Bi-LSTM) network is used to process the compressed feature sequence, capturing bidirectional long temporal dependencies. A temporal attention mechanism is introduced, calculating the attention weights for each time step through fully connected layers and Softmax, weighting and summing the LSTM outputs, and finally mapping them to a fixed-dimensional spatiotemporal conditional feature vector through a fully connected layer.

[0046] S3. Based on the spatiotemporal condition features, a visual image of the future developmental state of the embryo is generated by using a temporal conditional diffusion model in an iterative denoising manner;

[0047] This step utilizes a U-Net-based conditional denoising network (VideoConditionedUNet) to generate images, such as... Figure 5 As shown.

[0048] Conditional Injection Mechanism (Feature Modulation): This embodiment employs a feature modulation mechanism instead of simple concatenation. The spatiotemporal conditional features extracted in step S2 are concatenated with the current time-step embedding vector to form a joint conditional vector. A modulator network (composed of multiple fully connected layers) maps the joint conditional vector to adjustment parameters Scale and Shift. In each downsampling, intermediate, and upsampling module of U-Net, a "hook" mechanism intercepts the output of the normalization layer, utilizing the formula... An affine transformation is performed on the feature map. This allows the spatiotemporal conditions to precisely guide the feature generation at each layer of the denoising process.

[0049] Iterative denoising process:

[0050] S31 Initialization: Generate an initial image tensor of the same size as the target image, consisting entirely of Gaussian random noise. .

[0051] S32 Iteration: Iterates backwards at preset time steps using a scheduler (DDPMScheduler). At each time step t:

[0052] Current noisy image Input U-Net; U-Net combines the injected spatiotemporal features to predict the noise distribution in the current image; based on the predicted noise, the scheduler formula is used to calculate the previous time-step image with less noise. .

[0053] Repeat until t=0 to obtain the generated denoised image.

[0054] S4. Construct a loss function that includes optical flow consistency loss to train the model and constrain the motion consistency of the generated visualization image sequence;

[0055] To ensure that the generated embryo images are spatiotemporally coherent and have realistic textures, this embodiment constructs a composite loss function during the training phase:

[0056] ;

[0057] in, To calculate the pixel-level Manhattan distance between the generated image and the ground truth image, Calculate the SSIM index between the generated image and the real image to constrain structural consistency. Features are extracted using a pre-trained VGG19 network, and content loss and style loss are calculated separately to improve the texture details and visual perception quality of the image. Optical flow consistency loss utilizes a pre-trained optical flow estimation network (such as the RAFT model) as a frozen feature extractor to calculate the true optical flow field from the previous conditional image to the current true frame, and the predicted optical flow field from the previous conditional image to the current generated frame. The L1 distance between the two optical flow fields is then calculated. This loss forces the model-generated image to maintain consistency with real physical development patterns in terms of motion trends, effectively suppressing jitter and artifacts in the generated video.

[0058] S5. The image tensor values ​​output by the post-processing model range from [-1, 1]. The post-processing steps include:

[0059] Performing inverse normalization maps the values ​​back to [0,1]. The processed data is then saved as a standard image file (such as PNG or JPEG) for clinicians to visualize and evaluate.

[0060] An application example of this invention:

[0061] (1) Acquire and preprocess embryo time sequence images.

[0062] In this embodiment, consecutive image frames are read from the embryo culture dataset, where the number of conditional video frames (cond_frames) is 20, and the images are single-channel grayscale images. The spatial resolution of each image frame is uniformly processed to 128×128 pixels. After preprocessing operations such as ToTensor and Normalize(-1,1), the dimension of the obtained conditional video tensor is 1×20×128×128 (number of channels × number of frames × height × width).

[0063] (2) Extract multidimensional spatiotemporal conditional features by video encoder. Input the conditional video tensor obtained in step (1) into video encoder to extract depth features that represent developmental dynamics.

[0064] The video tensor first undergoes a 3D convolutional module to extract shallow features. It then enters a Spatiotemporal Flow Attention (STFA) module, which executes three branches of operations in parallel. Convolutional kernels extract local features and utilize them respectively. and The asymmetric convolutional kernels capture long-range spatial dependencies in the horizontal and vertical directions, and finally, the three are superimposed. A Fast Fourier Transform (FFT) is performed on the feature maps, followed by convolutional interactions in the frequency domain, and then the spatiotemporal domain is recovered using an Inverse Fast Fourier Transform (IFFT). This step significantly enhances the model's ability to capture high-frequency details such as embryonic cell edges and fragments. Motion edge features between frames are extracted. Finally, the features from the spatial, frequency, and temporal branches are concatenated, and adaptive weights are calculated and weighted fused using a fusion gate mechanism. The fused feature sequence is input into a Bidirectional Long Short-Term Memory (Bi-LSTM) network to capture forward and reverse developmental dependencies. Subsequently, the weights for each time step are calculated using a temporal attention mechanism, and the outputs of the Bi-LSTM are weighted and summed, ultimately outputting a spatiotemporal conditional feature vector with a dimension of 768.

[0065] The spatiotemporal conditional features obtained in step (2) are used as a guide to generate future developmental images based on the temporal conditional diffusion model. First, a 1×128×128 Gaussian random noise image is generated. In each layer of the denoising network, a modulator network is used to process the spatiotemporal conditional features and the concatenated vector of time-step embedding vectors, generating adjustment parameters scale and shift. Using the formula... Affine transformation is performed on the convolutional layer outputs of the denoising network (U-Net). This method injects historical developmental information into the generation process more accurately than simple concatenation.

[0066] The network iterates backwards over 50 time steps using a scheduler. At each step, the network predicts and removes the current noise, gradually restoring a clear anatomical structure of the embryo, ultimately obtaining the predicted tensor of the 21st frame image.

[0067] Optical flow consistency loss, combined with L1 pixel loss, SSIM loss, and perceptual loss, optimizes model parameters, forcing the images generated by the model to conform to the real biological laws of embryonic development in terms of motion trends.

[0068] Image post-processing and output: The image tensor generated in step (3) is inversely normalized to map the numerical range from [-1,1] back to [0,255] and converted to an 8-bit grayscale image format for saving or display, providing clinicians with intuitive auxiliary information on the future development status of the embryo.

[0069] The method proposed in this embodiment can reconstruct a high-dimensional, high-fidelity visual image of the future developmental state of an embryo from its past high-dimensional temporal image sequence, thereby obtaining an assessment basis that has both objective data foundation and intuitive visual interpretability. This can further promote the in-depth application of generative artificial intelligence in the fields of precision medicine and assisted reproduction.

[0070] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for generating embryo images based on a spatiotemporal flow attention-driven diffusion model, characterized in that, Includes the following steps: S1. Obtain a time-series microscopic image sequence of embryos during in vitro culture; specifically, this includes: reading consecutive image frames from the embryo culture dataset, and obtaining conditional video tensors after preprocessing. S2. Spatiotemporal conditional features are extracted from the time-series microscopic image sequence using a video encoder. The video encoder integrates a spatiotemporal flow attention module for weighted fusion of spatial, temporal, and frequency domain information from the feature map. The video encoder includes a three-dimensional convolution module, a spatiotemporal flow attention module, and a bidirectional long short-term memory network module connected in sequence. The spatiotemporal flow attention module contains parallel spatial feature extraction branches, temporal feature extraction branches, and frequency domain feature extraction branches. The frequency domain feature extraction branch performs a fast Fourier transform on the feature map and recovers it through an inverse fast Fourier transform after frequency domain convolution to enhance the understanding of the feature map. High-frequency detail capture includes: first, the conditional video tensor obtained by S1 is processed through a 3D convolutional module to extract shallow features; then, it enters a spatiotemporal flow attention module, where the operations of three branches are executed in parallel; the features of the spatial feature extraction branch, the temporal feature extraction branch, and the frequency domain feature extraction branch are concatenated, and adaptive weights are calculated and weighted fusion is performed through a fusion gate mechanism; the fused feature sequence is input into a bidirectional long short-term memory network to capture forward and backward developmental dependencies; subsequently, the weights of each time step are calculated through a temporal attention mechanism, and the output of the Bi-LSTM is weighted and summed to finally output spatiotemporal conditional features; S3. Based on the spatiotemporal condition features, a visualization image of the future developmental state of the embryo is generated through an iterative denoising method using a temporal conditional diffusion model; the temporal conditional diffusion model includes a modulator network; the modulator network receives the concatenated vector of the spatiotemporal condition features and the time step embedding vector to generate adjustment parameters; S4. Construct a loss function that includes optical flow consistency loss to train the temporal conditional diffusion model and constrain the motion consistency of the generated visual image sequence.

2. The method as described in claim 1, characterized in that, The spatial feature extraction branch utilizes Convolutional kernels extract local spatial features and utilize and The convolutional kernel extracts horizontal and vertical spatial features respectively, and then combines the three as the output spatial features.

3. The method as described in claim 1, characterized in that, In S3, the temporal conditional diffusion model includes a modulator network; the modulator network receives the concatenated vector of the spatiotemporal conditional features and the time step embedding vector to generate adjustment parameters; the adjustment parameters are applied to the output of the convolutional layer of the diffusion model through feature affine transformation to achieve conditional injection, as shown in the formula: 。 4. The method as described in claim 1, characterized in that, In S4, the optical flow consistency loss is obtained by calculating the difference in optical flow field between the predicted image frame and the real image frame, and the calculation formula is as follows: 。 5. The method as described in claim 4, characterized in that, The loss function also includes a weighted combination of L1 pixel loss, structural similarity loss, and perceptual loss.

Images