Multi-focal embryo image fusion method based on multi-gradient collaborative fusion and structure global decoding

Abstract

The application discloses a multi-focal embryo image fusion method based on multi-gradient collaborative fusion and structure global decoding, comprising the following steps: 1. obtaining multiple images of the same embryo under different focal lengths, and generating a multi-channel input feature map by preprocessing; 2. constructing a parallel multi-branch gradient feature extraction network, each branch being composed of a multi-scale convolution attention module, and being used for extracting gradient features of different types; 3. dynamically weighting and fusing multi-scale gradient features output by each branch through a gating mechanism to generate unified fusion features; 4. constructing a structure global decoding network to reconstruct the fusion features into a clear focused image; 5. jointly introducing a self-supervised gradient alignment loss and a structural similarity loss to optimize network parameters; and 6. training and testing on a constructed dataset to complete the multi-focal embryo image fusion task and improve the focusing quality and structure restoration degree of the embryo image.

Description

Technical Field

[0001] This invention relates to the field of image autofocusing and image fusion technology, and more specifically, to a multifocal embryo image fusion method based on multi-gradient collaborative fusion and structure-wide decoding. Background Technology

[0002] In vitro fertilization (IVF), as one of the most widely used and technologically mature assisted reproductive methods, involves core processes including oocyte retrieval, in vitro fertilization, embryo culture, and the selection and transfer of high-quality embryos. Among these, the assessment and selection of embryo quality directly determines the success rate of pregnancy and is a crucial step in the entire IVF treatment process. Clinically, doctors typically rely on embryo development images acquired through a time-lapse culture system to systematically analyze the morphological characteristics of the embryos to assess their developmental potential and implantation likelihood. This assessment process is not only the core basis for embryo selection but also a key factor influencing the success of an IVF cycle.

[0003] Embryo images are typically acquired using a camera system within a time-lapse incubator. To obtain structurally complete and clearly focused images, clinical practice generally avoids using only a fixed focal length for image acquisition. Due to factors such as focal length settings, embryo positional shifts, and uneven culture dish thickness, single images often appear blurry or lack information. Therefore, acquiring multiple images at different focal lengths consecutively within a very short timeframe has become standard operating procedure. For multiple focal length images of the same embryo acquired at the same time point, automatically and accurately selecting or fusing them into the clearest image has become one of the key challenges in current IVF image processing workflows.

[0004] Traditional methods typically rely on a series of focus evaluation metrics to quantitatively analyze image quality and select images with the optimal focal length for subsequent evaluation. These metrics include, but are not limited to, image sharpness, edge sharpness, texture detail intensity, and frequency domain energy distribution. While these metrics can reflect the image's focus effect to some extent and help identify the image samples with the clearest structure, these methods generally depend on predefined rules and manually extracted features. They are significantly affected by factors such as imaging conditions, parameter selection, and image complexity, resulting in a highly subjective evaluation process and weak stability and generalization ability.

[0005] Furthermore, with the continuous growth of embryonic image data in clinical and research settings, traditional methods face bottlenecks in processing efficiency and automation, making it difficult to meet the practical needs of high-throughput image analysis. A more complex situation arises because real-world multifocal images often exhibit localized sharpness while other areas remain blurred, indicating spatial non-uniformity in focus. This inconsistent spatial focal length distribution makes image selection strategies based on unified evaluation metrics insufficient to fully reflect the true interpretability and clinical value of images. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies in multifocal embryo image processing, this invention proposes a multifocal embryo image fusion method based on multi-gradient collaborative fusion and structural global decoding. This method aims to significantly improve the overall focusing quality and structural restoration capability of the fused images, thereby providing a more stable and reliable image basis for morphological assessment and clinical interpretation.

[0007] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0008] The present invention provides a multi-focal-length embryo image fusion method based on multi-gradient collaborative fusion and structure-wide decoding, characterized by the following steps:

[0009] Step 1: Obtain two microscopic images of the embryo taken at the same time with different focal lengths. ,in, This represents the first microscopic image of the embryo. This refers to the second microscopic image of the embryo. Indicates the length of the embryo micrograph. 3 represents the width of the embryo micrograph; 3 represents the number of channels in the embryo micrograph.

[0010] right Grayscale processing was performed to obtain a set of embryonic microscopic grayscale images. ,in, This represents the first grayscale image of the embryo. This represents the second grayscale image of the embryo; 1 represents the number of channels in the grayscale image of the embryo.

[0011] Based on the channel dimension of the grayscale image and Stacking them together forms a multi-channel input feature map. ;

[0012] take Extracting different gradient operators respectively Structural gradient plots ,in, Indicates the first The group of structure gradient plots extracted by the gradient operator, Indicates the first Gradient operator extraction Structural gradient plot, Indicates the first Gradient operator extraction Structural gradient plot;

[0013] right and Perform a pixel-by-pixel maximum value operation to obtain the first pixel. Zhang embryo reference gradient map Thus obtain Zhang embryo reference gradient map ;

[0014] Step 2: Construct a multi-gradient collaborative fusion network, including: a multi-scale feature extraction module guided by multiple gradient features, a feature fusion module, and perform... Processing was performed to obtain multi-scale embryonic gradient fusion features. ;

[0015] Step 3: Construct a global decoding module for embryonic structure, including: a patch embedding layer, a rotation-invariant position encoding layer, a multi-layer global decoding unit, and a patch restoration layer; wherein, the multi-layer global decoding unit consists of multiple Transformer decoding layers with the same structure, each Transformer decoding layer including: a multi-head attention unit, a feedforward neural network, and normalization and residual connection units; and perform... Processing is performed to obtain a focused embryo image. ;

[0016] Step 4: Construct the joint loss function ;

[0017] Step 5: Train the multifocal embryo image fusion network using the AdamW optimizer and minimize the joint loss function. The network parameters are updated to obtain the optimal multifocal embryo image fusion model, which is used to achieve adaptive focusing fusion of multifocal embryo images.

[0018] The multi-focal-length embryo image fusion method based on multi-gradient collaborative fusion and structure-wide decoding described in this invention is characterized in that step 2 is performed as follows:

[0019] Step 2.1: Construct a multi-scale feature extraction module, including: There are 10 parallel branches, each containing 10 parallel branches. The system comprises several convolutional attention modules of different scales; each scale includes a convolutional layer, a normalization layer, an activation function layer, and a cross-channel attention mechanism layer; the cross-channel attention mechanism layer includes a pooling layer, a fully connected layer, an activation function layer, and a channel attention weighting layer; and... Enter the first After passing through several parallel branches, they proceed sequentially. Processing convolutional attention modules of different scales to obtain the first... The intermediate gradient feature sequence output by each branch ,in, Indicates the first The intermediate gradient feature at the k-th scale of the output of each branch. , ;

[0020] Step 2.2: Construct a feature fusion network, including: a concatenation unit, a gating network, and a gradient fusion unit; wherein, the gating network includes: two convolutional layers, an activation function layer, and a normalization processing unit; and processes the intermediate gradient feature sequences output by all branches. Processing to obtain embryo fusion feature maps .

[0021] Furthermore, step 2.2 is performed as follows:

[0022] Step 2.2.1, the splicing unit uses formula (1) to... After concatenating intermediate gradient features of the same scale in each branch according to channel, we obtain... splicing features at different scales :

[0023] (1)

[0024] In equation (1), Indicates the first splicing features at various scales; Indicates a splicing operation;

[0025] Step 2.2.2, will The input is processed in a gating network to obtain the first... Under each scale Weighted graph Thus obtain Weighting maps at different scales ,in, Indicates the first The first scale Each weight;

[0026] Step 2.2.3: The gradient fusion unit obtains the result using equation (2). Fusion gradient features at different scales :

[0027] (2)

[0028] In equation (2), This represents element-wise multiplication; Indicates the first Fusion gradient features at various scales;

[0029] Step 2.2.4: Use equation (3) to fuse gradient features Perform upsampling and channel-wise stitching to obtain stitching features. :

[0030] (3)

[0031] In equation (3), Indicates to Perform an upsampling operation to make its size consistent with... Alignment;

[0032] Step 2.2.5, will The input is processed in a gating network to obtain... Weighted graph The gradient fusion unit uses equation (4) to obtain the embryo fusion feature map. :

[0033] (4)

[0034] In equation (4), This represents the j-th weighted graph.

[0035] Furthermore, step 3 is performed as follows:

[0036] Step 3.1: Obtain the embryo fusion feature map The input is divided into a set of two-dimensional image patches in the Patch embedding layer, and each two-dimensional image patch is mapped to a token vector of uniform length;

[0037] Step 3.2: The rotation-invariant position encoding layer uses two-dimensional polar coordinate position encoding to add a rotation-invariant position vector to be learned to each token vector, and then fuses it with the corresponding token vector to obtain different global position-aware token vectors.

[0038] Step 3.3: Input each global location-aware token vector into the multi-layer global decoding unit for decoding processing to obtain the corresponding global location-aware token decoding vector;

[0039] Step 3.4: The Patch reconstruction layer uses linear projection to reconstruct each global location-aware token decoding vector into its corresponding image patch vector, thereby... The original spatial location of the two-dimensional image patch is used to reconstruct the image patch vector, resulting in a reconstructed focused embryo image. .

[0040] Furthermore, step 4 is performed as follows:

[0041] Step 4.1: Establish the self-supervised gradient feature alignment loss using equation (5):

[0042] (5)

[0043] In equation (5), Representative embryo reference gradient map Embryo reference gradient map after normalization. Indicates the first Zhang's reference gradient diagram of embryos after normalization; represent The output of the branch intermediate gradient feature sequences at different scales The intermediate gradient features after normalization. Indicates the first The output of the branch Intermediate gradient features after normalization at each scale; represent Self-supervised gradient feature alignment loss for each branch Indicates the first Self-supervised gradient feature alignment loss for each branch;

[0044] Step 4.2: Establish the structural similarity loss function using equation (6). :

[0045] (6)

[0046] In equation (6), Represents the structural similarity index;

[0047] Step 4.3: Construct the joint loss function using equation (7). :

[0048] (7)

[0049] In equation (7), represent The corresponding regularization parameters; and This represents two regularization parameters.

[0050] The present invention provides an electronic device, comprising a memory and a processor, wherein the memory is used to store a program that supports the processor in executing the multifocal embryo image fusion method, and the processor is configured to execute the program stored in the memory.

[0051] The present invention discloses a computer-readable storage medium on which a computer program is stored, wherein the computer program, when executed by a processor, performs the steps of the multifocal embryo image fusion method.

[0052] Compared with existing technologies, the beneficial effects of this invention are reflected in:

[0053] 1. This invention proposes an end-to-end multi-focal-length embryo image fusion network model for automatically processing multiple embryo images with different focal lengths and generating a clear, information-complete fused image. This method requires no manual intervention, effectively improving the efficiency and consistency of image processing, and solving the problems of reliance on manual screening, cumbersome operation, and poor repeatability in traditional methods.

[0054] 2. This invention designs a parallel multi-branch gradient feature extraction network, with each branch containing a multi-scale convolutional attention module. The multi-scale convolutional module extracts image gradient features at different scales, and the cross-channel attention mechanism adaptively enhances the channel responses highly correlated with embryonic structure, providing multi-level and complementary feature support for subsequent fusion.

[0055] 3. In the feature fusion stage, this invention introduces a gated feature fusion mechanism. This mechanism first concatenates the gradient features extracted from each branch along the channel dimension, and then dynamically generates multiple spatial weight maps through a gated network, achieving pixel-by-pixel weighted integration of the features from each branch. This approach effectively enhances the structural response of clear regions while suppressing redundant or blurred information, thereby achieving efficient fusion of multi-scale gradient features.

[0056] 4. In the embryo image reconstruction stage, this invention constructs a global decoding network for embryo structure. This network adopts a global modeling strategy based on rotation-invariant position encoding to achieve spatial semantic restoration of fused features within the decoding framework. By jointly introducing self-supervised gradient feature alignment loss and unsupervised structural similarity loss, a complete optimization objective function is constructed, effectively improving the focusing degree and structural fidelity of the fused image. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of the overall process of the method proposed in this invention;

[0058] Figure 2 This is a schematic diagram of the structure of the multi-branch gradient feature extraction network in this invention;

[0059] Figure 3 This is a schematic diagram illustrating the process of multi-gradient feature fusion in the gated network of this invention;

[0060] Figure 4 A visual comparison of the fusion effects of different methods on the same set of embryo images. Detailed Implementation

[0061] In this embodiment, a multi-focal-length embryonic image fusion method based on multi-gradient collaborative fusion and structure-wide decoding firstly performs grayscale and stacking processing on the embryonic image data; secondly, it constructs three convolutional attention modules with identical architectures; features of different scales and gradients are fused using a gating method; then, the fused features are fed into an embryonic structure-wide decoding network to obtain multi-focal-length fused embryonic images; finally, it establishes self-supervised gradient feature alignment loss and unsupervised structural similarity loss functions to optimize the overall network architecture and achieve the fusion of focal-length embryonic images; as shown below. Figure 1 As shown, specifically, the method is carried out in the following steps:

[0062] Step 1: In this example, tens of thousands of embryo images from multiple reproductive centers were collected. This data covers the entire developmental process of an embryo, from fertilized egg to late blastocyst ready for transfer. Embryo images were acquired using a built-in camera in a time-lapse incubator. A set of images was captured consecutively within a very short timeframe, acquiring two embryo images with different focal lengths. This resulted in a set of two embryo microscopic images taken simultaneously at different focal lengths. ,in, This represents the first microscopic image of the embryo. This indicates the second embryo micrograph, where 512 represents the length of the embryo micrograph, 612 represents the width of the embryo micrograph, and 3 represents the number of channels in the embryo micrograph.

[0063] right Grayscale processing was performed to obtain a set of microscopic grayscale images of the embryo. ,in, This represents the first grayscale image of the embryo. This represents the second grayscale image of the embryo; 1 represents the number of channels in the grayscale image of the embryo.

[0064] Based on the channel dimension of the grayscale image and Stacking them together forms a multi-channel input feature map. .

[0065] Three different gradient operators (Canny, Tenengrad, and Laplacian) are used to extract... Structural gradient plots , among which, among which express and Feature map after processing with the Canny gradient operator express and Feature map after processing with the Tenengrad gradient operator express and Feature maps processed by the Laplacian gradient operator; then structural gradient maps. By performing pixel-wise maximum value operations on maps with the same gradient operator, three unified embryo reference gradient maps are obtained. The embryo image data used in this example comes from tens of thousands of images from multiple reproductive centers. This data covers the entire developmental process of an embryo, from fertilized egg to late blastocyst ready for transfer. The embryo images were acquired using a built-in camera in a time-lapse incubator. A set of images was captured in rapid succession within a very short period of time, acquiring two embryo images with different focal lengths.

[0066] Step 2: In this invention, to improve the perception of information at different scales and enhance the performance of fused embryo images in terms of structural integrity and local sharpness, a three-branch parallel gradient feature extraction module is designed, introducing three gradient algorithms with complementary characteristics: Canny, Tenengrad, and Laplacian. Addressing the practical need for significant scale differences in structural features and focus areas presented by embryo images at different developmental stages, a multi-gradient collaborative fusion network is constructed, including: a multi-scale feature extraction module guided by multiple gradient features, a feature fusion module, and... Processing was performed to obtain multi-scale embryonic gradient fusion features. Network architecture such as Figure 1 As shown:

[0067] Step 2.1: Due to the practical need for different structural scales of focus in embryonic images at different developmental stages, a unified multi-scale feature extraction module is constructed to address the characteristics of embryonic images at different developmental stages. This module includes three parallel branches, each containing three convolutional attention modules at different scales, guided by the classic gradient algorithms Canny, Tenengrad, and Laplacian. Each scale's convolutional attention module includes a convolutional layer, a normalization layer, an activation function layer, and a cross-channel attention mechanism layer. Appropriate padding parameters are set for the convolutional layers to ensure that the image spatial dimensions remain unchanged after the convolution operation. First, the multi-channel input feature map... Using the first convolutional block, A convolutional kernel with padding set to 1 is used to maintain the spatial size, increasing the number of channels from 2 to 16. Batch normalization is then applied to stabilize the feature distribution, followed by ReLU activation to enhance non-linear expressiveness. The output feature size is... Next, this feature map is used as input to the second convolutional block, and the convolutional layer uses... The convolution kernel has a padding of 2, and the number of channels is further increased to 32. It also undergoes normalization and ReLU activation processing, resulting in an output size of [size missing]. Then we move on to the third convolutional block, using... The convolution kernel is padded to 3, the number of channels is expanded to 64, and normalization and activation operations are repeated to obtain the final output. The model employs a multi-scale gradient enhancement feature map. This structure allows for the progressive expansion of the receptive field and the gradual enhancement of channel information while maintaining the same spatial size. This effectively fuses multi-level gradient information from local edges, detailed textures, and global structure in embryonic images, providing richer and more stable feature support for subsequent fusion processing. The cross-channel attention mechanism layer includes: pooling layers, fully connected layers, activation function layers, and channel attention weighted layers. The overall structure is as follows: Figure 2 As shown. First, the pooling layer uses global average pooling, compressing the spatial dimension of each channel into a scalar, thus forming channel-level global statistical features. The features after pooling are input to the first fully connected layer, reducing the number of feature channels to 1 / 8 of the original. Next, a ReLU activation function is used to introduce a non-linear mapping, enhancing the discriminative power of the channel features. Then, a second fully connected layer restores the number of channels to the original input channels. Finally, a Sigmoid activation function is used for normalization, obtaining the attention weights corresponding to each channel. The input is processed through three parallel branches and then sequentially through three convolutional attention modules of different scales, resulting in the intermediate gradients of the first branch output at three different scales. The intermediate gradients at three different scales output by the second branch And the intermediate gradients at three different scales of the third branch output. .

[0068] Step 2.2: Construct a feature fusion network, including: a splicing unit, a gating network, and a gradient fusion unit; wherein, the gating network includes: two convolutional layers, an activation function layer, and a normalization processing unit; as shown below. Figure 3 As shown, convolutional block 1 and convolutional block 2 in the gated network have the same network architecture. Convolutional block 1 uses a convolutional kernel size of [missing information]. The activation function is ReLU; convolution block 2 uses a kernel size of [missing value]. The activation function is Softmax; the output dimension of the gated network is consistent with the spatial size of the input embryo feature map. The three generated weight maps will be used as channel weighting factors to apply to the corresponding intermediate gradient features, realizing the weighted fusion of features from different branches in the spatial dimension. This enhances the structural response of the effective region, suppresses redundant or interfering information, and provides a stable and reliable weight basis for the subsequent generation of fused embryo images. Furthermore, the intermediate gradient feature sequences output by all branches are processed. Processing to obtain embryo fusion feature maps .

[0069] Step 2.2.1: To achieve a unified representation of feature information from different gradient branches at the same scale, the splicing unit uses equation (1) to splice the intermediate gradient features with the same scale in the three branches by channel, resulting in spliced ​​features at three different scales. :

[0070] (1)

[0071] In equation (1), These represent the splicing features at three different scales; This indicates a splicing operation.

[0072] Step 2.2.2: After concatenating the intermediate gradient features of the three different branches by channel, in order to further achieve adaptive control over the contribution of each branch feature, the concatenated features at the three scales are... The input is processed in a gating network, and features at each scale obtain three weight maps. ,in Indicates the first Three weights at each scale, thus obtaining weight maps at three different scales. ,like Figure 3 As shown.

[0073] Step 2.2.3: The gradient fusion unit uses equation (2) to obtain the fused gradient features at three different scales. :

[0074] (2)

[0075] In equation (2), This represents element-wise multiplication; These represent the fusion features at three scales respectively; this weighting process enables dynamic allocation of importance for different regions in the image, so that the branch features with stronger responses at structural boundaries, detailed regions or clear focal points receive greater retention weights, while the corresponding feature channel contributions are automatically weakened in redundant or blurred regions.

[0076] Step 2.2.4: Use equation (3) to fuse gradient features Perform upsampling and channel-wise stitching to obtain stitching features. :

[0077] (3)

[0078] In equation (3), , Indicates to Perform an upsampling operation to make the size and Alignment and upsampling aim to achieve channel alignment between features at different scales.

[0079] Step 2.2.5, will The input is processed in a gating network to obtain three weight graphs. Then, the gradient fusion unit is used to obtain the embryo fusion feature map using equation (4). :

[0080] (4)

[0081] Step 3: Construct a global decoding module for embryonic structure, including: a patch embedding layer, a rotation-invariant position encoding layer, a multi-layer global decoding unit, and a patch restoration layer; the multi-layer global decoding unit consists of multiple Transformer decoding layers with identical structures, each Transformer decoding layer including a multi-head attention unit, a feedforward neural network, normalization and residual connection units; and perform... Processing is performed to obtain a focused embryo image. .

[0082] Step 3.1: Obtain the embryo fusion feature map The input is divided into a set of two-dimensional image blocks in the Patch embedding layer, and each two-dimensional image block is mapped to a token vector of uniform length.

[0083] Step 3.2: The rotation-invariant position encoding layer uses two-dimensional polar coordinate position encoding to add a learnable rotation-invariant position vector to each token vector, and then fuses it with the corresponding token vector to obtain different global position-aware token vectors. Polar coordinate encoding significantly improves the model's tolerance to random embryo orientation (such as culture dish placement deviation ±90°), avoiding structural misalignment caused by traditional Cartesian coordinates. Simultaneously, the rotation-invariant position encoding vector can be initialized with learnable parameters and optimized along with the network during training, thereby achieving long-term memory and adaptation of the relative and absolute position information of embryo image blocks.

[0084] Step 3.3: Input each global location-aware token vector into the multi-layer global decoding unit for decoding processing to obtain the corresponding global location-aware token decoding vector;

[0085] Step 3.4: The Patch reconstruction layer uses linear projection to reconstruct each global location-aware token decoding vector into its corresponding image patch vector, thereby... The original spatial location of the two-dimensional image patch is used to reconstruct the image patch vector, resulting in a reconstructed focused embryo image. Through the process of restoring and reconstructing two-dimensional embryonic image blocks, the semantic features output by the decoding network are successfully restored into an embryonic image representation with real spatial structure and focus details, realizing end-to-end reconstruction from multi-focal-length embryonic image fusion features to a single clear embryonic image.

[0086] Step 4: Construct the joint loss function :

[0087] Step 4.1: In order to guide multiple independent gradient branches in the fusion network to learn their corresponding features, achieve gradient semantic decoupling, and enable the fusion process to have clear feature sources and interpretability, a self-supervised gradient feature alignment loss is established using equation (5):

[0088] (5)

[0089] In equation (5), Representative embryo reference gradient map The normalized embryo reference gradient map corresponds to the feature maps processed by the Canny, Tenengrad and Laplacian gradient operators in the embryo image, respectively. Represents the third scale of the three branches The intermediate gradient features after normalization correspond to the Canny, Tenengrad, and Laplacian gradient features learned by the three gradient branches in the embryo image, respectively. The L1 norm is used to encourage the model to fit pixel by pixel, making it easy to train and converge stably. Normalization maintains the consistency of numerical scale. , and This represents the self-supervised gradient feature alignment loss in the three branches, used to balance the influence weights of the feature alignment losses in different branches.

[0090] Step 4.2: To evaluate the focused embryo image reconstructed by the model. With multifocal embryo images To ensure structural consistency, the output image of the fusion network decoder retains information about the original embryonic image in terms of spatial structure and texture details. Under unsupervised conditions, the final reconstructed embryonic image is optimized by introducing a structural similarity loss function. This loss function uses each input image in the multifocal embryonic image as a reference, comparing the fidelity of the final fused image in terms of local structure, brightness, and contrast. The aim is to constrain the image reconstruction effect from a perceptual perspective. The structural similarity loss function is established using equation (6). :

[0091] (6)

[0092] In equation (6), Represents the structural similarity index; The smaller the loss function value, the closer the fused embryo image is to the structure of the original image at the structural level. By minimizing this loss during training, the ability of the fused embryo image to preserve the original structural information can be effectively enhanced, improving its sharpness, edge fidelity, and visual consistency, thereby improving the overall quality of embryo image fusion.

[0093] Step 4.3: Construct the joint loss function using equation (7). :

[0094] (7)

[0095] In equation (7), , and This represents the regularization parameter corresponding to the gradient feature alignment loss; and The regularization parameter represents the gradient feature alignment loss and structural similarity loss function in network optimization. It is used to balance the optimization objectives between self-supervised embryo image structure fidelity and local branch feature guidance, so that the network can learn multiple gradient features of embryo images while maintaining structural consistency.

[0096] Step 5: In the training phase of this invention, the preprocessed embryonic image data is input into the multifocal embryonic image fusion network for training. The network achieves joint perception and feature integration of structural information and clear regions in multifocal embryonic images by introducing a gradient-guided feature extraction module and a cross-channel attention mechanism. During training, the AdamW optimizer is used to train the multifocal embryonic image fusion network and minimize the joint loss function. The network parameters are updated to obtain the optimal multifocal embryo image fusion model, which is used to achieve adaptive focusing fusion of multifocal embryo images.

[0097] In this embodiment, an electronic device includes a memory and a processor. The memory stores a program that supports the processor in executing the above-described method, and the processor is configured to execute the program stored in the memory.

[0098] In this embodiment, a computer-readable storage medium stores a computer program, which is executed by a processor to perform the steps of the above method.

[0099] In practice, the network model under the joint optimization framework uses the results of the test set as the evaluation metric. Experiments demonstrate the effectiveness of the regularization parameter for the hybrid loss. , The optimal effect can be achieved at this time. Furthermore, this invention conducted extensive comparative experiments on 50 sets of embryo image datasets to verify the effectiveness of the proposed method. The experimental results are shown in Table 1:

[0100] Table 1. Experimental results of different models on embryo image datasets

[0101]

[0102] Table 1 shows the experimental results, indicating that the proposed multi-focal-length embryo image fusion method based on multi-gradient collaborative fusion and structure-wide decoding performs well in six mainstream image fusion evaluation metrics: normalized mutual information. Human visual perception indicators Gradient fidelity Phase consistency characteristic index Structural similarity index and visual information fidelity All of these methods outperform other methods (1. the CNN method in the paper "Medical image fusion based on convolutional neural networks and non-subsampled contourlet transform", 2. the U-Net method in the paper "Fast multi-focus fusion based on deep learning for early-stage embryo image enhancement", 3. the Transformer method in the paper "Image fusion transformer", and 4. the U2Fusion method in the paper "U2Fusion: Aunified unsupervised image fusion network"). The metric reached 0.7244, an improvement of approximately 16% compared to the best comparison method, U2Fusion. For example... Figure 4 The image shown is a visualization of the results of image fusion using different methods. The image fusion method proposed in this invention can achieve the optimal result, demonstrating the significant advantages of the proposed method (Ours) in terms of structure restoration and sharpness enhancement.

[0103] To further verify the effectiveness of key components in the model structure, this invention designed a series of ablation experiments, focusing on evaluating the contributions of three gradient feature-guided branches (Canny, Tenengrad, and Laplacian) to the overall fusion performance. The results show that retaining only a single gradient branch generally reduces the fusion effect, especially when using only the Laplacian branch, where all metrics are significantly worse than the complete model. However, by simultaneously introducing three gradient information and performing collaborative fusion through a gating mechanism, the model achieves optimal results across all metrics. This fully demonstrates that the three gradient operators are complementary in capturing different structural features (such as edges, details, and gray-scale abrupt changes), and the collaborative fusion mechanism can significantly improve the overall focusing quality and interpretation value of multifocal embryo images.

Claims

1. A multi-focal embryo image fusion method based on multi-gradient collaborative fusion and structure global decoding, characterized in that, is performed as follows: Step 1, obtaining a set of two different focal length embryo microscopic images taken at the same time wherein, represents the first embryo microscopic image, represents the second embryo microscopic image, represents the length of the embryo microscopic image, represents the width of the embryo microscopic image; 3 represents the channel number of the embryo microscopic image; To The embryo microscopic gray image group is obtained by performing gray processing on the embryo microscopic image group , wherein represents the first embryo microscopic gray image, represents the second embryo microscopic gray image; 1 represents the channel number of the embryo microscopic gray image; stacking along the channel dimension of the grayscale image and stacking to form a multi-channel input feature map ; adopt a different gradient operator respectively extracts a structure gradient group wherein, represents a structure gradient group extracted by a first gradient operator, represents a structure gradient group extracted by a second gradient operator, represents a structure gradient extracted by a first gradient operator, represents a structure gradient extracted by a second gradient operator, represents a structure gradient extracted by a third gradient operator, represents a structure gradient extracted by a fourth gradient operator, represents a structure gradient extracted by a fifth gradient operator, represents a structure gradient extracted by a sixth gradient operator; right and Perform a pixel-by-pixel maximum value operation to obtain the first pixel. Zhang embryo reference gradient map Thus obtain Zhang embryo reference gradient map ; Step 2, constructing a multi-gradient collaborative fusion network, including: a plurality of gradient feature guided multi-scale feature extraction modules, a feature fusion module, and processing the obtained multi-scale embryo gradient fusion features . ​ Step 3, constructing the embryo structure global decoding module, including: Patch embedding layer, rotation invariant position encoding layer, multi-layer global decoding unit and Patch restoration layer; wherein the multi-layer global decoding unit is composed of multiple layers of the same structure of the Transformer decoding layer, and each Transformer decoding layer includes: multi-head attention unit, feedforward neural network, normalization and residual connection unit; and processing to obtain a focused embryo image ; Step 4, Constructing the joint loss function ; Step 5, train the multi-focus embryo image fusion network using the AdamW optimizer and minimize the joint loss function Update the network parameters to obtain an optimal multi-focus embryo image fusion model for realizing adaptive focus fusion of multi-focus embryo images.

2. The multi-focus embryo image fusion method based on multi-gradient collaborative fusion and structural global decoding according to claim 1, characterized in that, Step 2 is performed as follows: Step 2.1, constructing a multi-scale feature extraction module, comprising: a plurality of parallel branches, each branch comprising a convolutional attention module of a different scale; wherein each scale of the convolutional attention module comprises: a convolutional layer, a normalization layer, an activation function layer, and a cross-channel attention mechanism layer; the cross-channel attention mechanism layer comprises: a pooling layer, a fully connected layer, an activation function layer, and a channel attention weighting layer; and inputting the first parallel branch, sequentially passing through a convolutional attention module of a different scale, thereby obtaining an intermediate gradient feature sequence output by the first branch , wherein represents the intermediate gradient feature of the kth scale output by the first branch, , ; Step 2.2, constructing a feature fusion network, comprising: a splicing unit, a gating network and a gradient fusion unit; wherein the gating network comprises: two convolutional layers, an activation function layer and a normalization processing unit; and the intermediate gradient feature sequence of all branch outputs processing, obtaining an embryo fusion feature map .

3. The multi-focal embryo image fusion method based on multi-gradient collaborative fusion and structural global decoding according to claim 2, characterized in that, Step 2.2 is performed as follows: Step 2.2.1, the concatenation unit concatenates the intermediate gradient features with the same scale in the branch by channel using formula (1) obtains the concatenated features under different scales :​ (1) In formula (1), denotes a stitching feature at a first scale; denotes a stitching operation; Step 2.2.2, will The input is processed in a gating network to obtain the first... Under each scale Weighted graph Thus obtain Weighting maps at different scales ,in, Indicates the first The first scale Each weight; Step 2.2.3, the gradient fusion unit utilizes formula (2) to obtain fusion gradient features under different scales : (2) In formula (2), represents element-wise multiplication; represents the fusion gradient feature at the th scale. Step 2.2.4, utilizing the fusion gradient feature of formula (3) performing an up-sampling operation and channel-wise concatenation to obtain a concatenated feature : (3) In formula (3), indicates that an up-sampling operation is performed so that its size is aligned with . Step 2.2.5, obtaining input into the gating network for processing to obtain a weight map The gradient fusion unit obtains the embryo fusion feature map using formula (4) : (4) In formula (4), denotes the jth weight map.

4. The multi-focus embryo image fusion method based on multi-gradient collaborative fusion and structural global decoding according to claim 3, characterized in that, Step 3 is performed as follows: Step 3.1, fusing embryo feature maps Input to the Patch Embedding layer is divided into a set of two-dimensional image patches, each of which is mapped to a Token vector of uniform length; Step 3.2, the rotation invariant position encoding layer adds a rotation invariant position vector to be learned to each Token vector in a two-dimensional polar coordinate position encoding manner, and is fused with the corresponding Token vector, so as to obtain different global position-aware Token vectors; Step 3.3, each global position-aware Token vector is input into a multi-layer global decoding unit for decoding processing, so as to obtain a corresponding global position-aware Token decoding vector; Step 3.4, the patch restoration layer restores each global position-aware token decoding vector to a corresponding image patch vector respectively by linear projection, and then reorganizes the image patch vectors according to the original spatial positions of the two-dimensional image patches to obtain a reconstructed focused embryo image .​ 5. The multi-focal embryo image fusion method based on multi-gradient collaborative fusion and structural global decoding according to claim 4, characterized in that, Step 4 is performed as follows: Step 4.1, a self-supervised gradient feature alignment loss is established by using formula (5): (5) In equation (5), Representative embryo reference gradient map Embryo reference gradient map after normalization. Indicates the first Zhang's reference gradient diagram of embryos after normalization; represent The output of the branch intermediate gradient feature sequences at different scales The intermediate gradient features after normalization. Indicates the first The output of the branch Intermediate gradient features after normalization at each scale; represent Self-supervised gradient feature alignment loss for each branch Indicates the first Self-supervised gradient feature alignment loss for each branch; Step 4.2, Establishing a structural similarity loss function with formula (6) : (6) In formula (6), represents a structure similarity index; Step 4.3, constructing a joint loss function with formula (7) : (7) In formula (7), represent corresponding regularization parameters; and denote 2 regularization parameters.

6. An electronic device comprising a memory and a processor, characterized in that The memory is configured to store a program supporting the processor to execute the multi-focal-length embryo image fusion method of any one of claims 1-5, and the processor is configured to execute the program stored in the memory.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to perform the steps of the multi-focal-length embryo image fusion method of any one of claims 1-5.

Images