Embryo image fusion methods, devices, electronic equipment and storage media

Abstract

This application provides an embryo image fusion method, apparatus, electronic device, and storage medium, relating to the field of deep learning technology. This method, tailored to the characteristics of embryos at different developmental stages, selects focal plane images corresponding to each developmental stage from multiple original focal plane images as the images to be processed. This avoids using all original focal plane images as the images to be processed for image fusion, thereby reducing data processing volume, improving image fusion efficiency, and balancing the fusion time. Furthermore, based on the feature analysis requirements of different developmental stages, it employs image fusion strategies corresponding to each developmental stage to fuse the images to be processed, achieving personalized image fusion processing. This ensures that the resulting fused image contains as many embryonic features as possible, thereby improving the accuracy of subsequent embryo recognition based on the fused image.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, and more specifically, to an embryo image fusion method, apparatus, electronic device, and storage medium. Background Technology

[0002] In embryo time-lapse culture systems, most imaging systems use Hoffman modulation contrast (HMC) microscopy to acquire multi-focal-plane embryo images, typically obtaining 7-11 focal-plane image data.

[0003] Current embryo image observation and analysis mainly rely on images from the F0 focal plane, while the feature information of other focal planes has not been used properly. The F0 focal plane can be understood as the best clear layer close to the embryo's equatorial plane.

[0004] Therefore, the question of how to utilize the rich feature information in the multifocal plane of the embryo in the selection and evaluation of embryos urgently needs to be addressed. Summary of the Invention

[0005] The purpose of this application is to address the shortcomings of the prior art by providing an embryo image fusion method, apparatus, electronic device, and storage medium to improve the accuracy of image fusion and enable the fused images to be better used for image analysis.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows:

[0007] In a first aspect, embodiments of this application provide an embryo image fusion method, including:

[0008] Based on multiple raw focal plane images of the acquired embryo, the target developmental stage of the embryo is predicted;

[0009] Based on the target developmental stage, determine each plane image to be processed corresponding to the target developmental stage from the plurality of original focal plane images;

[0010] The target image fusion strategy corresponding to the target developmental stage is used to fuse the plane images to be processed to generate the fused image corresponding to the embryo. The target image fusion strategy includes: an image channel fusion strategy or a fusion strategy based on an image fusion model. The image fusion model is pre-trained based on multiple sets of training samples composed of original focal plane images of the embryo.

[0011] Optionally, determining the plane image to be processed corresponding to the target developmental stage from the plurality of original focal plane images based on the target developmental stage includes:

[0012] Based on the target developmental stage and the pre-created correspondence between the developmental stage and the planar image to be processed, each planar image to be processed corresponding to the target developmental stage is determined from the plurality of original focal plane images.

[0013] Optionally, the developmental stages of the embryo include: pronuclear stage, cleavage stage I, cleavage stage II, fusion stage, and blastocyst stage; the plurality of original focal plane images respectively include: F-45 focal plane image, F-30 focal plane image, F-15 focal plane image, F0 focal plane image, F15 focal plane image, F30 focal plane image, and F45 focal plane image;

[0014] The step of determining, based on the target developmental stage, each to-be-processed planar image corresponding to the target developmental stage from the plurality of original focal plane images includes:

[0015] If the target developmental stage is the pronuclear stage, then the F0 focal plane image is determined as the plane image to be processed from the plurality of original focal plane images;

[0016] If the target developmental stage is the first cleavage stage, then the F-30 focal plane image, the F0 focal plane image, and the F30 focal plane image are determined from the plurality of original focal plane images as the plane image to be processed;

[0017] If the target developmental stage is the second cleavage stage, the fusion stage, or the blastocyst stage, then the plurality of original focal plane images are determined as the plane images to be processed.

[0018] Optionally, the step of fusing the planar images to be processed using the target image fusion strategy corresponding to the target developmental stage to generate the fused image corresponding to the embryo includes:

[0019] If the target developmental stage is the first cleavage stage, then the image channel fusion strategy is used to fuse the plane images to be processed to generate the fused image corresponding to the embryo.

[0020] If the target developmental stage is the second cleavage stage, the fusion stage, or the blastocyst stage, then the image fusion model is used to fuse the planar images to be processed, generating the fused image corresponding to the embryo.

[0021] Optionally, the step of employing an image channel fusion strategy to fuse the various planar images to be processed, generating a fused image corresponding to the embryo, includes:

[0022] A channel fusion algorithm is used to merge the channels of each plane image to be processed, generating a target RGB three-channel image.

[0023] The target RGB three-channel image is used as the fused image corresponding to the embryo.

[0024] Optionally, before fusing the various planar images to be processed using the image fusion model to generate the fused image corresponding to the embryo, the following steps are included:

[0025] A channel fusion algorithm is used to merge the channels of each plane image to be processed, generating an RGB three-channel image corresponding to each plane image to be processed.

[0026] The RGB three-channel images corresponding to each plane image to be processed are subjected to color model conversion processing to generate the YCrCb image corresponding to each plane image to be processed.

[0027] The step of fusing the image fusion model to generate the fused image corresponding to the embryo includes:

[0028] The YCrCb images corresponding to each planar image to be processed are input into the image fusion model for image fusion processing to obtain the fused image corresponding to the embryo.

[0029] Optionally, the image fusion model includes a feature extraction module, which includes a first feature extraction layer, a second feature extraction layer, and a third feature extraction layer.

[0030] The step of inputting the YCrCb images corresponding to each planar image to be processed into the image fusion model for image fusion processing to obtain the fused image corresponding to the embryo includes:

[0031] The YCrCb images corresponding to each planar image to be processed are input into the feature extraction module, and convolution and pooling are performed through the first feature extraction layer and the second feature extraction layer to obtain the first shallow feature information and the second shallow feature information.

[0032] After inputting the first and second shallow feature information into the third feature extraction layer for convolution processing, the third, fourth, and fifth shallow feature information are obtained respectively.

[0033] The third, fourth, and fifth shallow feature information are processed by full connection respectively to output the target third shallow feature information, the target fourth shallow feature information, and the target fifth shallow feature information.

[0034] The first shallow feature information, the second shallow feature information, the third shallow feature information of the target, the fourth shallow feature information of the target, and the fifth shallow feature information of the target are used as the feature information output by the feature extraction module for information measurement.

[0035] Optionally, before fusing the planar images to be processed using the target image fusion strategy corresponding to the target developmental stage to generate the fused image corresponding to the embryo, the following steps are included:

[0036] Image contour compensation is performed on the target images in each plane image to be processed to obtain the compensated target image; the target image includes: each plane image to be processed with an F0 focal length or above;

[0037] The contrast of the compensated target image is enhanced to obtain the enhanced target image;

[0038] The enhanced target image is subjected to noise filtering to generate a denoised target image.

[0039] The denoised target image is then subjected to image enhancement processing to generate the processed image corresponding to each plane image to be processed.

[0040] Optionally, the image fusion model has multiple input data channels; the image fusion model is trained using the following method:

[0041] Multiple sets of training sample images are collected, and each set of training sample images includes multiple original focal plane images corresponding to one embryo.

[0042] Each group of training sample images is preprocessed to obtain the processed training sample images.

[0043] Each group of training sample images is processed by channel fusion to generate target training sample images for each group.

[0044] Each group of target training sample images is input into the initial image fusion model for model training, thereby generating the image fusion model.

[0045] Optionally, the step of inputting each group of target training sample images into the initial image fusion model for model training to generate the image fusion model includes:

[0046] The initial image fusion model is used to extract features from each group of target training sample images to obtain multiple feature information of each original focal plane image in each group of target training sample images;

[0047] Information is measured based on multiple feature information of each original focal plane image in each group of target training sample images, and a measurement result is generated. The measurement result is used to characterize the degree of feature information extracted from each original focal plane image.

[0048] The fusion weights of the image are generated based on the measurement results;

[0049] The loss of the initial image fusion model is calculated based on the fusion weights of each original focal plane image in each group of target training sample images.

[0050] Based on the loss, the initial image fusion model is iteratively optimized to generate the image fusion model.

[0051] Optionally, the method further includes:

[0052] Based on the original F0 focal plane image of the target embryo and the corresponding fused image of the target embryo, at least one performance index algorithm is used to calculate and obtain the performance index of the image fusion model; the performance index is used to evaluate the fusion performance of the image fusion model; the performance index algorithm includes: gradient-based fusion performance algorithm, image mean square error algorithm, structural similarity algorithm and peak signal-to-noise ratio algorithm.

[0053] Secondly, embodiments of this application also provide an embryo image fusion device, including: a prediction module, a determination module, and a processing module;

[0054] The prediction module is used to predict the target developmental stage of the embryo based on multiple original focal plane images of the acquired embryo.

[0055] The determining module is used to determine, from the plurality of original focal plane images, each plane image to be processed corresponding to the target developmental stage, based on the target developmental stage.

[0056] The processing module is used to perform fusion processing on each plane image to be processed using the target image fusion strategy corresponding to the target developmental stage to generate a fused image corresponding to the embryo. The target image fusion strategy includes: an image channel fusion strategy or a fusion strategy based on an image fusion model. The image fusion model is pre-trained based on multiple sets of training samples composed of original focal plane images of the embryo.

[0057] Optionally, the determining module is specifically used to determine, from the plurality of original focal plane images, each plane image to be processed corresponding to the target developmental stage, based on the target developmental stage.

[0058] Optionally, the developmental stages of the embryo include: pronuclear stage, cleavage stage I, cleavage stage II, fusion stage, and blastocyst stage; the plurality of original focal plane images respectively include: F-45 focal plane image, F-30 focal plane image, F-15 focal plane image, F0 focal plane image, F15 focal plane image, F30 focal plane image, and F45 focal plane image;

[0059] The determining module is specifically used for

[0060] If the target developmental stage is the pronuclear stage, then the F0 focal plane image is determined as the plane image to be processed from the plurality of original focal plane images;

[0061] If the target developmental stage is the first cleavage stage, then the F-30 focal plane image, the F0 focal plane image, and the F30 focal plane image are determined from the plurality of original focal plane images as the plane image to be processed;

[0062] If the target developmental stage is the second cleavage stage, the fusion stage, or the blastocyst stage, then the plurality of original focal plane images are determined as the plane images to be processed.

[0063] Optionally, the processing module is specifically used for

[0064] If the target developmental stage is the first cleavage stage, then the image channel fusion strategy is used to fuse the plane images to be processed to generate the fused image corresponding to the embryo.

[0065] If the target developmental stage is the second cleavage stage, the fusion stage, or the blastocyst stage, then the image fusion model is used to fuse the planar images to be processed, generating the fused image corresponding to the embryo.

[0066] Optionally, the processing module is specifically used for

[0067] A channel fusion algorithm is used to merge the channels of each plane image to be processed, generating a target RGB three-channel image.

[0068] The target RGB three-channel image is used as the fused image corresponding to the embryo.

[0069] Optionally, the device further includes: a channel fusion module;

[0070] The channel fusion module is used to merge the channels of each plane image to be processed using a channel fusion algorithm to generate an RGB three-channel image corresponding to each plane image to be processed.

[0071] The RGB three-channel images corresponding to each plane image to be processed are subjected to color model conversion processing to generate the YCrCb image corresponding to each plane image to be processed.

[0072] The processing module is specifically used to input the YCrCb images corresponding to each planar image to be processed into the image fusion model for image fusion processing to obtain the fused image corresponding to the embryo.

[0073] Optionally, the image fusion model includes a feature extraction module, which includes a first feature extraction layer, a second feature extraction layer, and a third feature extraction layer.

[0074] The processing module is specifically used to input the YCrCb image corresponding to each planar image to be processed into the feature extraction module, and perform convolution and pooling processing through the first feature extraction layer and the second feature extraction layer to obtain the first shallow feature information and the second shallow feature information.

[0075] After inputting the first and second shallow feature information into the third feature extraction layer for convolution processing, the third, fourth, and fifth shallow feature information are obtained respectively.

[0076] The third, fourth, and fifth shallow feature information are processed by full connection respectively to output the target third shallow feature information, the target fourth shallow feature information, and the target fifth shallow feature information.

[0077] The first shallow feature information, the second shallow feature information, the third shallow feature information of the target, the fourth shallow feature information of the target, and the fifth shallow feature information of the target are used as the feature information output by the feature extraction module for information measurement.

[0078] Optionally, the apparatus further includes: a preprocessing module;

[0079] The preprocessing module is used to perform image contour compensation on the target images in each planar image to be processed, so as to obtain the compensated target image; the target image includes: each planar image to be processed with an F0 focal length or above;

[0080] The contrast of the compensated target image is enhanced to obtain the enhanced target image;

[0081] The enhanced target image is subjected to noise filtering to generate a denoised target image.

[0082] The denoised target image is then subjected to image enhancement processing to generate the processed image corresponding to each plane image to be processed.

[0083] Optionally, the image fusion model may have multiple input data channels; the device may also include a training module.

[0084] The training module is used for

[0085] Multiple sets of training sample images are collected, and each set of training sample images includes multiple original focal plane images corresponding to one embryo.

[0086] Each group of training sample images is preprocessed to obtain the processed training sample images.

[0087] Each group of training sample images is processed by channel fusion to generate target training sample images for each group.

[0088] Each group of target training sample images is input into the initial image fusion model for model training, thereby generating the image fusion model.

[0089] Optionally, the training module is specifically used for

[0090] The initial image fusion model is used to extract features from each group of target training sample images to obtain multiple feature information of each original focal plane image in each group of target training sample images;

[0091] Information is measured based on multiple feature information of each original focal plane image in each group of target training sample images, and a measurement result is generated. The measurement result is used to characterize the degree of feature information extracted from each original focal plane image.

[0092] The fusion weights of the image are generated based on the measurement results;

[0093] The loss of the initial image fusion model is calculated based on the fusion weights of each original focal plane image in each group of target training sample images.

[0094] Based on the loss, the initial image fusion model is iteratively optimized to generate the image fusion model.

[0095] Optionally, the apparatus further includes: an evaluation module;

[0096] The evaluation module is used to calculate and obtain the performance index of the image fusion model based on the original F0 focal plane image of the target embryo and the corresponding fused image of the target embryo, using at least one performance index algorithm; the performance index is used to evaluate the fusion performance of the image fusion model; the performance index algorithm includes: gradient-based fusion performance algorithm, image mean square error algorithm, structural similarity algorithm and peak signal-to-noise ratio algorithm.

[0097] Thirdly, embodiments of this application provide an electronic device, including: a processor, a storage medium, and a bus. The storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to implement the embryo image fusion method provided in the first aspect.

[0098] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the embryo image fusion method as provided in the first aspect.

[0099] The beneficial effects of this application are:

[0100] This application provides an embryo image fusion method, apparatus, electronic device, and storage medium. The method uses an embryonic developmental stage prediction network to classify and predict multiple input raw focal plane images to determine the developmental stage of the embryo at various points. Based on the developmental stage, focal plane images corresponding to the developmental stage are selected from the multiple raw focal plane images as the images to be processed. An image fusion strategy corresponding to the developmental stage is then applied to fuse these images, ultimately obtaining a fused image. This method, considering the characteristics of embryos at different developmental stages, selects focal plane images corresponding to the developmental stage from multiple raw focal plane images as the images to be processed, eliminating the need to use all raw focal plane images as the images to be processed for image fusion. This reduces the amount of data processing, improves image fusion efficiency, and balances the image fusion time. Furthermore, based on the feature analysis requirements of different developmental stages, an image fusion strategy corresponding to the developmental stage is used to fuse the images to be processed, achieving personalized image fusion processing. This also ensures that the resulting fused image contains as many embryonic features as possible, thereby improving the accuracy of subsequent embryo identification based on the fused image.

[0101] Secondly, this application improves the input layer and feature extraction layer based on the existing fusion network. On the one hand, it enables the trained image fusion model to be applicable to the processing of multiple inputs. On the other hand, the extracted features are richer, making the image fusion result more ideal. Attached Figure Description

[0102] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0103] Figure 1 A schematic diagram of the flowchart of an embryo image fusion method provided in an embodiment of this application;

[0104] Figure 2 A flowchart illustrating the embryo image fusion method provided in this application embodiment. Figure 1 ;

[0105] Figure 3 A timeline of embryonic development stages provided in this application embodiment;

[0106] Figure 4 A flowchart illustrating the embryo image fusion method provided in this application embodiment. Figure 2 ;

[0107] Figure 5 A flowchart illustrating the embryo image fusion method provided in this application embodiment. Figure 3 ;

[0108] Figure 6 A schematic diagram of a feature extraction module provided in an embodiment of this application;

[0109] Figure 7 A flowchart illustrating the embryo image fusion method provided in this application embodiment. Figure 4 ;

[0110] Figure 8 A flowchart illustrating the embryo image fusion method provided in this application embodiment. Figure 5 ;

[0111] Figure 9 This is a schematic diagram illustrating an image preprocessing process provided in an embodiment of this application;

[0112] Figure 10 This is another schematic diagram of image preprocessing provided in an embodiment of this application;

[0113] Figure 11 This is a schematic diagram illustrating the image preprocessing result provided in an embodiment of this application.

[0114] Figure 12 A flowchart illustrating the embryo image fusion method provided in this application embodiment. Figure 6 ;

[0115] Figure 13 A flowchart illustrating the embryo image fusion method provided in this application embodiment. Figure 7 ;

[0116] Figure 14 This is a schematic diagram of an image fusion result provided in an embodiment of this application;

[0117] Figure 15 This is a schematic diagram of another image fusion result provided in an embodiment of this application;

[0118] Figure 16 This is a schematic diagram of an embryo image fusion device provided in an embodiment of this application;

[0119] Figure 17 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0120] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0121] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0122] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0123] First, a brief explanation of the relevant background technologies involved in this solution will be provided:

[0124] This approach is mainly used in the field of assisted reproduction. In embryo time-lapse culture systems, most imaging systems use Hoffman modulation contrast (HMC) microscopy to acquire multi-focal plane embryo images, typically obtaining 7-11 focal plane image data.

[0125] In current embryo image observation and analysis, the observation and analysis of embryos mainly rely on images from the F0 focal plane (which can be understood as the best-clear image close to the embryo's equatorial plane), while the feature information of other focal planes is not being used appropriately. The vast majority of use of multi-focal-plane data is based on direct observation and identification by the human eye, increasing the workload of manual image review.

[0126] Due to the unique characteristics of the Hohmann imaging system, local defocusing and overlapping artifacts between layers are present in multi-focal plane data. The initial purpose of acquiring multi-focal plane data is to enable embryologists to observe embryo growth at different depths more clearly and three-dimensionally and obtain more comprehensive embryo feature data. However, for an automated embryo analysis system, 11-layer focal plane data means a huge amount of computation. In order to make the most of all feature information from the acquired 7 or 11 focal plane images for embryo observation and analysis, this solution proposes to fuse the acquired focal plane images through an image fusion strategy to obtain a fused image containing more rich feature information. The fused image is then used for subsequent embryo identification.

[0127] Since embryos exhibit different morphological differences at different stages of development, corresponding to different feature analysis needs, this solution designs different image fusion strategies for embryo images at different developmental stages to perform image fusion operations according to feature analysis needs. This can balance the image fusion time and improve the accuracy of image fusion results.

[0128] Figure 1 This is a schematic diagram illustrating the flowchart of an embryo image fusion method provided in an embodiment of this application. Figure 1 As shown, the overall process mainly consists of the following steps:

[0129] First, multiple raw focal plane images of the embryo are input. In this embodiment, the raw focal plane images are F-45, F-30, F-15, F0, F15, F30, and F45, respectively, to illustrate the solution. However, it is worth noting that in practical applications, the raw focal plane images mentioned in this solution are not limited to these seven focal plane images, but can also be more focal plane images, such as F-75, F75, etc. This is because the imaging system of the timelapse incubator can usually acquire 11 focal plane data. Due to the characteristics of the imaging system, the further away from the 0 focal plane, the more obvious the defocusing phenomenon and the more blurred the image, especially the upper focal planes (F15 to F45), which are accompanied by artifact overlap in addition to defocusing. Therefore, the severely defocused (F75, F-75, F60, F-60) are removed, and the 7 focal planes close to the F0 focal plane are selected.

[0130] However, it is worth noting that with the development of imaging technology, in practical applications, the original focal plane image mentioned in this solution is not limited to the seven focal length images listed, but can also be more focal lengths, such as F-75 focal length, F75 focal length, etc.

[0131] A pre-trained embryonic development stage prediction network is used to predict multiple original focal plane images of the input embryo to determine the developmental stage at various points in the embryo. This scheme adopts an InceptionV3-based classification model framework, which divides the embryonic developmental stages into the pronuclear stage (the embryo has not yet divided), cleavage stage I (C2-C4), cleavage stage II (C5-C8), fusion stage, and blastocyst stage.

[0132] Depending on the developmental stage of the embryo, image fusion strategies corresponding to that developmental stage can be employed. For example... Figure 1 As shown, when the embryo is in the pronuclear stage, the embryo image is not fused and the F0 focal plane image can be directly used for feature extraction to analyze the embryo.

[0133] When the embryo is in the first cleavage stage, image fusion processing can be performed by selecting F-30, F0, and F30 focal plane images from the 7 focal plane images. At the first cleavage stage, image fusion processing can be performed based on an image channel fusion strategy, that is, directly performing channel fusion on the F-30, F0, and F30 focal plane images. The principle is to synthesize a single RGB three-channel image from three single-channel grayscale images by merging the channels.

[0134] When the embryo is in any of the following developmental stages: cleavage stage, fusion stage, or blastocyst stage, a pre-trained image fusion model can be used to perform image fusion processing on each original focal plane (7 focal plane images) to obtain a fused image.

[0135] The resulting fused images are then used for subsequent embryo feature extraction or blastomere segmentation for embryo observation and analysis.

[0136] Figure 2 A flowchart illustrating the embryo image fusion method provided in this application embodiment. Figure 1 The execution subject of this method can be a computer device or a server. For example... Figure 2 As shown, the method may include:

[0137] S201. Based on the multiple original focal plane images of the obtained embryo, predict the target developmental stage of the embryo.

[0138] Hoffman modulation contrast (HMC) microscopy can be used to image embryos in culture dishes, obtaining multiple raw focal plane images of the embryos. These raw focal plane images are then used as input data to an embryo developmental stage prediction network for classification and prediction, thereby determining the current target developmental stage of the embryo.

[0139] S202. Based on the target developmental stage, determine the plane images to be processed corresponding to the target developmental stage from multiple original focal plane images.

[0140] In this embodiment, for different target developmental stages, the corresponding planar images to be processed can be adaptively selected from multiple original focal plane images to perform subsequent image fusion operations. That is, it is not necessary to use all original focal plane images for fusion processing at each developmental stage. Based on some embryonic characteristics at each developmental stage, some focal plane images with fusion significance can be selected from multiple original focal plane images as planar images to be processed.

[0141] S203. The target image fusion strategy corresponding to the target developmental stage is used to fuse the plane images to be processed to generate the fused image corresponding to the embryo. The target image fusion strategy includes: image channel fusion strategy or fusion strategy based on image fusion model. The image fusion model is pre-trained based on multiple sets of training samples composed of original focal plane images of the embryo.

[0142] Since embryos exhibit different morphological differences at different developmental stages, corresponding to different feature analysis needs, this embodiment can also adopt the target fusion strategy corresponding to the developmental stage for image fusion according to the different developmental stages of the embryo. This can balance the image fusion time and ensure that the fused image contains as many embryonic features as possible, thereby improving the accuracy of subsequent embryo analysis.

[0143] In some embodiments, this solution can provide two image fusion strategies, namely, an image channel fusion strategy and an image fusion model-based fusion strategy. Based on the determined target developmental stage of the embryo, the image fusion strategy corresponding to the target developmental stage can be selected to perform the image fusion operation, thereby realizing personalized image fusion processing.

[0144] In summary, the embryo image fusion method provided in this embodiment classifies and predicts multiple input original focal plane images using an embryonic developmental stage prediction network to determine the developmental stage of the embryo at various points. Based on the developmental stage, focal plane images corresponding to the developmental stage are selected from the multiple original focal plane images as the plane images to be processed. An image fusion strategy corresponding to the developmental stage is then applied to fuse these plane images, ultimately resulting in a fused image. This method, considering the characteristics of embryos at different developmental stages, selects focal plane images corresponding to the developmental stage from multiple original focal plane images as the plane images to be processed, eliminating the need to use all original focal plane images as plane images for image fusion. This reduces the amount of data processing, improves image fusion efficiency, and balances the image fusion time. Furthermore, for different developmental stages, based on the feature analysis requirements of different developmental stages, an image fusion strategy corresponding to the developmental stage is used to fuse the plane images to be processed, achieving personalized image fusion processing. This ensures that the resulting fused image contains as many embryonic features as possible, thereby improving the accuracy of subsequent embryo identification based on the fused image.

[0145] Optionally, in step S202, determining each plane image to be processed corresponding to the target developmental stage from multiple original focal plane images may include: determining each plane image to be processed corresponding to the target developmental stage from multiple original focal plane images based on the target developmental stage and the pre-created correspondence between developmental stages and plane images to be processed.

[0146] In some embodiments, a correspondence between the developmental stage of an embryo and the planar image to be processed can be pre-constructed based on experiments or empirical analysis. This correspondence is used to indicate the relationship between the developmental stage of the embryo and which focal plane images are required to be processed when performing image fusion processing on the embryo image of that developmental stage.

[0147] Based on the established correspondence, once the target developmental stage is determined, the corresponding correspondence can be found to identify the planar images to be processed that correspond to the target developmental stage.

[0148] Optionally, in one feasible approach, the developmental stages of the embryo include: pronuclear stage, cleavage stage I, cleavage stage II, fusion stage, and blastocyst stage; here, taking multiple primitive focal plane images including a seven-focal image as an example, it is not actually limited to seven focal planes, nor is it limited to the seven focal planes listed; assuming that the multiple primitive focal plane images include: F-45 focal plane image, F-30 focal plane image, F-15 focal plane image, F0 focal plane image, F15 focal plane image, F30 focal plane image, and F45 focal plane image respectively.

[0149] Figure 3 This is a timeline of embryonic development stages provided in an embodiment of this application.

[0150] Embryo development stage prediction network: Developmental stages can be determined using multiple raw focal plane images of the embryo acquired through a time-lapse incubator. The embryo is categorized into five stages based on its development. In this embodiment, the InceptionV3 classification model framework is used, dividing the embryonic development stages into the pronuclear stage (no embryonic division), cleavage stage 1 (C2-C4), cleavage stage 2 (C5-C8), fusion stage, and blastocyst stage. Figure 3 The diagram shows the timeline of embryonic development. The horizontal axis represents the developmental time of the embryo since fertilization, and the vertical axis represents the classification labels of the five stages of embryonic development: 0: pronuclear stage, 1: cleavage stage 1 (C2-C4), 2: cleavage stage 2 (C5-C8), 3: fusion stage, and 4: blastocyst stage.

[0151] Optionally, in step S202, determining the corresponding planar images to be processed from multiple original focal plane images based on the target developmental stage may include:

[0152] If the target developmental stage is the pronuclear stage, then the F0 focal plane image is determined as the plane image to be processed from multiple original focal plane images.

[0153] When it is in the pronuclear stage, the F0 focal plane image can be directly identified as the plane image to be processed. Since the plane image to be processed only contains the F0 focal plane image at this stage, and there is only one image, there is no need to perform image fusion processing. The F0 focal plane image can be directly used in subsequent embryo analysis, and embryo analysis can be performed by extracting features from the F0 focal plane image.

[0154] If the target developmental stage is cleavage stage 1, then the F-30 focal plane image, F0 focal plane image, and F30 focal plane image are identified from multiple original focal plane images as the plane images to be processed.

[0155] When the embryo is in the first cleavage stage, the embryonic structure is not complex, and F-30 focal plane images, F0 focal plane images, and F30 focal plane images can be selected as the plane images to be processed.

[0156] If the target developmental stage is cleavage stage II, fusion stage, or blastocyst stage, then multiple original focal plane images are identified as plane images to be processed.

[0157] When the cleavage stage is in phase II, the blastomeres increase in size, the image complexity increases, and the determination of the number of blastomeres depends on the analysis of more layers of images. In this case, multiple original focal plane images (i.e., 7 focal plane images) can be used as the plane images to be processed.

[0158] During the fusion phase, multi-focal plane fused images are more convenient for observing the density of the embryo. The multi-focal plane fused images have rich detail and graininess. Similarly, multiple original focal plane images are used as plane images to be processed.

[0159] When the embryo is in the blastocyst stage, in order to facilitate the extraction of features of the trophoblast and inner cell mass, the image needs to have clear image edge features. During the morphological assessment of the blastocyst stage, embryologists can usually only refer to the trophoblast features on the equatorial plane. In this embodiment, multiple original focal plane images are also used as the plane images to be processed. After the blastocyst images are fused, clearer images with richer details can be obtained.

[0160] Optionally, in step S203, the target image fusion strategy corresponding to the target developmental stage is used to fuse the plane images to be processed to generate a fused image corresponding to the embryo, which may include:

[0161] If the target developmental stage is cleavage stage 1, an image channel fusion strategy is used to fuse the images of each plane to be processed, generating a fused image corresponding to the embryo.

[0162] In some embodiments, when the target developmental stage is the prokaryotic stage, image fusion processing is not required and is therefore ignored.

[0163] When the target developmental stage is cleavage stage 1, there are 3 planar images to be processed. At this time, an image channel fusion strategy can be used to perform channel fusion processing on the 3 planar images to be processed, and generate a fused RGB three-channel image.

[0164] If the target developmental stage is the second cleavage stage, the fusion stage, or the blastocyst stage, then an image fusion model is used to fuse the images of each plane to be processed, generating a fused image corresponding to the embryo.

[0165] When the target developmental stage is the second cleavage stage, the fusion stage, or the blastocyst stage, since there are many planar images to be processed, and in order to obtain a fused image containing more image details, a pre-trained image fusion model can be used to fuse the planar images to be processed to generate a fused image.

[0166] Figure 4 A flowchart illustrating the embryo image fusion method provided in this application embodiment. Figure 2 Optionally, in the above steps, the image channel fusion strategy is used to fuse the plane images to be processed to generate a fused image corresponding to the embryo, which may include:

[0167] S401. Using a channel fusion algorithm, the channels of each plane image to be processed are merged to generate the target RGB three-channel image.

[0168] This embodiment describes the image fusion method using an image channel fusion strategy during the first cleavage stage. Since each plane image to be processed is a single-channel grayscale image, mainly characterized by the brightness of the image, a channel fusion algorithm can be used to merge the channels of each plane image to be processed, that is, to fill the RGB three channels.

[0169] Here, existing algorithms in OpenCV can be used for implementation. For example, the RGB_merge function can be used to perform channel fusion, merging three planar images to be processed into a single target RGB three-channel image.

[0170] S402. Use the target RGB three-channel image as the fused image corresponding to the embryo.

[0171] Therefore, the target RGB three-channel image obtained after fusion is the fused image corresponding to the embryo at the first cleavage stage.

[0172] Figure 5 A flowchart illustrating the embryo image fusion method provided in this application embodiment. Figure 3 Optionally, before the above steps involve fusing the images of each plane to be processed using an image fusion model to generate the fused image corresponding to the embryo, this method may further include:

[0173] S501. Using a channel fusion algorithm, each plane image to be processed is merged into a channel image to generate an RGB three-channel image corresponding to each plane image to be processed.

[0174] This embodiment describes the method of performing image fusion processing using an image fusion model. Since each planar image to be processed is a single-channel grayscale image, and the input image fusion model needs to be a YCrCb image when performing image fusion processing, a channel fusion algorithm can be used to merge the channels of each planar image to be processed, that is, to fill each planar image to be processed into an RGB three-channel image.

[0175] S502. Perform color model conversion processing on the RGB three-channel images corresponding to each plane image to be processed, and generate the YCrCb image corresponding to each plane image to be processed.

[0176] For each planar image to be processed, the corresponding RGB three-channel image also needs to be converted into a color model, converting the RGB color model into YCrCb. Here, the Y channel represents the grayscale value, Cr reflects the difference between the red part of the RGB input signal and the brightness value of the RGB signal, and Cb reflects the difference between the blue part of the RGB input signal and the brightness value of the RGB signal.

[0177] Therefore, the step of using an image fusion model to fuse the plane images to be processed to generate a fused image corresponding to the embryo may include:

[0178] S503. Input the YCrCb images corresponding to each plane image to be processed into the image fusion model for image fusion processing to obtain the fused image corresponding to the embryo.

[0179] After obtaining the YCrCb images corresponding to each plane image to be processed, the image fusion model mainly performs fusion on the Y channel, since the structural details and brightness changes of the embryo image are reflected in the Y channel image data, while Cr and Cb are fused through conventional weighted fusion.

[0180] Therefore, the image fusion model can first split the YCrCb images corresponding to each plane image to be processed, extract the Y channel image data, and then perform image fusion processing on the Y channel image data. By combining the information of Cr and Cb after conventional weighted fusion processing, the fused image corresponding to the embryo can be obtained.

[0181] Optionally, the image fusion model described above may include a feature extraction module. Figure 6 This is a schematic diagram of a feature extraction module provided in an embodiment of this application. Figure 6 As shown, the feature extraction module may include: a first feature extraction layer, a second feature extraction layer, and a third feature extraction layer.

[0182] Figure 7 A flowchart illustrating the embryo image fusion method provided in this application embodiment. Figure 4 In step S503 above, the YCrCb images corresponding to each planar image to be processed are input into the image fusion model for image fusion processing to obtain the fused image corresponding to the embryo, which may include:

[0183] S701. Input the YCrCb image corresponding to each planar image to be processed into the feature extraction module, and perform convolution and pooling processing through the first feature extraction layer and the second feature extraction layer to obtain the first shallow feature information and the second shallow feature information.

[0184] Optionally, the YCrCb image corresponding to each planar image to be processed can be used as the input of the feature extraction module. After feature extraction through the first feature extraction layer and the second feature extraction layer, the first shallow feature information and the second shallow feature information can be obtained respectively. Then, convolution operation is performed on the first shallow feature information and the second shallow feature information respectively to obtain the first shallow feature information and the second shallow feature information after convolution processing.

[0185] S702. After inputting the first shallow feature information and the second shallow feature information into the third feature extraction layer for convolution processing, the third shallow feature information, the fourth shallow feature information and the fifth shallow feature information are obtained respectively.

[0186] The first and second shallow feature information obtained after convolution are used as inputs to the third feature extraction layer. After convolution, the third feature extraction layer generates third, fourth, and fifth shallow feature information, respectively. These three shallow feature information are all feature tensors of three layers.

[0187] S703. Perform fully connected processing on the third, fourth, and fifth shallow feature information respectively, and output the target third shallow feature information, target fourth shallow feature information, and target fifth shallow feature information.

[0188] Furthermore, by performing fully connected processing on the third, fourth, and fifth shallow feature information, the three feature tensors of the feature information extracted from the third shallow layer can be merged into a single feature tensor (i.e., the target third shallow feature information) for output, the three feature tensors of the feature information extracted from the fourth shallow layer can be merged into a single feature tensor (the target fourth shallow feature information) for output, and the three feature tensors of the feature information extracted from the fifth shallow layer can be merged into a single feature tensor (the target fifth shallow feature information) for output.

[0189] S704. The first shallow feature information, the second shallow feature information, the third shallow feature information of the target, the fourth shallow feature information of the target, and the fifth shallow feature information of the target are used as feature information output by the feature extraction module for information measurement.

[0190] The first and second shallow feature information extracted by the first and second feature extraction layers, along with the third, fourth, and fifth shallow feature information of the target output here, are combined as feature information extracted by the feature extraction module and output to the information measurement module in the image fusion model for information measurement processing.

[0191] This method uses the aforementioned feature extraction module for feature extraction. Compared with existing feature extraction networks, it reduces the deep feature extraction layer and adds a third feature extraction layer, focusing on the extraction of shallow feature information of the embryo, thereby extracting the texture and detail information of the embryo for embryo fusion and embryo analysis.

[0192] In other embodiments, this scheme can also perform image feature extraction based on the Unet network. The Unet network uses the Unet encoding and decoding network as the backbone network. The Unet network structure is a fully convolutional symmetrical network structure. The left side structure consists of convolutional layers forming downsampling layers, and the right side structure consists of deconvolutional layers forming upsampling layers. Each planar image to be processed is input into the Unet network, and the outputs of the first three layers of the downsampling layer and the upsampling layer are used as the feature extraction results, thereby obtaining the feature extraction results. Its advantage is that it can preserve the detailed features of small targets and images, and has a good effect in embryo feature extraction.

[0193] Figure 8 A flowchart illustrating the embryo image fusion method provided in this application embodiment. Figure 5 Optionally, in step S203, before fusing the planar images to be processed using a target image fusion strategy corresponding to the target developmental stage to generate the fused image corresponding to the embryo, the following may be included:

[0194] S601. Perform image contour compensation on the target image in each plane image to be processed to obtain the compensated target image; the target image includes: each plane image to be processed above F0 focal length.

[0195] This implementation describes the preprocessing operations for each plane image to be processed. After selecting each plane image to be processed from multiple original focal plane images according to the embryonic development stage, image preprocessing can be performed on the specified target image in each plane image to be processed.

[0196] Therefore, when using the target image fusion strategy corresponding to the target development stage to fuse each plane image to be processed, the preprocessed plane images to be processed are used as input to execute the image fusion strategy.

[0197] Since the camera is positioned below during imaging, artifacts from the lower layers of the embryo's equatorial plane will superimpose when capturing images of the upper layers, which is detrimental to feature extraction. Therefore, it is necessary to select the upper focal plane images, i.e., the various plane images to be processed above F0, such as the F15, F30, and F45 focal plane images mentioned above, for image preprocessing so that they can be better used for image fusion.

[0198] Figure 9 This diagram illustrates an image preprocessing procedure provided in an embodiment of this application. Taking the preprocessing of an F45 focal plane image as an example:

[0199] like Figure 9Image (a) is the unprocessed F45 focal plane image. An impulse filter is first applied to compensate for the image's contours, increasing edge detail and grayscale variations, and enhancing the contours of cell edges and features of some linear target elements in the embryo image. For example... Figure 9 (b) in the image is the compensated image obtained after processing with the impulse filter.

[0200] S602. Perform contrast enhancement on the compensated target image to obtain the enhanced target image.

[0201] Then, contrast enhancement is used to further amplify the enhanced edge contours, such as... Figure 9 (c) in the image is the image after contrast enhancement.

[0202] S603. Perform noise filtering on the enhanced target image to generate a denoised target image.

[0203] Then, Gaussian filtering is used for denoising to reduce noise interference that may be introduced during image fusion, such as... Figure 9 (d) in the image is the image after noise reduction.

[0204] S604. Perform image enhancement processing on the denoised target image to generate the processed image corresponding to each plane image to be processed.

[0205] Figure 10 This is a schematic diagram illustrating another image preprocessing method provided in an embodiment of this application. For example... Figure 10 As shown, a deconvolution operation is performed on the denoised image to improve image clarity. Then, edge enhancement and histogram equalization are performed on the image to obtain the processed image.

[0206] Figure 11 This is a schematic diagram illustrating an image preprocessing result provided in an embodiment of this application. For example... Figure 11 The image shown is a display of the results of image edge enhancement and grayscale histogram equalization processing on 11 focal plane images.

[0207] Optionally, the input data channels of the image fusion model used in the above embodiments of this solution may include multiple channels;

[0208] The image fusion model used in this embodiment is a modified version of the U2Fusion network model, with modifications to the input layer and feature extraction layer. Firstly, the input data channels were modified from the original two images to multiple focal plane images.

[0209] The modifications to the feature extraction layer can be understood by referring to the above explanation of the feature extraction module, and will not be repeated here.

[0210] Figure 12 A flowchart illustrating the embryo image fusion method provided in this application embodiment. Figure 6 The image fusion model can be trained using the following methods:

[0211] S1001. Collect multiple sets of training sample images. Each set of training sample images includes multiple original focal plane images corresponding to an embryo.

[0212] Optionally, multiple sets of training sample images can be randomly collected from the database. Each set of training sample images may include multiple original focal plane images corresponding to an embryo. These may be multiple original focal plane images of the same embryo at different developmental stages, or multiple original focal plane images of different embryos.

[0213] S1002. Perform image preprocessing on each group of training sample images to obtain the processed training sample images for each group.

[0214] Optionally, the training sample images can be preprocessed in the manner described in steps S601-S604 above, and the same preprocessing can be performed on each planar image to be processed above F0 in each group of training sample images to obtain the processed training sample images.

[0215] S1003. Perform channel fusion processing on each group of processed training sample images to generate each group of target training sample images.

[0216] For the processed training sample images, the channel fusion processing method in steps S501-S502 can be used to perform channel fusion processing on each image in the processed training sample images to generate target training sample images for each group. Each target training sample image is a YCrCb image that meets the input requirements of the image fusion model.

[0217] S1004. Input the target training sample images of each group into the initial image fusion model for model training to generate the image fusion model.

[0218] Optionally, the target training sample images of each group can be used as input data into the initial image fusion model. After model training, the image fusion model can be generated. The initial image fusion model can be the model with the initial parameters. After model training, the model can have the required network parameters, thus obtaining the image fusion model.

[0219] Figure 13 A flowchart illustrating the embryo image fusion method provided in this application embodiment. Figure 7Optionally, in step S1004, inputting the target training sample images of each group into the initial image fusion model for model training to generate the image fusion model may include:

[0220] S1101. Using the initial image fusion model, feature extraction is performed on each group of target training sample images to obtain multiple feature information of each original focal plane image in each group of target training sample images.

[0221] Optionally, for each original focal plane image in each set of target training sample images, a set of feature information can be generated through the Unet feature extraction layer, including: f1(I1), f2(I1)...f6(I1). Thus, multiple sets of feature information can be generated for each set of target training sample images.

[0222] S1102. Based on multiple feature information of each original focal plane image in each group of target training sample images, perform information measurement and generate measurement results. The measurement results are used to characterize the degree of feature information extracted from each original focal plane image.

[0223] The information measurement process involves measuring the extracted feature information using gradient operators to ensure that the information in the source image is not distorted.

[0224] Therefore, an information metric calculation formula can be used to calculate the metric result based on the extracted feature information. This formula generates a metric result represented by two parameters, gI1 and gI2. Specifically, the metric result can be calculated using the following formula:

[0225]

[0226] in, Let k represent the feature map of the lower j-th layer, and k represent D. j The feature map in the k-th channel of the given channels. This represents the Laplace operator.

[0227] It should be noted that the metric result is used to characterize the degree of feature information extracted by the feature extraction layer, that is, the information retention, which is the amount of feature information that can be learned from the original image. The larger the metric result, the richer the extracted feature information.

[0228] S1103. Generate the fusion weights of the image based on the measurement results.

[0229] Optionally, based on the generated metrics gI1 and gI2, the fusion weights of the images can be further calculated:

[0230]

[0231] Here, softmax represents normalization, C is a constant, and scaling is performed by adjusting the value of C to obtain better weights. In this embodiment, the value of C can be 0.01, and ω1 and ω2 represent two weights respectively.

[0232] S1104. Calculate the loss of the initial image fusion model based on the fusion weights of each original focal plane image in each group of target training sample images.

[0233] In this embodiment, the model loss can be calculated using the following formula:

[0234]

[0235]

[0236]

[0237] in, This represents the model's loss value, SSIM is the image structural similarity, and MES is the image mean squared error.

[0238] S1105. Based on the loss, iteratively optimize the initial image fusion model to generate an image fusion model.

[0239] Optionally, the model training process is performed iteratively. In each iteration, the loss of the current round can be calculated. If the loss of the current round meets the loss accuracy, the iteration stops and the model obtained by the current training is used as the image fusion model. If the loss of the current round does not meet the loss accuracy, the next round of training is carried out in the above manner, and a new loss is calculated until the new loss meets the loss accuracy and then the process stops.

[0240] Figure 14 This is a schematic diagram of an image fusion result provided in an embodiment of this application, such as... Figure 14 The image shown is a fused image obtained after image fusion processing of a cleavage-stage embryo image using the image fusion model trained based on this scheme. It can be seen that the blastomeres of the embryo have a stronger three-dimensional effect and clearer edges.

[0241] Figure 15 This is a schematic diagram of another image fusion result provided in an embodiment of this application, such as... Figure 15 The image shown is a fused image obtained after image fusion processing of blastocyst stage embryo images using the image fusion model trained based on this scheme. It can be seen that the sharp regions at different depths of field are superimposed, allowing for the observation of more details in the internal structure of the blastocyst.

[0242] Optionally, the method of this application may further include: calculating and obtaining the performance index of the image fusion model based on the original F0 focal plane image of the target embryo and the corresponding fused image of the target embryo using at least one performance index algorithm; the performance index is used to evaluate the fusion performance of the image fusion model; the performance index algorithm includes: gradient-based fusion performance algorithm, image mean square error algorithm, structural similarity algorithm and peak signal-to-noise ratio algorithm.

[0243] In some embodiments, for the image fusion model trained above, this scheme also employs at least one performance metric calculation method to evaluate the performance of the image fusion model.

[0244] These can include the following performance metrics: QUtil (gradient-based fusion performance), image mean square error (RMES), structural similarity (SSIM), and peak signal-to-noise ratio (PSNR).

[0245] Optionally, when calculating performance metrics, an original image and a fused new image can be selected for fusion quality evaluation. The original image can be the F0 focal plane image among multiple original focal plane images, and the fused new image is the fused image generated after fusion processing by the image fusion model.

[0246] The calculation methods for each performance index can be referred to the existing formulas, and will not be elaborated further here.

[0247] In summary, the embryo image fusion method provided in this embodiment classifies and predicts multiple input original focal plane images using an embryonic developmental stage prediction network to determine the developmental stage of the embryo at various points. Based on the developmental stage, focal plane images corresponding to the developmental stage are selected from the multiple original focal plane images as the plane images to be processed. An image fusion strategy corresponding to the developmental stage is then applied to fuse these plane images, ultimately resulting in a fused image. This method selects focal plane images corresponding to the developmental stage from multiple original focal plane images to be processed, avoiding the need to use all original focal plane images as plane images for image fusion. This reduces the amount of data processing, improves image fusion efficiency, and balances the image fusion time. Furthermore, based on the feature analysis requirements of different developmental stages, an image fusion strategy corresponding to the developmental stage is used to fuse the plane images, achieving personalized image fusion processing. This ensures that the resulting fused image contains as many embryonic features as possible, thereby improving the accuracy of subsequent embryo identification based on the fused image.

[0248] Secondly, this application improves the input layer and feature extraction layer based on the existing fusion network. On the one hand, it enables the trained image fusion model to be applicable to the processing of multiple inputs. On the other hand, the extracted features are richer, making the image fusion result more ideal.

[0249] The apparatus, device, and storage medium used to perform the embryo image fusion method provided in this application are described below. The specific implementation process and technical effects are described above and will not be repeated below.

[0250] Figure 16 This is a schematic diagram of an embryo image fusion device provided in an embodiment of this application. The function implemented by this embryo image fusion device corresponds to the steps performed by the above-described method. This device can be understood as the aforementioned computer equipment or server, or the processor of a server, or it can be understood as a component that implements the functions of this application under the control of a server, independent of the aforementioned server or processor. Figure 16 As shown, the device may include: a prediction module 140, a determination module 141, and a processing module 142;

[0251] The prediction module 140 is used to predict the target developmental stage of the embryo based on multiple raw focal plane images of the acquired embryo.

[0252] The determination module 141 is used to determine, based on the target developmental stage, each plane image to be processed corresponding to the target developmental stage from multiple original focal plane images;

[0253] The processing module 142 is used to perform fusion processing on each plane image to be processed using the target image fusion strategy corresponding to the target developmental stage to generate a fused image corresponding to the embryo. The target image fusion strategy includes: an image channel fusion strategy or a fusion strategy based on an image fusion model. The image fusion model is pre-trained based on multiple sets of training samples composed of original focal plane images of the embryo.

[0254] Optionally, the determining module 141 is specifically used to determine, based on the target developmental stage, each to-be-processed planar image corresponding to the target developmental stage from multiple original focal plane images.

[0255] Optionally, the developmental stages of the embryo include: pronuclear stage, cleavage stage I, cleavage stage II, fusion stage, and blastocyst stage; the multiple primitive focal plane images include: F-45 focal plane image, F-30 focal plane image, F-15 focal plane image, F0 focal plane image, F15 focal plane image, F30 focal plane image, and F45 focal plane image, respectively;

[0256] Determine module 141, specifically used for

[0257] If the target developmental stage is the pronuclear stage, then the F0 focal plane image is determined as the plane image to be processed from multiple original focal plane images;

[0258] If the target developmental stage is cleavage stage 1, then the F-30 focal plane image, F0 focal plane image, and F15 focal plane image are identified from multiple original focal plane images as the plane images to be processed.

[0259] If the target developmental stage is cleavage stage II, fusion stage, or blastocyst stage, then multiple original focal plane images are identified as plane images to be processed.

[0260] Optionally, the processing module 142 is specifically used for

[0261] If the target developmental stage is cleavage stage 1, an image channel fusion strategy is used to fuse the images of each plane to be processed, generating a fused image of the embryo.

[0262] If the target developmental stage is the second cleavage stage, the fusion stage, or the blastocyst stage, then an image fusion model is used to fuse the images of each plane to be processed, generating a fused image corresponding to the embryo.

[0263] Optionally, the processing module 142 is specifically used for

[0264] A channel fusion algorithm is used to merge the channels of each plane image to be processed, generating a target RGB three-channel image.

[0265] The target RGB three-channel image is used as the fused image corresponding to the embryo.

[0266] Optionally, the device further includes: a channel fusion module;

[0267] The channel fusion module is used to merge the channels of each plane image to be processed using a channel fusion algorithm, and generate an RGB three-channel image corresponding to each plane image to be processed.

[0268] The RGB three-channel images corresponding to each plane image to be processed are subjected to color model conversion processing to generate the YCrCb image corresponding to each plane image to be processed.

[0269] The processing module 142 is specifically used to input the YCrCb images corresponding to each plane image to be processed into the image fusion model for image fusion processing to obtain the fused image corresponding to the embryo.

[0270] Optionally, the image fusion model includes a feature extraction module, which includes a first feature extraction layer, a second feature extraction layer, and a third feature extraction layer.

[0271] Processing module 142 is specifically used to input the YCrCb image corresponding to each planar image to be processed into the feature extraction module, and perform convolution and pooling processing through the first feature extraction layer and the second feature extraction layer to obtain the first shallow feature information and the second shallow feature information.

[0272] After inputting the first and second shallow feature information into the third feature extraction layer for convolution processing, the third, fourth, and fifth shallow feature information are obtained respectively.

[0273] Fully connected processing is performed on the third, fourth, and fifth shallow feature information respectively to output the target's third, fourth, and fifth shallow feature information.

[0274] The first shallow feature information, the second shallow feature information, the third shallow feature information, the fourth shallow feature information, and the fifth shallow feature information of the target are used as the feature information output by the feature extraction module for information measurement.

[0275] Optionally, the apparatus further includes: a preprocessing module;

[0276] The preprocessing module is used to perform image contour compensation on the target images in each planar image to be processed, so as to obtain the compensated target image; the target images include: each planar image to be processed above F0 focal length;

[0277] The contrast of the compensated target image is enhanced to obtain the enhanced target image;

[0278] The enhanced target image is subjected to noise filtering to generate a denoised target image.

[0279] The denoised target image is then subjected to image enhancement processing to generate the processed image corresponding to each plane image to be processed.

[0280] Optionally, the image fusion model may have multiple input data channels; the device may also include a training module.

[0281] Training module, used for

[0282] Multiple sets of training sample images are collected, and each set of training sample images includes multiple original focal plane images corresponding to one embryo.

[0283] Each group of training sample images is preprocessed to obtain the processed training sample images.

[0284] Each group of training sample images is processed by channel fusion to generate target training sample images for each group.

[0285] Each group of target training sample images is input into the initial image fusion model for model training, generating the image fusion model.

[0286] Optionally, the training module is specifically used for

[0287] The initial image fusion model is used to extract features from each group of target training sample images to obtain multiple feature information of each original focal plane image in each group of target training sample images;

[0288] Information is measured based on multiple feature information of each original focal plane image in each set of target training sample images, and measurement results are generated. The measurement results are used to characterize the degree of feature information extracted from each original focal plane image.

[0289] Generate fusion weights for the image based on the measurement results;

[0290] The loss of the initial image fusion model is calculated based on the fusion weights of each original focal plane image in each group of target training sample images.

[0291] Based on the loss, the initial image fusion model is iteratively optimized to generate an image fusion model.

[0292] Optionally, the device further includes: an evaluation module;

[0293] The evaluation module is used to calculate and obtain the performance index of the image fusion model based on the original F0 focal plane image of the target embryo and the corresponding fused image of the target embryo, using at least one performance index algorithm. The performance index is used to evaluate the fusion performance of the image fusion model. The performance index algorithms include: gradient-based fusion performance algorithm, image mean square error algorithm, structural similarity algorithm, and peak signal-to-noise ratio algorithm.

[0294] The above-described device is used to execute the method provided in the foregoing embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0295] These modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more digital signal processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). Alternatively, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a system-on-a-chip (SOC).

[0296] The modules described above can be connected or communicate with each other via wired or wireless connections. Wired connections can include metal cables, optical fibers, hybrid cables, or any combination thereof. Wireless connections can include connections via LAN, WAN, Bluetooth, ZigBee, or NFC, or any combination thereof. Two or more modules can be combined into a single module, and any module can be divided into two or more units. Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here.

[0297] Figure 17 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application; the device includes: a processor 801 and a storage medium 802.

[0298] Storage medium 802 is used to store programs, and processor 801 calls the programs stored in storage medium 802 to execute the above method embodiments. The specific implementation and technical effects are similar, and will not be described in detail here.

[0299] The storage medium 802 stores program code, which, when executed by the processor 801, causes the processor 801 to perform various steps in the embryo image fusion method according to various exemplary embodiments of this application as described in the "Exemplary Methods" section above.

[0300] The processor 801 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0301] Storage medium 802, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. The storage medium can include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type storage medium, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage medium, magnetic disk, optical disk, etc. The storage medium is any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. In the embodiments of this application, storage medium 802 can also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0302] Optionally, this application also provides a program product, such as a computer-readable storage medium, including a program that, when executed by a processor, performs the above-described method embodiments.

[0303] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0304] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0305] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units.

[0306] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for embryo image fusion, characterized in that, include: Based on multiple raw focal plane images of the acquired embryo, the target developmental stage of the embryo is predicted; Based on the target developmental stage, determine each plane image to be processed corresponding to the target developmental stage from the plurality of original focal plane images; The target image fusion strategy corresponding to the target developmental stage is used to fuse the plane images to be processed to generate the fused image corresponding to the embryo. The target image fusion strategy includes: image channel fusion strategy or fusion strategy based on image fusion model. The image fusion model is pre-trained based on multiple sets of training samples composed of original focal plane images of the embryo. The developmental stages of the embryo include: pronuclear stage, cleavage stage I, cleavage stage II, fusion stage, and blastocyst stage; the multiple original focal plane images include: F-45 focal plane image, F-30 focal plane image, F-15 focal plane image, F0 focal plane image, F15 focal plane image, F30 focal plane image, and F45 focal plane image; The step of determining, based on the target developmental stage, each to-be-processed planar image corresponding to the target developmental stage from the plurality of original focal plane images includes: If the target developmental stage is the pronuclear stage, then the F0 focal plane image is determined as the plane image to be processed from the plurality of original focal plane images; If the target developmental stage is the first cleavage stage, then the F-30 focal plane image, the F0 focal plane image, and the F30 focal plane image are determined from the plurality of original focal plane images as the plane image to be processed; If the target developmental stage is the second cleavage stage, the fusion stage, or the blastocyst stage, then the plurality of original focal plane images are determined as the plane images to be processed; The step of fusing the planar images to be processed using the target image fusion strategy corresponding to the target developmental stage to generate the fused image corresponding to the embryo includes: If the target developmental stage is the first cleavage stage, then the image channel fusion strategy is used to fuse the plane images to be processed to generate the fused image corresponding to the embryo. If the target developmental stage is the second cleavage stage, the fusion stage, or the blastocyst stage, then the image fusion model is used to fuse the planar images to be processed, generating the fused image corresponding to the embryo.

2. The method according to claim 1, characterized in that, The step of determining, based on the target developmental stage, each to-be-processed planar image corresponding to the target developmental stage from the plurality of original focal plane images includes: Based on the target developmental stage and the pre-created correspondence between the developmental stage and the planar image to be processed, each planar image to be processed corresponding to the target developmental stage is determined from the plurality of original focal plane images.

3. The method according to claim 2, characterized in that, Before fusing the various planar images to be processed using the image fusion model to generate the fused image corresponding to the embryo, the process includes: A channel fusion algorithm is used to merge the channels of each plane image to be processed, generating an RGB three-channel image corresponding to each plane image to be processed. The RGB three-channel images corresponding to each plane image to be processed are subjected to color model conversion processing to generate the YCrCb image corresponding to each plane image to be processed. The step of fusing the image fusion model to generate the fused image corresponding to the embryo includes: The YCrCb images corresponding to each planar image to be processed are input into the image fusion model for image fusion processing to obtain the fused image corresponding to the embryo.

4. The method according to claim 3, characterized in that, The image fusion model includes a feature extraction module, which includes a first feature extraction layer, a second feature extraction layer, and a third feature extraction layer. The step of inputting the YCrCb images corresponding to each planar image to be processed into the image fusion model for image fusion processing to obtain the fused image corresponding to the embryo includes: The YCrCb images corresponding to each planar image to be processed are input into the feature extraction module, and convolution and pooling are performed through the first feature extraction layer and the second feature extraction layer to obtain the first shallow feature information and the second shallow feature information. After inputting the first and second shallow feature information into the third feature extraction layer for convolution processing, the third, fourth, and fifth shallow feature information are obtained respectively. The third, fourth, and fifth shallow feature information are processed by full connection respectively to output the target third shallow feature information, the target fourth shallow feature information, and the target fifth shallow feature information. The first shallow feature information, the second shallow feature information, the third shallow feature information of the target, the fourth shallow feature information of the target, and the fifth shallow feature information of the target are used as the feature information output by the feature extraction module for information measurement.

5. The method according to claim 1, characterized in that, Before fusing the planar images to be processed using the target image fusion strategy corresponding to the target developmental stage to generate the fused image corresponding to the embryo, the process includes: Image contour compensation is performed on the target images in each plane image to be processed to obtain the compensated target image; the target image includes: each plane image to be processed with an F0 focal length or above; The contrast of the compensated target image is enhanced to obtain the enhanced target image; The enhanced target image is subjected to noise filtering to generate a denoised target image. The denoised target image is then subjected to image enhancement processing to generate the processed image corresponding to each plane image to be processed.

6. The method according to claim 1, characterized in that, The image fusion model has multiple input data channels; the image fusion model is trained using the following method: Multiple sets of training sample images are collected, and each set of training sample images includes multiple original focal plane images corresponding to one embryo. Each group of training sample images is preprocessed to obtain the processed training sample images. Each group of training sample images is processed by channel fusion to generate target training sample images for each group. Each group of target training sample images is input into the initial image fusion model for model training, thereby generating the image fusion model.

7. The method according to claim 6, characterized in that, The step of inputting each group of target training sample images into the initial image fusion model for model training to generate the image fusion model includes: The initial image fusion model is used to extract features from each group of target training sample images to obtain multiple feature information of each original focal plane image in each group of target training sample images; Information is measured based on multiple feature information of each original focal plane image in each group of target training sample images, and a measurement result is generated. The measurement result is used to characterize the degree of feature information extracted from each original focal plane image. The fusion weights of the image are generated based on the measurement results; The loss of the initial image fusion model is calculated based on the fusion weights of each original focal plane image in each group of target training sample images. Based on the loss, the initial image fusion model is iteratively optimized to generate the image fusion model.

8. An embryo image fusion device, characterized in that, include: Prediction module, determination module, processing module; The prediction module is used to predict the target developmental stage of the embryo based on multiple original focal plane images of the acquired embryo. The determining module is used to determine, from the plurality of original focal plane images, each plane image to be processed corresponding to the target developmental stage, based on the target developmental stage. The processing module is used to perform fusion processing on each plane image to be processed using the target image fusion strategy corresponding to the target developmental stage to generate a fused image corresponding to the embryo. The target image fusion strategy includes: an image channel fusion strategy or a fusion strategy based on an image fusion model. The image fusion model is pre-trained based on multiple sets of training samples composed of original focal plane images of the embryo. The developmental stages of the embryo include: pronuclear stage, cleavage stage I, cleavage stage II, fusion stage, and blastocyst stage; the multiple original focal plane images include: F-45 focal plane image, F-30 focal plane image, F-15 focal plane image, F0 focal plane image, F15 focal plane image, F30 focal plane image, and F45 focal plane image; The determining module is specifically used to determine the F0 focal plane image as the plane image to be processed from the plurality of original focal plane images if the target developmental stage is the prokaryotic stage; If the target developmental stage is the first cleavage stage, then the F-30 focal plane image, the F0 focal plane image, and the F30 focal plane image are determined from the plurality of original focal plane images as the plane image to be processed; If the target developmental stage is the second cleavage stage, the fusion stage, or the blastocyst stage, then the plurality of original focal plane images are determined as the plane images to be processed; The processing module is specifically used to perform fusion processing on each plane image to be processed using the image channel fusion strategy if the target developmental stage is the first cleavage stage, so as to generate the fused image corresponding to the embryo. If the target developmental stage is the second cleavage stage, the fusion stage, or the blastocyst stage, then the image fusion model is used to fuse the planar images to be processed, generating the fused image corresponding to the embryo.

9. An electronic device, characterized in that, include: The device includes a processor, a storage medium, and a bus, wherein the storage medium stores program instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the program instructions to implement the embryo image fusion method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which is executed by a processor to implement the embryo image fusion method as described in any one of claims 1 to 7.

Images