Prokaryotic target detection method based on multi-focal length embryo image fusion based on double-layer optimization

Through the double-layer optimization model and gradient residual module, the accuracy of prokaryotic object detection in multi-focal embryonic cells is solved, and the accuracy and robustness of the detection system are improved, especially prokaryotic detection in the case of inconsistent and overlapping focal points Effect.

CN118609123BActive Publication Date: 2024-11-19WUHAN MUTUAL UNITED TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411077329.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-07
Publication Date
2024-11-19
Estimated Expiration
2044-08-07

Smart Images

  • Figure CN118609123B_ABST
    Figure CN118609123B_ABST
Patent Text Reader

Abstract

The present invention discloses a prokaryotic target detection method based on multi-focal length embryo image fusion based on double-layer optimization, comprising the following steps: step S1: preprocessing the multi-focal length embryo image; step S2: establishing a double-layer optimization model with the inner layer minimizing the mean square error of the image fusion module as the goal and the outer layer minimizing the bounding box regression loss of the target detection module as the goal, taking the optimal solution of the inner layer as the input of the outer layer, and feeding back the optimization result of the outer layer to the inner layer to optimize the weight parameter of the image fusion module; step S3: inputting the preprocessed image into the double-layer optimization model to obtain a fused image; step S4: inputting the fused image into the target detection module to obtain the prokaryotic detection result of the multi-focal length embryo image. The present invention solves the problem that the prokaryotic target in the multi-focal length embryo image is difficult to identify.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a prokaryotic target detection method based on multi-focal length embryo image fusion based on double-layer optimization. Background Art

[0002] Pronuclei are a type of cell nucleus that is not bounded by a nuclear membrane and contains the cell's genetic material, such as DNA and RNA. Pronuclei are the basic form of cells in the early stages of embryonic cell development, with simple structures and functions. They provide the necessary foundation for the further development of embryonic cells. However, if there are too many pronuclei in embryonic cells, it will cause cytoplasmic deformities. Current studies have shown that pronuclei are associated with changes in the cytoskeleton and can affect the function of embryonic cells, leading to chromosome and mitochondrial abnormalities during embryonic cell activation. Therefore, pronucleus detection of embryonic cells is necessary.

[0003] By using advanced computer vision technology to build a detection model, doctors can better understand the specific situation of the pronucleus in embryonic cells and make more accurate judgments. However, there are still some unresolved issues in the current target detection method for pronuclei in embryonic cells. These issues will affect the model's judgment of the pronucleus, and thus affect the assessment of embryo quality.

[0004] Under the electron microscope, due to the slight instability of the focusing system, the focus position is slightly offset during imaging, resulting in inconsistent imaging clarity. The prokaryotic data generated under different frames have a multi-focus problem, that is, the image is partially clear and partially blurred. Some even make it difficult to distinguish the prokaryotic target position with the naked eye, which greatly affects the accuracy of prokaryotic target detection. Pronuclei are crucial in embryonic cells and play a vital role in cell function and development. However, due to factors such as unequal chromosome segregation or increased cellular metabolic activity, multiple pronuclei may exist at the same time, and they may overlap or occlude each other, thus affecting the detection performance of prokaryotic target detection. Summary of the invention

[0005] The present invention proposes a prokaryotic target detection method based on multi-focal length embryo image fusion based on double-layer optimization, which solves the problem that the existing model is difficult to identify prokaryotic targets in multi-focal length embryo cells.

[0006] In order to solve the above technical problems, the present invention provides a prokaryotic target detection method based on multi-focal length embryo image fusion based on double-layer optimization, comprising the following steps:

[0007] Step S1: selecting a pair of images with complementary focal lengths from the multi-focal length embryo image to form a complementary image pair, wherein the corresponding position of the blurred focus area in the first image of the complementary image pair is a clearly focused area in the second image, and the corresponding position of the blurred focus area in the second image is a clearly focused area in the first image, and selecting a plurality of complementary image pairs to form a complementary image set;

[0008] Step S2: Establish a two-layer optimization model with the inner layer minimizing the mean square error of the image fusion module as the goal and the outer layer minimizing the bounding box regression loss of the object detection module as the goal. The optimal solution of the inner layer is used as the input of the outer layer, and the optimization result of the outer layer is fed back to the inner layer to optimize the weight parameters of the image fusion module.

[0009] Step S3: inputting the plurality of complementary image pairs into an optimized image fusion module to obtain a plurality of composite images, and inputting the plurality of composite images into the optimized image fusion module again to obtain a fused image;

[0010] Step S4: inputting the fused image into a target detection module to obtain a pronucleus detection result of a multi-focal length embryo image.

[0011] Preferably, step S1 pre-processes the multi-focal length embryo image before selecting the image, including the following steps: evaluating the clarity of the multi-focal length embryo image, deleting images with clarity lower than the set clarity, and directly inputting the image with the highest clarity into the target detection module.

[0012] Preferably, the expression of the two-layer optimization model in step S2 is:

[0013] ;

[0014] In the formula, represents the bounding box regression loss function of the object detection module; are the parameters of the target detection module; The optimal solution for the inner optimization model; is the mean square error function of the image fusion module; Represents the parameters of the image fusion module; Represents the output function of the target detection module.

[0015] Preferably, the bounding box regression loss function is expressed as:

[0016]

[0017] In the formula, is the number of input images; , Respectively i The horizontal and vertical coordinates of the center point of the predicted bounding box; , Respectively i The width and height of the predicted bounding box, , Respectively i The horizontal and vertical coordinates of the center point of the real bounding box; , Respectively i The width and height of the ground-truth bounding box; for The weight value of .

[0018] Preferably, the expression of the mean square error function is:

[0019] ;

[0020] ;

[0021] In the formula, The image generated for fusion; is the input image; W , H is the width and height of the image; is the pixel coordinate; For the image at pixel points The pixel value of Pixel The weight value of .

[0022] Preferably, when the plurality of synthesized images are input into the optimized image fusion module again to obtain a fused image in step S3, a gradient residual module is added to the image fusion module, and the gradient residual module combines the image features and gradient information to strengthen the image fusion module's retention of edge information, comprising the following steps:

[0023] Step S31: Randomly select two synthetic images from multiple synthetic images as input images, extract the features of the input images, and obtain a feature map , ;

[0024] Step S32: Remove the noise in the feature map to obtain a denoised feature map , ;

[0025] Step S33: Use the gradient operator to calculate , The gradient information is combined with , The features of and ;

[0026] Step S34: stitching and , then perform image reconstruction to generate a preliminary fused image, use the preliminary fused image as a new composite image, add it to multiple composite images, and repeat the above steps until the final fused image is obtained.

[0027] Preferably, the expression for removing noise in the feature map in step S32 is:

[0028] ;

[0029] In the formula, is the convolution operation; is the regularization operation; Indicates the activation function operation; is the feature map, .

[0030] Preferably, step S33 combines the gradient information with , The expression combining the features is:

[0031] ;

[0032] In the formula, To fusion features, ; It is a convolution with a convolution kernel size of 1; Includes a convolution layer and an activation function; is the gradient operator.

[0033] Preferably, the image fusion module also introduces a gradient loss, and the expression of the gradient loss is:

[0034] ;

[0035] In the formula, is the gradient loss; W and H are the width and height of the input image respectively; Indicates the image at pixel points The pixel gradient value of To fuse the images, is the input image; represents L2 regularization.

[0036] Preferably, in step S4, the expression for obtaining the pronucleus detection result of the multi-focal length embryo image by inputting the fused feature map into the target detection module is:

[0037] ;

[0038] inD represents a detector; I d is the final test result; I f is the fusion feature map.

[0039] The benefits of the present invention include at least:

[0040] 1. A two-layer optimization model is proposed. Through iteration and feedback mechanism, close coordination between the image fusion module and the target detection module is achieved, so that the image fusion module and the target detection module can influence and optimize each other, making the entire prokaryotic detection have a higher accuracy rate;

[0041] 2. By selecting images with complementary focal lengths, the clear areas in each image can be maximized, improving the clarity of the overall image; by fusing multiple complementary image pairs in steps, the image quality can be gradually improved instead of processing all images at once, which helps to control the fusion process more finely. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 Schematic diagram of multi-focus pronuclear images generated under an electron microscope;

[0043] Figure 2 Schematic diagram of overlapping images of pronuclei in embryonic cells;

[0044] Figure 3 A schematic diagram of a method flow chart of an embodiment of the present invention;

[0045] Figure 4 Schematic diagram of the interaction between the inner and outer layers of the two-layer optimization model;

[0046] Figure 5 This is a comparison chart of prokaryotic detection results before and after adding double-layer optimization;

[0047] Figure 6 It is a schematic diagram of the structure of the CBR module;

[0048] Figure 7 It is a schematic diagram of the structure of the GDM module;

[0049] Figure 8 It is a comparison chart of prokaryotic detection results before and after adding the GDM module and content loss;

[0050] Fig. 9 This is a schematic diagram of the network structure of the prokaryotic target detection network model MBPD;

[0051] Fig.10 This is the prokaryotic detection effect diagram of the prokaryotic target detection network model MBPD. DETAILED DESCRIPTION

[0052] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention.

[0053] Pronuclei play a vital role in the function and development of embryonic cells, but if a large number of pronuclei aggregate, it may cause impaired cell function, abnormal cell development and abnormal reproductive ability. Therefore, the detection of pronuclei is very important in embryonic cells. However, during electron microscopy imaging, slight changes in focal length may cause a slight shift in the focus position during imaging. This slight instability may be caused by the mechanical structure of the equipment itself or the characteristics of the electron beam. Since the pronucleus is located inside the cell, it is usually small and located relatively deep, so a slight shift in focus may cause multi-focus problems in the pronuclear data generated in different frames. Figure 1 The prokaryotic data images generated under an electron microscope are shown. Figure 1 It can be seen that the prokaryotic data in different frames have the problem of partial clarity and partial blur, and some prokaryotic target areas are very blurry. As shown in images No. 1 and 3, the image No. 1 is relatively clear as a whole, and the image No. 3 is relatively blurry as a whole. However, by comparing the details, it can be found that the edge information of a prokaryotic target in No. 3 is clearer than that in No. 1, and the rest of the places are clearer in No. 1. Other images also have the problem of partial clarity and partial blur. Since there is no completely clear prokaryotic image data, it will have a great impact on the subsequent prokaryotic detection.

[0054] Figure 2 Depicts the overlap of prokaryotic targets during cell division, from Figure 2From the two images numbered 1 and 2, it can be seen that there are two prokaryotic detection targets in the detection image, of which areas a and c are non-overlapping areas, and area b is an overlapping area. Due to the overlap of prokaryotic targets, it can be clearly seen that there is a certain loss of edge information in the overlapping area. This loss of edge information will make it difficult for the algorithm to determine the boundary of the target, so that there are problems such as failure to detect prokaryotic targets or large detection errors in the detection results. The most obvious one is the image numbered 3. The prokaryotic target pointed by the white arrow has no obvious features such as edge texture in the overlapping area or the non-overlapping area, which is similar to the background. This situation undoubtedly increases the difficulty of detection. It can be seen that the loss of a large amount of key information such as edges and textures caused by overlap has greatly affected the selection and adjustment of the detection algorithm in the subsequent anchor frame, and the robustness requirements for the prokaryotic detection algorithm have increased, increasing the difficulty of prokaryotic detection. Therefore, how to design the prokaryotic detection architecture for the inconsistency of the best focus of the prokaryotic data and the occlusion and overlap problems of the prokaryotic data is undoubtedly a huge challenge.

[0055] To address this challenge, the present invention proposes Figure 3 A prokaryotic target detection method based on multi-focal length embryo image fusion based on double-layer optimization is shown, comprising the following steps:

[0056] Step S1: selecting a pair of images with complementary focal lengths from the multi-focal length embryo images to form a complementary image pair, wherein the corresponding position of the blurred focus area in the first image of the complementary image pair is a clearly focused area in the second image, and the corresponding position of the blurred focus area in the second image is a clearly focused area in the first image, and a number of complementary image pairs are selected to form a complementary image set;

[0057] Step S2: Establish a two-layer optimization model with the inner layer minimizing the mean square error of the image fusion module as the goal and the outer layer minimizing the bounding box regression loss of the object detection module as the goal. The optimal solution of the inner layer is used as the input of the outer layer, and the optimization result of the outer layer is fed back to the inner layer to optimize the weight parameters of the image fusion module.

[0058] Step S3: inputting a plurality of complementary image pairs into an optimized image fusion module to obtain a plurality of composite images, and inputting the plurality of composite images into the optimized image fusion module again to obtain a fused image;

[0059] Step S4: input the fused image into the target detection module to obtain the pronucleus detection result of the multi-focal length embryo image.

[0060] The present invention aims to improve the extraction of specific feature information of overlapping problems and the inconsistency of the optimal focus of prokaryotic data, and introduces the specific solution in detail in the model building stage. The following is the implementation process of this embodiment:

[0061] Data collection and preprocessing stage The present invention collected 20,000 images from reproductive centers, all of which were obtained in a time-difference incubator and carefully labeled by multiple professional embryo experts. In order to ensure the accuracy of the annotation, an expert voting method was adopted to select the label with the most votes as the final label of the image. Considering the uncertainty of the embryo in the time-difference incubator within the image field of view, the present invention utilized rotation and flipping in image processing technology to expand the image data. This processing not only enhances the recognition ability of embryonic cells, but also improves the accuracy of the subsequent prokaryotic detection algorithm.

[0062] Finally, the dataset is divided into training set, validation set and test set in a ratio of 8:1:1. This division ensures that the model fully utilizes the data for learning during the training process and is fully evaluated on the validation set and test set. Through this dataset division, the performance and generalization ability of the model can be effectively evaluated.

[0063] 2. Model training phase

[0064] (1) Model construction

[0065] In this construction of the embryo image prokaryotic detection network model, in order to ensure the operability of the detection, the present invention uses Yolov5 as the basic framework for target detection. The Yolov5 framework has the advantages of high efficiency, simplicity and lightness, flexibility, and open source community support, which improves the operability of prokaryotic target detection.

[0066] However, since the optimal focal lengths of different prokaryotic targets are inconsistent in the prokaryotic detection target data, the prokaryotic target data is partially clear and partially blurred. The present invention first needs to perform multi-focal image fusion on the initial data and send the fusion features to the detection system. Therefore, the detection network this time is a detection network that combines image fusion with prokaryotic detection.

[0067] Different from the previous target detection, this prokaryotic target detection is a combination of image fusion and prokaryotic detection. Previously, when designing how to achieve better detection results, only the target detection algorithm was often considered. Now, the prokaryotic detection algorithm and the image fusion algorithm need to be considered at the same time. From this analysis, the present invention believes that the core of this prokaryotic target detection is how to retain more information that is conducive to prokaryotic detection during the fusion of multi-focal length embryo images, and how to modify the parameters of the detection model and the fusion model according to the results of the training, so that the detection model and the fusion model can promote and optimize each other, forming a double-layer optimization, so that the prokaryotic target problem can be better solved and better detection can be achieved.

[0068] In addition, it can be seen from the prokaryotic data that in embryonic cells, due to abnormal cell division, cell differentiation and development, the prokaryotic overlap and occlusion problems occur, which will cause the edge, texture and other detailed information of the prokaryotic target to be lost, making it difficult for the prokaryotic detection algorithm to determine the boundary of the prokaryotic target. Therefore, when fusing multi-focal length images, the present invention needs to design the overlap and occlusion problems in terms of information extraction and loss function construction. However, the existing network architecture that combines image fusion with prokaryotic detection cannot achieve a good detection effect. The main reason is that the existing network architecture that combines image fusion with prokaryotic detection has the following difficulties: (1) The double-layer optimization between the image fusion module and the target detection module is not considered, and (2) There is a loss of edge and texture information during image fusion.

[0069] (1) Dual-layer optimized image fusion and prokaryotic detection

[0070] The general process of target detection is to label the data, and then input it into the target detection model for training. However, for this prokaryotic target detection, since the optimal focal lengths of different prokaryotic targets in the given data set are inconsistent, the prokaryotic target data has the problem of local clarity and local blur. Therefore, directly inputting the original image data into the target detection model cannot obtain better detection results. Therefore, for this prokaryotic detection, a new image fusion module is added. The main function of the image fusion module is to synthesize multiple images with different focal lengths into a higher-definition image. By fusing images at different focal lengths, the lack of focus range in each image can be compensated, resulting in a clearer image with richer details. For image fusion, it is generally necessary to fuse images in different frames at the same time. However, by performing the following operations on the prokaryotic target: Figure 1 The analysis of the prokaryotic data shown in the figure shows that, for the same moment, among the eleven photos, there is no complementary relationship between them in terms of clarity. Usually there are only two or one focused photos. Some images are seriously out of focus, and these severely out of focus prokaryotic images have no effect on multi-focus image fusion. Therefore, this fusion first manually selected the data set, deleted the out-of-focus images, and directly input the single image with the highest clarity into the target detection model without image fusion. Two images with complementary focal lengths were selected from the remaining images as complementary image pairs. In a pair of images with complementary focal lengths, the corresponding position of the blurred focus area in the first image of the complementary image pair is the focused area in the second image, and the corresponding position of the blurred focus area in the second image is the focused area in the first image. Select a number of complementary image pairs to form a complementary image set, and perform image fusion on each complementary image pair.

[0071] A good prokaryotic detection network architecture should be able to accurately identify targets in the image, including targets of various sizes, shapes and postures, and accurately locate their positions. For this prokaryotic detection, it is necessary to first fuse multi-focal length images, and then perform prokaryotic detection after fusion. Therefore, in order to achieve a good prokaryotic detection effect, theoretically, we can start from optimizing the image fusion module and the target detection module to achieve the best prokaryotic detection effect. This method directly optimizes the two core modules of image fusion and target detection, so that the performance of each module can be significantly improved. However, only optimizing the image fusion module and the target detection module may ignore the inherent connection and mutual influence between them. In practical applications, the output of the image fusion module will directly affect the performance of the target detection module, and the performance of the target detection module will also return to affect the performance of the image fusion module. In addition, there are problems of overlap and occlusion in the prokaryotic detection target this time. In this complex scene, it is impossible to achieve a good detection effect by simply optimizing the image fusion module and the target detection module separately. Therefore, it is necessary to make the two modules cooperate and collaborate more closely to deal with complex scene problems such as overlap and occlusion in the prokaryotic detection target this time.

[0072] From the above analysis, we can know that in this prokaryotic target detection task, there is a special requirement: it is necessary to fuse multi-focal length images before detection to obtain richer target information, and the image fusion module and the target detection module are closely related and influence each other. The output quality of the image fusion module directly affects the performance of the target detection module, and the performance of the target detection module in turn affects the effect of the image fusion module. Although optimizing these two modules separately has its advantages, ignoring the inherent connection and mutual influence between them may result in the overall performance not being optimal. In order to solve this problem, the present invention introduces the concept of a two-layer optimization model, which considers these two modules simultaneously in a unified architecture, and optimizes their parameters and strategies to achieve optimal performance for the entire system. The expression of the two-layer optimization model is:

[0073] ;

[0074] In the formula, Represents the bounding box regression loss function of the object detection module, which is used to measure the difference between the predicted bounding box and the actual bounding box; are the parameters of the target detection module; is the optimal solution for the inner optimization model, that is, when using The mean square error function can minimize the difference between the fused image and the original image. value; is the mean square error function of the image fusion module; Represents the parameters of the image fusion module, and its value will affect the fusion effect; Represents the output function of the target detection module, and its value is initialized to 1.

[0075] As can be seen from the formula, the present invention introduces an ingenious two-layer optimization mechanism to improve the overall performance of the prokaryotic detection system. At the outer optimization level, the present invention is committed to minimizing the boundary regression loss function in target detection. , which is responsible for evaluating the error between the predicted bounding box and the true bounding box. By continuously reducing this error, the positioning accuracy of the prokaryotic target bounding box can be significantly improved, thereby enhancing the accuracy of prokaryotic detection. This optimization process is affected by the parameters of the target detection module. is also affected by the constraints in the image fusion process. The formula of the boundary regression loss function is as follows:

[0076]

[0077] In the formula, is the number of input images; , Respectively i The horizontal and vertical coordinates of the center point of the predicted bounding box; , Respectively i The width and height of the predicted bounding box, , Respectively i The horizontal and vertical coordinates of the center point of the real bounding box; , Respectively i The width and height of the ground-truth bounding box; for The weight value of the position, the initial value is 1, according to the target detection result function The purpose of generation is to guide the detection of prokaryotic images.

[0078] At the inner optimization level, the focus turns to the quality of image fusion. The goal of this invention is to minimize the mean square error function , ensuring that the fused image is both clear and rich in detail information. This optimization is also affected by the parameters in the image fusion module is affected by the prokaryotic test results. Specifically, in the results generated by the prokaryotic detection, if the confidence of a certain area is very high, the target detection module will give a higher weight to the area. This process is achieved by continuously updating a certain initial value of 1. Function to achieve, It can further guide the fusion process of prokaryotic images and contribute to improving the performance of prokaryotic detection. The formula of the mean square error function is as follows:

[0079] ;

[0080] ;

[0081] In the formula, The image generated for fusion; is the input image; W , H is the width and height of the image; is the pixel coordinate; For the image at pixel points The pixel value of Pixel The weight value, the initial value is 1, according to the result of target detection The purpose of function generation is to guide image fusion.

[0082] There is a mutually reinforcing relationship between the improvement of image fusion quality and the improvement of target detection accuracy. Better image fusion helps to achieve more accurate target detection, and more accurate target detection can in turn guide more efficient image fusion. This complementary optimization strategy has jointly created a prokaryotic target detection system with excellent performance. The specific two-layer optimization diagram is as follows Figure 4 As shown in the figure, the prokaryotic target detection process based on the double-layer optimization model can be intuitively understood from the double-layer optimization diagram. This diagram not only clearly shows the interactive relationship between the inner and outer layer optimizations, but also incorporates the constraints in various methods, such as the boundary regression function in prokaryotic detection is directly affected by the prokaryotic feature extraction method. This design can more deeply understand the core innovation of this invention: there is a close mutually reinforcing relationship between the improvement of image fusion quality and the improvement of target detection accuracy.

[0083] Specifically, by optimizing the image fusion technology, higher quality fused images can be obtained. These images are not only clearer but also have richer detail information. These high-quality images provide more reliable input for the target detection module, thereby significantly improving the detection accuracy. In turn, with the improvement of detection accuracy, the target detection module can more accurately identify the key information in the image, which can further guide the image fusion process to ensure that the fused image better meets the detection requirements. This complementary optimization strategy not only improves the overall performance of the system, but also makes the prokaryotic target detection system more intelligent and efficient. Through continuous iteration and optimization, the system can gradually adapt to various complex scenes and achieve more accurate and reliable target detection. Among them Figure 5The figure below is a comparison of the prokaryotic detection results before and after the addition of the double-layer optimization model. The figure in the middle is the prokaryotic detection result before the addition of the double-layer optimization model. S represents the detected prokaryotic. Comparing the two result figures, it can be seen that the prokaryotic detection results before the addition of the double-layer optimization model had missed detections. After the addition of the double-layer optimization model, the target detection module not only has a greatly improved detection confidence, but can also detect overlapping targets well.

[0084] (2) Edge and texture information is lost in image fusion

[0085] Edges are the boundaries between targets and backgrounds, and between targets in an image. Edge information can provide key features of the target object, such as shape, direction, and texture. By extracting edge information, important features in the prokaryotic target can be highlighted to achieve the purpose of improving the accuracy of prokaryotic detection. However, the existing image fusion algorithms overemphasize the overall results and average information when processing images, resulting in the loss of edges and details during the fusion process. This leads to problems such as loss of edge and detail information and lack of attention to texture information in existing image fusion. More importantly, due to the overlap and occlusion of prokaryotic detection targets, the edge of the prokaryotic target may become blurred or partially hidden. Therefore, retaining its edge information in the prokaryotic image fusion stage is crucial for accurately locating the prokaryotic target. Therefore, how to retain more edge and texture detail information in the existing multi-focus image fusion has an important impact on improving the performance of the prokaryotic detection model.

[0086] The present invention starts from retaining more edge and texture detail information in the fusion of multi-focal length embryo images, and solves the problems of edge information loss and overlapping of prokaryotic detection targets in existing image fusion algorithms. First of all, it can be known from basic knowledge that the gradient operator is usually used to calculate the gradient of an image or feature map, that is, the rate of change of pixel intensity or eigenvalue. Gradient information is very useful for detecting edge, contour and other information. High-frequency convolution kernels are sensitive to rapid changes and details in images, and can effectively extract and retain edge and texture details. Therefore, in order to extract edge information, the present invention proposes a gradient residual module GDM (Gradient Residual Module). In the original image fusion module, a gradient residual module GDM (Gradient Residual Module) is added to better extract edge and texture information. Its main architecture design is as follows:

[0087] First, two pairs of size The image is input into the Res2Net backbone network, and two feature maps of the same size are extracted, which are represented as , , then and The features are taken as input and transmitted to the CBR module. The CBR module consists of a set of It consists of convolution, regularization and activation function operations. The specific architecture is as follows Figure 6 As shown. For the feature map and The noise contained in is purified:

[0088] (2)

[0089] in: is the convolution operation; is the regularization operation; Indicates the activation function operation.

[0090] Will and After the feature image in is processed by formula (2), the feature images after noise purification are obtained. and , and then the result and Sent to the gradient residual module GDM to generate feature information and , to retain edge information. The gradient residual module GDM formula is shown in the figure below:

[0091] ;

[0092] In the formula, is the convolution operation; is the regularization operation; Indicates the activation function operation.

[0093] Will and After the feature images in are processed, the feature images after noise purification are obtained. and , and then the result and Sent to the gradient residual module GDM to generate feature information and , to preserve edge information:

[0094] ;

[0095] In the formula, The convolution kernel size is 1, and its purpose is to extract detail information; Includes a convolution layer Con3x3 and an activation function LRelu; It is a gradient operator, which contains high-frequency convolution kernels and can better extract edge information.

[0096] Finally, the generated feature information and Stitching is performed, and then image reconstruction is performed to generate a fused pronuclear image.

[0097] The specific structure of the gradient residual module is as follows Figure 7 As shown in the figure, it contains high-frequency convolution kernels and sobel gradient operators, where gradient operator is a gradient operator, which is usually used to calculate the gradient of an image or feature map, that is, the rate of change of pixel intensity or feature value. Gradient information is very useful for detecting edge and contour information, so it is used to further improve the retention of edge details and texture information in the process of prokaryotic image fusion. It provides greater guarantee for solving the overlapping and occlusion problems in prokaryotic detection in the future. As can be seen from the figure, the gradient residual module applies residual learning, dense connection and gradient residual. The addition of residual learning is to help the network retain more details such as the edges of prokaryotic targets and improve the stability of training. Dense connection is to allow each layer to access the feature information of all previous layers, thereby enhancing the algorithm's ability to describe fine-grained spatial details. Among them, the gradient residual is the core idea of ​​the GDM module. The gradient residual calculates the gradient differences between adjacent layers and passes these differences as additional feature information to subsequent layers, which can further improve the sensitivity to detail information.

[0098] The present invention also innovates the loss function by adding content loss to the loss function. The content loss includes mean square loss and gradient loss, aiming to ensure that the fused image achieves the best in terms of detail retention and clarity.

[0099] When introducing the design ideas of the loss function, it is necessary to consider the core goal of the image fusion module, which is to merge clear areas at different focal lengths into one image. Due to the overlap and occlusion of the original kernel targets, it is particularly important to retain the detail information in the source image and enhance the extraction of edge information. The mean squared error (MSE) calculates the difference between each pixel between the fused image and the source image, squares these differences and averages them to obtain a global difference measure. This method is very sensitive to capturing small changes in the image, including differences in brightness, contrast, and structural information. For target detection tasks, MSE can ensure that the fused image retains similar detail information to the source image on a global scale. In the original kernel image fusion process, the mean squared error can effectively reflect the difference between the fused image and the source image, thereby guiding the optimization of the fusion algorithm and the improvement of the original kernel detection performance.

[0100] After calculating the mean square error of the two images in the complementary image pair using the above formula, take the average of the two mean square errors as the mean square error of the composite image:

[0101] .

[0102] However, relying solely on MSE is not enough to fully preserve edge and texture information. Therefore, the gradient loss is introduced, which focuses on the structural changes in local areas of the image, especially edges and textures. Gradient loss can more accurately capture and preserve these important structural information in the original kernel by calculating the difference between the fused image and the source image in the gradient domain. By introducing the gradient loss function and calculating the difference between the fused image and the source image at the gradient level, it is possible to directly measure the degree to which the fused image retains detail information such as edges and textures, thereby guiding the model to better preserve the gradient information of the source image during the fusion process. By optimizing the gradient loss function, the model can learn how to generate a fused image with richer details and higher clarity, so that the fusion result is not only more natural in overall visual effect, but also better consistent with the source image in details such as edges and textures. The expression designed for the gradient loss is:

[0103] ;

[0104] ;

[0105] Where W and H are the width and height of the input image respectively; Indicates the image at pixel points The pixel gradient value of To fuse the images, is the input image, ; represents L2 regularization.

[0106] Traditional image fusion algorithms overemphasize overall results and average information when processing images, resulting in the loss of edges and details during the fusion process. To address this problem, the present invention first proposes a gradient residual module GDM, which can more effectively capture high-frequency information in the image, namely, edge and texture details, mainly by introducing high-frequency convolution kernels and Sobel gradient operators. High-frequency information is crucial for prokaryotic image fusion because it directly determines the clarity and detail performance of the fused prokaryotic image. In addition, the present invention also adds gradient loss and mean square loss. For gradient loss, it guides the model to retain more edge texture information during the fusion process by calculating the difference in gradient between the fused image and the source image. When the gradient loss of the model decreases during the optimization process, it tends to generate a fused image with similar edge and texture information to the source image. For mean square loss, in multi-focus image fusion, it can ensure that the fused prokaryotic image is consistent with the source image at the pixel level, helping the model to optimize the fusion result globally, so that the fused prokaryotic image is closer to the source image as a whole.

[0107] in Figure 8 The results of prokaryotic detection before and after adding GDM and content loss are shown. It can be seen that after adding GDM and content loss, due to the enhanced extraction of edge, texture and other detail information, the target detection system can distinguish the overlapping parts well, achieve good detection results, and be more robust. Compared with the case without adding GDM and content loss, due to the lack of extraction of edge and texture information, the detection algorithm cannot detect the position of prokaryotic images with low edge, texture and other information discrimination, resulting in missed detection.

[0108] The network structure of the overall prokaryotic target detection network model MBPD is as follows Fig. 9 As shown. The weights generated by the target detection module Compared with the above two-level optimization model Associated. The main process is to adjust the weight of the corresponding area according to the detection results, and update it to the image fusion module and the target detection module. As can be seen from the figure, for the input data, it is first necessary to label it to make a label file, and then select the input raw data and input it into the image fusion module. In the process of image fusion, the gradient residual module GDB is added to extract more edge, texture and other detail information. In view of the coverage and occlusion problems of the prokaryotic detection target, the present invention also designs content loss, wherein the content loss includes gradient loss and intensity loss. The fused image is then sent to the target detection module to detect the prokaryotic target in the image.

[0109] In order to optimize the image fusion module and the target detection module to each other and improve the detection accuracy, the present invention constructs a bidirectional fusion of the image fusion module and the target detection module, thereby achieving better image fusion and more accurate target detection. The overall detection effect is as follows Fig.10 As shown, it can be seen that for prokaryotic targets with different overlaps and occlusions, good detection effects can be achieved through the extraction of edge and texture detail information and the image multi-focus fusion module and the double-layer optimization of the target detection system. The confidence level is greatly improved and the expected detection effect is achieved.

[0110] (2) Model training

[0111] The set training set and validation set were input into the Yolov5 network model for training. The hardware used was NVIDIA GeForce RTX 4090 with 32G memory. The number of training rounds was set to 1000 times. The number of samples selected for one training was 32. The type was identified as smooth endoplasmic reticulum. The actual training time was 5 hours, and finally the configuration file required for smooth endoplasmic reticulum detection was obtained.

[0112] In the image detection stage, the trained model is used to detect the test set embryo images, and the prediction results of the test set are evaluated. The embryo images to be identified are input into the prokaryotic detection system model, and the embryonic cell prokaryotic target detection system model is used to achieve detection.

[0113] Based on the above research, the present invention provides a prokaryotic target detection method based on double-layer optimized multi-focal length embryo image fusion, comprising the following steps:

[0114] S1 collects embryo images containing pronuclei, constructs embryo image datasets, and performs data preprocessing;

[0115] S2 screens the prokaryotic data set, selects prokaryotic images with high image quality, and constructs a new data set;

[0116] S3 builds the MBPD network model for pronuclear target detection in embryo images;

[0117] S4 uses the embryo image data set constructed in step S2 to train the embryo image prokaryotic target detection network model to obtain the model and its parameters;

[0118] S5 uses the trained model to detect the embryo images in the test set and completes the test set prediction and result evaluation;

[0119] S6 inputs the embryo image to be identified into the embryo image smooth surface prokaryotic target detection network model of step S4, and detects it using the embryo cell prokaryotic target detection algorithm.

[0120] The present invention overcomes the problem that the model is difficult to identify the pronuclear target of embryonic cells from two perspectives:

[0121] 1. In order to improve the accuracy of prokaryotic detection and enable the image fusion module and the target detection module to influence and optimize each other, a two-layer optimization model is proposed. Through iteration and feedback mechanism, close coordination between the prokaryotic detection system and the image fusion module and the target detection module is achieved, so that the entire prokaryotic detection system has a higher accuracy.

[0122] 2. The present invention adds a gradient residual module and content loss in the image fusion module, which not only improves the extraction of edge details and texture information, but also further improves the quality of image fusion, lays the foundation for subsequent prokaryotic detection, and greatly solves the overlapping and occlusion problems existing in prokaryotic detection.

[0123] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. Only the preferred embodiments of the present invention are expressed. The description is more specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. As long as there is no contradiction in the combination of these technical features, they should be considered as the scope recorded in this specification.

[0124] It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these modifications and improvements all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A prokaryotic target detection method based on multi-focal length embryo image fusion based on double-layer optimization, characterized in that: The following steps are involved: Step S1: selecting a pair of images with complementary focal lengths from the multi-focal length embryo image to form a complementary image pair, wherein the corresponding position of the blurred focus area in the first image of the complementary image pair is a clearly focused area in the second image, and the corresponding position of the blurred focus area in the second image is a clearly focused area in the first image, and selecting a plurality of complementary image pairs to form a complementary image set; Step S2: Establish a two-layer optimization model with the inner layer minimizing the mean square error of the image fusion module as the goal and the outer layer minimizing the bounding box regression loss of the object detection module as the goal. The optimal solution of the inner layer is used as the input of the outer layer, and the optimization result of the outer layer is fed back to the inner layer to optimize the weight parameters of the image fusion module. The expression of the two-layer optimization model is: Where, L reg (·,·) represents the bounding box regression loss function of the object detection module; θ d is the parameter of the target detection module; f* is the optimal solution of the inner optimization model; L mse (·,·) is the mean square error function of the image fusion module; f represents the parameters of the image fusion module; w(·) represents the output function of the target detection module; The expression of the bounding box regression loss function is: Where N is the number of input images; x i ,y i are the horizontal and vertical coordinates of the center point of the i-th predicted bounding box; w i 、h i are the width and height of the i-th predicted bounding box, respectively. are the horizontal and vertical coordinates of the center point of the i-th true bounding box respectively; are the width and height of the i-th ground-truth bounding box respectively; is (x i ,y i )’s weight value; The expression of the mean square error function is: ΔI n (i,j)=I n (i,j)-I F (i,j); In the formula, I F The image generated by fusion; I n is the input image; W and H are the width and height of the image; (i, j) is the pixel coordinate; I(i, j) is the pixel value of the image at the pixel point (i, j); w i,j is the weight value of pixel (i, j); Step S3: inputting the plurality of complementary image pairs into an optimized image fusion module to obtain a plurality of composite images, and inputting the plurality of composite images into the optimized image fusion module again to obtain a fused image; Step S4: inputting the fused image into a target detection module to obtain a pronucleus detection result of a multi-focal length embryo image.

2. The prokaryotic target detection method based on multi-focal length embryo image fusion of double-layer optimization according to claim 1, characterized in that: Step S1 pre-processes the multi-focal length embryo image before selecting the image, including the following steps: evaluating the clarity of the multi-focal length embryo image, deleting images with clarity lower than the set clarity, and directly inputting the image with the highest clarity into the target detection module.

3. The prokaryotic target detection method based on multi-focal length embryo image fusion based on double-layer optimization according to claim 1, characterized in that: Step S3: when the plurality of synthesized images are input into the optimized image fusion module again to obtain a fused image, a gradient residual module is added to the image fusion module, and the gradient residual module combines the image features and gradient information to strengthen the image fusion module to retain edge information, including the following steps: Step S31: randomly selecting two synthetic images from a plurality of synthetic images as input images, extracting features of the input images, and obtaining feature maps C1 and C2; Step S32: removing noise from the feature graph to obtain denoised feature graphs C1' and C'2; Step S33: Use the gradient operator to calculate the gradient information of C1' and C'2 respectively, and combine the gradient information with the features of C1' and C'2 to obtain the fusion feature and Step S34: stitching and Then, image reconstruction is performed to generate a preliminary fused image, and the preliminary fused image is used as a new composite image and added to multiple composite images, and the above steps are repeated until a final fused image is obtained.

4. The prokaryotic target detection method based on multi-focal length embryo image fusion based on double-layer optimization according to claim 3, characterized in that: The expression for removing the noise in the feature map in step S32 is: C i '=Re LU(BN(Conv(C i ))); In the formula, Conv(·) is the convolution operation; BN(·) is the regularization operation; ReLU(·) represents the activation function operation; C i is the feature map, i=1,2.

5. The prokaryotic target detection method based on multi-focal length embryo image fusion based on double-layer optimization according to claim 3, characterized in that: Step S33 combines the gradient information with the features of C1' and C'2 to form the following expression: In the formula, is the fusion feature, i=1,2; Conv1(·) is the convolution with a kernel size of 1; CR(·) includes a convolution layer and an activation function; GrO(·) is the gradient operator.

6. The prokaryotic target detection method based on multi-focal length embryo image fusion based on double-layer optimization according to claim 1, characterized in that: The image fusion module also introduces a gradient loss, and the expression of the gradient loss is: Where, L grad is the gradient loss; W and H are the width and height of the input image respectively; Represents the pixel gradient value of the image at pixel point (i, j); I F is the fused image, I n is the input image; represents L2 regularization.

7. The prokaryotic target detection method based on multi-focal length embryo image fusion based on double-layer optimization according to claim 1, characterized in that: Step S4: Input the fused image into the target detection module to obtain the expression of the pronucleus detection result of the multi-focal length embryo image: I d =D(I f ); Where D represents the detector; Id is the final detection result; If is the fused image.