Cell spindle positioning system and method based on ordinary brightfield or hoffmann imaging

Abstract

The cell spindle positioning system and method based on ordinary bright field or hoffmann imaging belong to the technical field of medical image processing, assisted reproductive microoperation and artificial intelligence assisted positioning technology. The positioning method is realized based on the positioning system, the oocyte region is standardized by ordinary bright field or hoffmann image; the polarization light style image with polarization light imaging characteristics is generated by using the conditional image generation model; the spindle candidate region is obtained by the spindle detection model, and the final spindle position is determined by combining the multi-result fusion positioning method; finally, the spindle positioning result is mapped back to the original ordinary bright field or hoffmann image coordinate system and visualized. The present application can improve the recognizability and positioning reliability of the oocyte spindle under the ordinary bright field image or the hoffmann image without directly obtaining the polarization light microscopic image, reduce the influence of single generation or single detection result fluctuation on the positioning result, and reduce the risk of spindle damage.

Description

Technical Field

[0001] This invention belongs to the fields of medical image processing, assisted reproductive microsurgery, and artificial intelligence-assisted positioning technology. It relates to a cell spindle positioning system and method based on ordinary bright field or Huffman imaging, specifically to an intelligent oocyte spindle positioning system and method based on ordinary bright field or Huffman images. Background Technology

[0002] Intracytoplasmic sperm injection (ICSI) is a commonly used micromanipulation technique in assisted reproductive technology. During ICSI, the operator injects a single sperm into the cytoplasm of the oocyte. Because the oocyte spindle plays a crucial role in chromosome alignment and separation during meiosis, its structural integrity and spatial position significantly influence subsequent fertilization, normal chromosome distribution, and embryonic development. Therefore, damage to the spindle using the injection needle should be avoided as much as possible during ICSI.

[0003] However, in conventional bright-field or Huffman microscopy systems commonly used in hospitals, structures such as the oocyte outline, zona pellucida, cytoplasmic region, and first polar body are relatively easy to observe, while the spindle is usually weakly visible or even invisible, making it difficult for operators to directly determine its true location. Current clinical procedures typically infer the spindle's location indirectly based on the position of the first polar body; however, the spatial relationship between the first polar body and the spindle is not always fixed, and relying solely on the first polar body for judgment can easily lead to errors. Therefore, how to achieve intelligent localization of the oocyte spindle based on conventional bright-field or Huffman imaging has become an urgent problem to be solved.

[0004] To address the problem of determining the location of interest (ROI) within germ cell structures during assisted reproductive technology (ART), Chinese invention patent CN116456892A provides a system and method for determining ROIs based on neural networks. This method acquires images of germ cell structures using an imager and inputs these images into a neural network to generate clinical parameters representing ROIs within the germ cell structure. In specific embodiments, these clinical parameters can be used to indicate the location of the polar body within the oocyte or a suitable location on the zona pellucida of the embryo for assisted manipulation. This patent illustrates the application value of artificial intelligence technology in image location determination in ART. However, this patent primarily focuses on classifying and indicating the locations of polar bodies, zona pellucida manipulation locations, or other ROIs. It does not address the issue of the spindle being weakly visible or even invisible in ordinary bright-field or Huffman images by establishing a conversion mechanism from conventional microscopic images to images where the spindle can be detected. Furthermore, it does not further perform detection, spatial clustering, and multi-result fusion localization of spindle candidate regions. Therefore, it cannot solve the problem of accurate oocyte spindle localization under ordinary bright-field or Huffman images.

[0005] Regarding direct observation of the spindle, US Patent US6879713B1, "Meiotic spindle imaging in oocytes and uses therefor in in vitro fertilization," proposes a method for imaging and analyzing the meiotic spindle in live oocytes. This method utilizes orientation-independent polarized light microscopy to acquire polarized light information from oocytes and determines the presence, position, and morphology of the spindle based on parameters such as birefringence, retardation, and azimuth, thereby enabling oocyte quality evaluation, fertilization potential assessment, or assistance in IVF. This patent allows observation of the live oocyte spindle without fixation and staining, providing a relatively direct basis for spindle location determination. However, this patent still relies on specialized polarized light microscopy equipment, liquid crystal compensators, polarized light image acquisition, and birefringence parameter calculation procedures, placing high demands on equipment and operating conditions. In conventional ICSI microscopy environments lacking polarized light imaging capabilities, this technique cannot utilize ordinary bright-field images or Hoffman images to locate the spindle.

[0006] In summary, existing technologies still have the following shortcomings: First, relying on the position of the first polar body makes it difficult to reliably determine the true position of the spindle; second, although spindle observation methods based on polarized light microscopy can directly display or measure the spindle, they rely on dedicated polarized light microscopy equipment and actual polarized light image acquisition; third, existing AI-assisted position determination methods mainly focus on classification prompts for polar body position, transparent zone operation position, or other regions of interest, and have not yet solved the problem of the spindle being weakly visible and difficult to detect directly under ordinary bright field or Huffman images; fourth, single image judgment or single detection results are easily affected by image quality, focal plane differences, cytoplasmic texture interference, and model inference fluctuations, resulting in insufficient positioning stability.

[0007] Therefore, there is an urgent need for an intelligent oocyte spindle localization system and method that can generate image representations with polarized light imaging features based on conventional bright field or Huffman images, and further realize spindle candidate detection, multi-result fusion localization, and original image coordinate back mapping. Summary of the Invention

[0008] To address the problems of existing technologies, this invention provides a cell spindle localization system and method based on ordinary bright-field or Huffman imaging. This invention standardizes the oocyte region by acquiring easily obtainable ordinary bright-field or Huffman images during routine microscopy; it generates polarized light style images with polarized light imaging characteristics using a pre-trained conditional image generation model; further, it obtains spindle candidate regions through a spindle detection model and determines the final spindle position using a multi-result fusion localization method; finally, it maps the spindle localization results back to the original ordinary bright-field or Huffman image coordinate system and visualizes them. Therefore, this invention improves the identifiability and localization reliability of oocyte spindles under ordinary bright-field or Huffman images without directly acquiring polarized light microscopic images, reduces the impact of fluctuations in single generation or detection results on the localization results, and ensures the spatial correspondence between the localization results and the original microscopic manipulation field of view. This provides a reference for selecting the needle insertion position and direction during ICSI operations, reducing the risk of spindle damage.

[0009] To achieve the above objectives, the technical solution adopted by the present invention is as follows: Cell spindle localization methods based on conventional bright-field or Hoffman imaging include the following steps: Step 1: Obtain raw microscopic images of oocytes using a microscope. and the original polarized light image The original image pairs Further, a set of original image pairs is obtained, consisting of the original image pairs. Specifically: Step 1.1: Use a camera to acquire a normal bright-field image of the oocyte in the microscope's normal bright-field imaging mode or an Huffman image of the oocyte in the microscope's Huffman imaging mode, as the original microscope image. Original microscope image The width is Original microscope image The height is ; Step 1.2: Use a camera in the polarized light imaging mode of a microscope to acquire corresponding images of the same oocyte. During the image capture, ensure the oocyte's position and orientation are completely consistent with those in Step 1.1. The captured image is called the original polarized light image. Original polarized light image Compared with the original microscope image Composition of original image pairs .

[0010] (1) in, This is the number of the oocyte sample among all samples.

[0011] Original image pairs The set of images is called the original image pair set. .

[0012] Step 2, process the original image pairs. Image preprocessing is performed to obtain a standard image pair set. Specifically: Step 2.1, for each original image pair On the original microscope image The coordinates of the geometric center point of the oocyte are manually marked.

[0013] Step 2.2, using the geometric center coordinates of the oocyte as the center The original microscope image is cropped and scaled, and then the image pixel values ​​are normalized to obtain a standard input image. .

[0014] (2) in, This represents the spatial transformation determined by the coordinates of the oocyte's geometric center point, the cropping size, and the scaling ratio, as well as the numerical transformation based on pixel normalization. Indicates spatial transformation parameters; Step 2.3, process the original image pair The original polarized light image in With the corresponding Using the geometric center coordinates of the oocyte as the center, the original microscope image is cropped and scaled at the same resolution as in step 2.2, and then the image pixel values ​​are normalized to obtain a standard polarized light image. .

[0015] (3) Step 2.4, for the standard polarized light image The region containing the spindle is intensified to obtain a standard intensified polarized light image. The region containing the spindle in the standard polarized light image was manually marked. Let the number of pixels in a standard polarized light image be... The original pixel value The enhanced pixel value The enhanced standard polarized light image is represented as: (4) in, Indicates the enhancement value of the spindle region; For standard image pairs, by and composition; Standard image pairs The set of these is called the standard image pair set. .

[0016] Step 3: Divide the standard image pair set obtained in Step 2 into a first training set, a first validation set, and a first test set.

[0017] Step 4: Construct a spindle detection model, including constructing a conditional image generation model. Generate one or more corresponding polarization style images based on the standard input image; construct an object detection model. Detect spindle-shaped regions in the generated polarized light style images; construct a voting model. The final spindle region is determined from one or more detected spindle regions. Specifically: Step 4.1, Construct a conditional image generation model These include image-to-image conversion models based on conditional generative adversarial networks, conditional image generation models based on encoder-decoder networks or autoencoder networks, conditional image generation models based on diffusion probability models or latent space diffusion models, conditional image generation models based on Transformer networks, and conditional image generation models composed of convolutional neural networks, Transformer networks, or a combination of both.

[0018] The constructed conditional image generation model uses a standard input image. The input is a polarized light style image, and the output is a polarized light style image. .

[0019] (5) in, This represents the conditional image generation model that has been trained. Indicates the first A generation condition, inference parameter, or random perturbation parameter.

[0020] By changing the generation conditions, inference parameters, or random perturbation parameters Obtain a set of polarized light style images .

[0021] (6) in, Indicates the first Zhang polarized light style image, This indicates the number of polarized light style images.

[0022] Step 4.2, Construct the object detection model The spindle detection model is selected from YOLO series object detection models, other convolutional neural networks suitable for small object detection, Transformer detection networks, or medical image object detection models.

[0023] Set up a target detection model For the input polarized light style image The output of the spindle detection model is expressed as: (7) (8) in, Indicates the first One candidate rectangle; This indicates the detection confidence level corresponding to the candidate bounding box; This indicates the number of candidate rectangles output. In order to be in The first detected The x-coordinate of the top-left vertex of each candidate rectangle; In order to be in The first detected The ordinate of the top-left vertex of each candidate rectangle; In order to be in The first detected The x-coordinate of the bottom right vertex of each candidate rectangle; In order to be in The first detected The ordinate of the top right corner vertex of each candidate rectangle; Step 4.3, Construct the voting model A set of candidate bounding boxes detected from multiple images of the same egg cell. An integrated rectangle was obtained through a voting process. Specifically: Step 4.3.1: Obtain the set of valid candidate rectangles.

[0024] The polarized light style image set obtained in step 4.1 Each polarized style image is input into the target detection model obtained in step 4.2. The corresponding candidate bounding boxes and detection confidence scores are obtained.

[0025] For each polarized light style image Target detection model Output at most one candidate bounding box and its corresponding detection confidence. All candidate bounding boxes constitute the candidate bounding box set. ; (9) in, For the first One candidate rectangle; The detection confidence level corresponding to the candidate bounding box; This represents the number of candidate rectangles to be output.

[0026] The prediction results with a detection confidence level not lower than the preset detection confidence threshold are used to form a set of valid candidate bounding boxes: (10) in, This represents the set of valid candidate rectangles; The confidence threshold for detection is set to a range of 0.6–0.8.

[0027] Step 4.3.2: Calculate the coordinates of the geometric center point of each candidate rectangle in the set of valid candidate rectangles.

[0028] For the set of valid candidate rectangles The first in Candidate rectangles : (11) in, For the first The x-coordinate of the top-left vertex of each candidate rectangle; For the first The ordinate of the top-left vertex of each candidate rectangle; For the first The x-coordinate of the bottom right vertex of each candidate rectangle; For the first The ordinate of the top right corner vertex of each candidate rectangle; Geometric center coordinates of the candidate rectangle Represented as: (12) Furthermore, the center point coordinates are normalized to the range of zero to one to obtain the normalized center point coordinates. : (13) in, and These represent the width and height of the polarized light style image, respectively; The coordinates of the geometric center point of the candidate rectangle The x-coordinate; The coordinates of the geometric center point of the candidate rectangle The ordinate.

[0029] Step 4.3.3: Perform spatial clustering and confidence fusion on the geometric center points of the candidate rectangles.

[0030] Based on the spatial distance between the geometric center points of the candidate rectangles, spatial clustering is performed on the candidate regions in the spindle-shaped candidate region set. Spatial clustering methods include DBSCAN clustering, threshold distance clustering, or other spatial clustering methods.

[0031] Suppose that R candidate clusters are obtained after clustering, denoted as: (14) in, This represents the number of candidate clusters obtained through spatial clustering methods; Indicates the first A candidate cluster is a set of indices corresponding to all candidate rectangles in the candidate cluster; Indicates the first candidate cluster; Indicates the second candidate cluster; Indicates the first There are 10 candidate clusters.

[0032] For the r-th candidate cluster ,calculate The sum of confidence scores of all candidate bounding boxes This is called the candidate cluster confidence score. (15) in, No. The detection confidence level corresponding to each candidate bounding box; The candidate cluster with the highest sum of candidate cluster confidence and that meets the preset fusion confidence threshold is determined as the target cluster. : (16) The target cluster must satisfy: (17) in, This indicates the preset fusion confidence threshold, ranging from 1.5 to 2.0.

[0033] When no target cluster meets the preset fusion confidence threshold requirement according to step 4.3.3, the voting model outputs the fusion rectangle. Empty, meaning no spindle was detected. If there is a target cluster that meets the preset fusion confidence threshold, continue to step 4.3.4.

[0034] Step 4.3.4: Calculate the integrated bounding box based on the bounding boxes within the target cluster.

[0035] For target clustering Calculate the confidence-weighted average center point of each candidate bounding box within the target cluster. : (18) in, For target cluster The Middle The center point of each candidate rectangle; Subsequently, within the target cluster, the true candidate bounding box whose center point is closest to the weighted average center point is selected as the final spindle-shaped positioning box. : (19) (20) in, This represents the integrated rectangle output by the voting model.

[0036] The detection confidence threshold Preset fusion confidence threshold These are the hyperparameters of the voting model.

[0037] Step 5, Model Training. Based on the conditional image generation model obtained in Step 4. Target detection model Voting Model The training set and validation set obtained in step 3 are used to train the spindle-shaped conditional image generation model. Spindle body detection model Furthermore, the hyperparameters of the voting model were tuned to obtain the optimal hyperparameters. Specifically: Step 5.1, based on the conditional image generation model The spindle-shaped conditional image generation model is obtained by training on the first training set and the first validation set. : Standard input images from the first training set Input the conditional image generation model from step 4.1 A polarized light style image was obtained. Enhanced polarized light images from the first training set (standard enhancement). As a supervised objective, the parameters of the conditional image generation model are iteratively updated to enable the model to learn the conditional mapping relationship for generating polarized style images from ordinary bright-field images or Huffman images, thereby obtaining the spindle-shaped conditional image generation model. And it is verified using the first verification set; specifically: Step 5.1.1: Construct the spindle region weighted loss.

[0038] This invention introduces a weighted loss function based on the spindle region to improve the spindle conditional image generation model. The ability to focus on the spindle structure region.

[0039] Construct a region weight map based on the region where the spindle is located in the standard polarized light image. It is represented as: (twenty one) in, This represents the loss weight of the spindle region, and ; Set pixel points The generation error at that location is The weighted loss of the spindle region It can be represented as: (twenty two) in, Represents a standard intensified polarized light image All pixel regions in; Represents pixels in a spindle-style image The corresponding value; Represents pixels in a standard polarized light image The corresponding value; Step 5.1.2: Verify the trained spindle-shaped conditional image generation model. .

[0040] The performance of the conditional image generation model during training was evaluated using the validation set of the spindle dataset. Based on the structural consistency, image quality, spindle region representation, and subsequent spindle detection performance between the generated polarized light style images and standard enhanced polarized light images, model parameters that meet the preset performance conditions were selected, resulting in the trained conditional image generation model for spindles. .

[0041] The preset performance conditions are as follows: the structural similarity index (SSIM) between the generated image and the standard enhanced polarized light image is not lower than a first preset threshold, and the peak signal-to-noise ratio (PSNR) is not lower than a second preset threshold, thereby ensuring that the generated image meets the requirements in terms of structural information preservation and image quality. The first preset threshold is 0.70–0.90, preferably 0.80; the second preset threshold is 20–40 dB, preferably 30 dB.

[0042] Step 5.2: Train the spindle-shaped target detection model. Specifically: Step 5.2.1: Construct the spindle detection dataset They were divided into a second training set, a second validation set, and a second test set.

[0043] For each standard image pair in the first training set Standard input image Input the spindle-shaped conditional image generation model trained in step 5.1. Inference is performed by changing the generation conditions, inference parameters, or random perturbation parameters. This generates multiple polarized light style images.

[0044] Let the first training set be... The standard input image corresponding to each sample is: , No. The generation conditions are: The corresponding polarization style image Represented as: (twenty three) in, This is the completed conditional image generation model. For the j-th generation condition, inference parameter, or random perturbation parameter; Step 5.2.2: Obtain the spindle-shaped rectangular annotation frame.

[0045] Manually annotate the corresponding rectangular annotation box of the spindle in the polarized light style image obtained in step 5.2.1.

[0046] Let the e-th spindle detection dataset be... The corresponding spindle-shaped rectangular bounding box in the image is Then the rectangular annotation box is represented as: (twenty four) in, This indicates the coordinates of the top left corner of the rectangular annotation box of the spindle shape. This indicates the coordinates of the lower right corner of the rectangular label box for the spindle shape.

[0047] Furthermore, the spindle-shaped rectangular annotation box is also represented by the coordinates of the geometric center point of the spindle, the width of the rectangle, and the height of the rectangle: (25) in, This indicates the coordinates of the geometric center point of the rectangular annotation box of the spindle shape. and These represent the width and height of the rectangular annotation box, respectively.

[0048] Step 5.2.3: Train the spindle-shaped target detection model .

[0049] Polarized light style image Input object detection model The corresponding spindle-shaped rectangular label box As supervisory information, the parameters of the target detection model are iteratively updated to enable the model to learn the correspondence between the spindle-shaped region features and the position of the spindle-shaped bounding box in the polarized light style image, thereby obtaining the spindle-shaped target detection model. .

[0050] The training loss of the spindle-shaped target detection model includes one or more of the following: bounding box regression loss, class loss, and detection confidence loss. The total training loss is expressed as: (26) in, This represents the regression loss between the predicted bounding box and the bounding box. This indicates the predicted loss based on the spindle type. This represents the confidence loss for predicting the bounding box target; , and These represent the weighting coefficients of the corresponding loss terms. These are the weighting coefficients of the regression loss between the rectangular bounding box and the spindle-shaped rectangular annotation box. The weighting coefficients for predicting the loss based on the spindle shape category. The weighting coefficients are used to predict the confidence loss of the bounding box target.

[0051] Step 5.2.4: Verify the trained spindle detection model.

[0052] The spindle detection model during training was evaluated using the second validation set. Based on the accuracy of the spindle candidate bounding box position, detection confidence, and spindle region detection performance, model parameters that meet the preset performance conditions were selected, resulting in the trained spindle target detection model. .

[0053] The preset performance conditions are as follows: in the second validation set, the intersection-union ratio (IoU) between the spindle detection results and the actual labeled regions is not less than 0.5, preferably not less than 0.6; the mean accuracy (mAP@0.5) of the detection model is not less than 0.80, preferably not less than 0.85; the detection precision is not less than 0.85, the recall is not less than 0.70, and the confidence level of the detection results is not less than 0.25.

[0054] Step 5.3: Adjust the hyperparameters of the voting model to obtain the optimal hyperparameters for the voting model.

[0055] Step 5.3.1: Input the first validation set into the spindle conditional image generation model obtained in Step 5.1. By changing the generation conditions, inference parameters, or random perturbation parameters Multiple polarized light style images are generated to obtain a set of polarized light style images. Polarized light style image collection From its subsets composition.

[0056] (27) (28) in, For the first verification set The image is under the condition The polarized light style image obtained by inference from the spindle-shaped conditional image generation model. The number of standard image pairs in the first validation set. The number of types of conditions when generating the model for the spindle conditional image.

[0057] Step 5.3.2, generate all polarized light style images. Input the spindle target detection model obtained in step 5.2 The detection process is performed to obtain candidate bounding boxes for each polarized style image. This results in a set of detected polarized style images. , From its subsets composition.

[0058] (29) (30) in, Polarized light style image The corresponding candidate rectangle, This represents the detection confidence level corresponding to the candidate detection box.

[0059] Step 5.3.3: Using the detected set of polarized light style images All subsets Get the set of candidate rectangles .

[0060] (31) Set of candidate rectangles Input the voting model obtained in step 4.3 The integrated rectangular frame is obtained. .

[0061] Step 5.3.4: Obtain the truth rectangle of the first validation set sample.

[0062] The first verification set A standard enhanced polarized light image Input the spindle target detection model obtained in step 5.2 Inference is performed to obtain the corresponding truth rectangle. .

[0063] Step 5.3.5: The hyperparameters of the grid traversal voting model are used to obtain multiple sets of candidate hyperparameter combinations. The integrated bounding box of each standard enhanced polarized light image in the first validation set is then calculated. With truth rectangle The Euclidean distance between the center points is the hyperparameter of the optimal voting model among all candidate hyperparameter combinations, and the one with the smallest average center point distance is the hyperparameter of the optimal voting model.

[0064] Step 6: Based on the image acquisition and preprocessing methods in Steps 1.1 and 2.2, the spindle conditional image generation model obtained in Step 5 is used. Spindle-shaped target detection model Voting Model Optimal hyperparameters were determined to construct a complete spindle detection system. Specifically: Step 6.1: After completing model training and determining the optimal model parameters, save and deploy the obtained spindle-shaped object detection model. In subsequent practical applications, there is no need to retrain the model for each detection task; the already trained spindle-shaped conditional image generation model can be directly called. Spindle-shaped target detection model Voting Model Perform inference on the input image.

[0065] When the detection environment, sample type, or application requirements change, new labeled data is collected, and the existing spindle conditional image generation model is used. Spindle-shaped target detection model Voting Model The parameters are transferred or incrementally fine-tuned to improve the spindle-shaped conditional image generation model. Spindle-shaped target detection model Voting Model Adapt to the new data distribution. During fine-tuning, only the new data is needed to modify the spindle conditional image generation model. Spindle-shaped target detection model Voting Model Optimizing and updating the parameters can improve the spindle conditional image generation model. Spindle-shaped target detection model Voting Model Improve the detection accuracy in specific scenarios without retraining the model from scratch, thereby enhancing the conditional image generation model for spindle bodies. Spindle-shaped target detection model Voting Model Efficiency and flexibility in practical applications.

[0066] Step 6.2: Obtain the original microscopic image of the oocyte to be tested based on Step 1.1. ,Will Input 2.2 to get ,Will enter Obtain a set of polarized light style images subset of ,Will Input spindle target detection model The resulting set of polarized light style images subset of Finally, the voting model Output integrated rectangle .

[0067] in, for The x-coordinate of the top-left vertex; for The ordinate of the top-left vertex; for The x-coordinate of the bottom right vertex; for The y-coordinate of the bottom right vertex; Step 6.3, based on the integrated rectangular frame obtained in step 6.2 The original integrated rectangular frame representing the position of the spindle of the oocyte under test on the original microscope image was calculated. .

[0068] (32) Finally, the spindle position detection box mapped to the original microscope image coordinate system is obtained. , is used to indicate the spatial position of the spindle of the oocyte under test in the original microscope image.

[0069] A cell spindle localization system based on ordinary bright-field or Huffman imaging is used to implement the aforementioned cell spindle localization method. The cell spindle localization system includes an image acquisition module, an image preprocessing module, a polarized light style image generation module, a spindle candidate detection module, a multi-result fusion localization module, and a coordinate mapping module; specifically: The image acquisition module is used to acquire raw microscopic images of the oocytes to be detected and transmit these raw microscopic images to the image preprocessing module. The raw microscopic images are either ordinary bright-field images or Hoffman images, and include one or more structural information from the oocyte outline, zona pellucida, cytoplasmic region, and first polar body. The images acquired by the image acquisition module can be obtained by a microscopic imaging device or by the user selecting existing images through a local software interface and inputting them into the system.

[0070] The image preprocessing module is connected to the image acquisition module and is used to extract the oocyte center point, crop, scale, and normalize the pixels of the original microscope image to obtain a standard input image that meets the model input requirements. The image preprocessing module is also used to record the height and width of the original microscope image, the coordinates of the upper left corner vertex of the cropped area, and the scaling ratio to obtain spatial transformation parameters.

[0071] The polarized light style image generation module is connected to the image preprocessing module and is used to input the standard input image into a pre-trained spindle conditional image generation model to generate a polarized light style image with polarized light imaging characteristics. The polarized light style image is used to enhance or express spindle-related information that is weakly visible or not directly observable in ordinary bright-field or Huffman images. The polarized light style image generation module is also used to generate one or more polarized light style images based on the same standard input image under different generation conditions, inference parameters, or random perturbation parameters, resulting in a polarized light style image set.

[0072] The spindle candidate detection module is connected to the polarized light style image generation module. It is used to input each polarized light style image in the polarized light style image set into the pre-trained spindle target detection model, and to perform target detection on the spindle region in the polarized light style image to obtain a set of candidate rectangles.

[0073] The multi-result fusion localization module is connected to the spindle candidate detection module and is used to perform spatial clustering and confidence fusion on multiple candidate rectangles in the candidate rectangle set to determine the integrated rectangle as the reference position of the spindle.

[0074] The coordinate mapping module is connected to both the image preprocessing module and the multi-result fusion localization module. Based on the spatial transformation parameters recorded by the image preprocessing module, it maps the integrated rectangular frame representing the spindle reference position obtained by the multi-result fusion localization module from the standard input image coordinate system to the original microscope image coordinate system, thus obtaining the spindle localization result in the original ordinary bright-field or Huffman image. The coordinate mapping module is also used to overlay and display the predicted spindle frame on the original ordinary bright-field or Huffman image, thereby providing the operator with spindle localization prompts to assist in ICSI microscopy operations.

[0075] Compared with the prior art, the beneficial effects of the present invention are as follows: First, this invention can reduce the dependence of oocyte spindle localization on real-time polarized light microscopy equipment.

[0076] This invention uses readily available bright-field or Huffman images, easily obtained during conventional microscopy, as input. A pre-trained spindle conditional image generation model generates polarized-style images with polarized light imaging characteristics, enhancing the representation of weakly visible or indirectly observable spindle-related information in the bright-field or Huffman images. Therefore, in actual localization, it is unnecessary to acquire real polarized light images every time, providing a more easily identifiable image representation for subsequent spindle detection, thereby reducing reliance on dedicated polarized light microscopy equipment and real-time polarized light imaging procedures.

[0077] Secondly, the present invention can improve the detectability of oocyte spindles under ordinary bright field or Hoffman images.

[0078] While ordinary bright-field or Huffman images can display structures such as the oocyte outline, zona pellucida, cytoplasmic region, and first polar body, the spindle apparatus typically exhibits weak features within these images, making direct detection impossible. This invention utilizes a polarized light style image generation module to convert the implicit spindle-related features in ordinary bright-field or Huffman images into polarized light style image features that are more readily recognized by detection models. A spindle candidate detection module then performs target detection on the polarized light style image, thereby improving the detection capability of spindle candidate regions.

[0079] Third, the present invention can improve the stability and reliability of the spindle positioning results.

[0080] Since the results of a single image generation or spindle detection may be affected by factors such as image quality, focal plane differences, cellular texture interference, and model inference fluctuations, this invention does not directly use the single detection result as the final output. Instead, it generates one or more polarized light style images based on the same standard input image, obtaining multiple candidate bounding boxes representing the spindle's location. Subsequently, a multi-result fusion localization module performs spatial clustering and confidence fusion on the candidate bounding boxes representing the spindle's location, selecting the target cluster that meets the clustering confidence requirements to determine the final spindle location. This reduces the impact of fluctuations in a single candidate result on the final localization result and improves the stability of the localization result.

[0081] Fourth, the present invention can directly feed back the spindle positioning results to the original image interface familiar to the operator.

[0082] This invention records the original microscopic image size, cropping region location, and scaling ratio during image preprocessing. After obtaining the final position of the target spindle, it maps this position back from the standard input image coordinate system to the original ordinary bright-field or Huffman image coordinate system via a coordinate mapping module. Therefore, the system can overlay and display the predicted spindle shape on the original ordinary bright-field or Huffman image, allowing the operator to intuitively view the possible location of the spindle within a conventional microscopic image interface.

[0083] Fifth, this invention can provide auxiliary reference for spindle avoidance in ICSI micromanipulation.

[0084] The spindle positioning results output by this invention can be used to indicate the spatial relationship between the spindle and the oocyte, the first polar body, and the injection direction, providing auxiliary reference for operators in oocyte rotation, injection path selection, and spindle avoidance. Compared with relying solely on experience to determine the position of the first polar body, this invention provides more direct spindle position indications, helping to reduce operational risks caused by errors in spindle position judgment.

[0085] Sixth, the present invention can form a complete intelligent positioning process from image acquisition, feature enhancement, candidate detection, fusion localization to result display.

[0086] Existing polarized light imaging schemes mainly rely on the acquisition of real polarized light images. Existing ICSI manipulator improvement schemes mainly address hardware issues such as needle fixation, injection needle, needle replacement, and needle adjustment. However, this invention addresses the problem of weak spindle visibility in ordinary bright-field or Huffman images by sequentially setting up technical steps such as image preprocessing, polarized light style image generation, spindle candidate detection, multi-result fusion localization, and coordinate mapping display. This can transform the implicit spindle information in ordinary microscopic images into detectable, fusionable, remappingable, and displayable localization results, thereby improving the practicality of the system in conventional ICSI micromanipulation scenarios.

[0087] In summary, this invention extracts and standardizes the oocyte region from ordinary bright-field or Huffman images, generates polarized light style images with polarized light imaging characteristics using a conditional image generation model, and locates the spindle in each polarized light style image using a pre-trained spindle detection model. Furthermore, for multiple detection results obtained from the same oocyte under different generation conditions, spatial clustering and confidence fusion are used to determine a reliable final spindle position. Finally, the positioning results are mapped back to the original microscopic image coordinate system and visualized. Therefore, without relying on real-time polarized light microscopy imaging, the stability, interpretability, and clinical applicability of oocyte spindle positioning results from ordinary bright-field or Huffman images can be improved, providing a spindle avoidance auxiliary reference for ICSI microscopy. Attached Figure Description

[0088] Figure 1 This is a flowchart of the cell spindle localization method of the present invention; Figure 2 A schematic diagram illustrating the reinforcement and weighted loss of the spindle region; Figure 3 A schematic diagram for constructing a spindle detection model; Figure 4 This is a schematic diagram of the voting model process; Figure 5 This is a schematic diagram of the cell spindle positioning system of the present invention. Detailed Implementation

[0089] The present invention will be further described below with reference to specific implementation examples.

[0090] Example: The flowchart of the cell spindle localization method based on ordinary bright field or Hoffman imaging is as follows: Figure 1 As shown, it includes the following steps: Step 1: Obtain raw microscopic images of oocytes using a microscope. and the original polarized light image The original image pairs Further, a set of original image pairs is obtained, consisting of the original image pairs. Specifically: Step 1.1: Use a camera to acquire a normal bright-field image of the oocyte in the microscope's normal bright-field imaging mode or an Huffman image of the oocyte in the microscope's Huffman imaging mode, as the original microscope image. .

[0091] Step 1.2: Use a camera in the polarized light imaging mode of a microscope to acquire corresponding images of the same oocyte. During the image capture, ensure the oocyte's position and orientation are completely consistent with those in Step 1.1. The captured image is called the original polarized light image. Original polarized light image Compared with the original microscope image obtained in step 1.1 Composition of original image pairs .

[0092] (1) in, This is the number of the oocyte sample among all samples.

[0093] Original image pairs The set of images is called the original image pair set. .

[0094] In this embodiment, the original microscope image and the original polarized light image The resolution of all images is 1392×1024, which is the original microscope image. width Original microscope image height .

[0095] Step 2, for the original image pair set from Step 1 Image preprocessing is performed to obtain a standard image pair set. Specifically: Step 2.1, for each original image pair On the original microscope image The coordinates of the geometric center point of the oocyte are manually marked.

[0096] Step 2.2: Using the geometric center coordinates of the oocyte as the center, crop the image to a resolution of 512×512, and further normalize the pixel values ​​to obtain the standard input image. .

[0097] (2) in, This represents the spatial transformation determined by the coordinates of the oocyte's geometric center point, the cropping size, and the scaling ratio, as well as the numerical transformation based on pixel normalization. The spatial transformation parameters are represented in this embodiment. The spatial transformation parameters include a resolution size of 512×512, a scaling ratio of 1:1, and manually labeled coordinates of the geometric center point of the oocyte.

[0098] Step 2.3, process the original image pair The original polarized light image in With the corresponding Using the geometric center coordinates of the oocyte as the center, the original microscope image is cropped and scaled to the same resolution as in step 2.2, and then the image pixel values ​​are normalized to obtain a standard polarized light image. .

[0099] (3) Step 2.4, as follows Figure 2 As shown, for standard polarized light images The region containing the spindle is intensified to obtain a standard intensified polarized light image. The region containing the spindle in the standard polarized light image was manually marked. Let the number of pixels in a standard polarized light image be... The original pixel value The enhanced pixel value The enhanced standard polarized light image is represented as: (4) in, Indicates the enhancement value of the spindle region; and The resulting image pairs are standard image pairs. Standard image pairs The set of these is called the standard image pair set. In this embodiment, the spindle region enhancement value .

[0100] Step 3: Divide the standard image pair set obtained in Step 2 into the first training set, the first validation set, and the first test set in a ratio of 7:1.5:1.5.

[0101] Step 4, construct the spindle detection model, such as Figure 3 As shown, this includes constructing a conditional image generation model. Generate one or more corresponding polarization style images based on the standard input image; construct an object detection model. Detect spindle-shaped regions in the generated polarized light style images; construct a voting model. The final spindle region is determined from one or more detected spindle regions. Specifically: Step 4.1, Construct a conditional image generation model In this embodiment, the conditional image generation model is the baseline model of ControlNet. The constructed conditional image generation model uses a standard input image. The input is a polarized light style image, and the output is a polarized light style image. .

[0102] (5) in, This is the completed conditional image generation model. For the first In this embodiment, the generation conditions, inference parameters, or random perturbation parameters are... The diffusion step size is the inference parameter of the Diffusion model, and the set of diffusion step sizes used is as follows. .

[0103] By changing the generation conditions, inference parameters, or random perturbation parameters Obtain a set of polarized light style images .

[0104] (6) in, Indicates the first Zhang polarized light style image, In this embodiment, the number of polarized style images is indicated. .

[0105] Step 4.2, Construct the object detection model .

[0106] In this embodiment, the target detection model is the YOLOv11 model.

[0107] Set up a target detection model For the input polarized light style image The model output is represented as: (7) (8) in, Indicates the first One candidate rectangle; This indicates the detection confidence level corresponding to the candidate bounding box; This indicates the number of candidate rectangles output. In order to be in The first detected The x-coordinate of the top-left vertex of each candidate rectangle; In order to be in The first detected The ordinate of the top-left vertex of each candidate rectangle; In order to be in The first detected The x-coordinate of the bottom right vertex of each candidate rectangle; In order to be in The first detected The ordinate of the top right corner vertex of each candidate rectangle; Step 4.3, as follows Figure 4 As shown, a voting model is constructed. The set of candidate bounding boxes detected from multiple images of the same egg cell. An integrated rectangle was obtained through a voting process. Specifically: Step 4.3.1: Obtain the set of valid candidate rectangles.

[0108] The polarized light style image set obtained in step 4.1 Each polarized style image is input into the target detection model obtained in step 4.2. The corresponding candidate bounding boxes and detection confidence scores are obtained.

[0109] For each polarized light style image Target detection model Output at most one candidate bounding box and its corresponding detection confidence. All candidate bounding boxes constitute the candidate bounding box set. , (9) in, Indicates the first One candidate rectangle; This indicates the detection confidence level corresponding to the candidate bounding box; This indicates the number of candidate rectangles output.

[0110] The prediction results with a detection confidence level not lower than the preset detection confidence threshold are used to form a set of valid candidate bounding boxes: (10) in, This represents the set of valid candidate rectangles; To determine the confidence threshold, this embodiment uses a value of 0.7.

[0111] Step 4.3.2: Calculate the coordinates of the geometric center point of each candidate rectangle in the set of valid candidate rectangles.

[0112] For the set of valid candidate rectangles The first in Candidate rectangles : (11) in, For the first The x-coordinate of the top-left vertex of each candidate rectangle; For the first The ordinate of the top-left vertex of each candidate rectangle; For the first The x-coordinate of the bottom right vertex of each candidate rectangle; For the first The ordinate of the top right corner vertex of each candidate rectangle; Geometric center coordinates of the candidate rectangle Represented as: (12) Furthermore, the center point coordinates are normalized to the range of zero to one to obtain the normalized center point coordinates. : (13) in, and These represent the width and height of the polarized light style image, respectively; The coordinates of the geometric center point of the candidate rectangle The x-coordinate; The coordinates of the geometric center point of the candidate rectangle The ordinate. In this embodiment, , .

[0113] Step 4.3.3: Perform spatial clustering and confidence fusion on the geometric center points of the candidate rectangles.

[0114] Based on the spatial distance between the geometric center points of the candidate rectangles, spatial clustering is performed on the candidate regions in the spindle-shaped candidate region set. In this embodiment, the DBSCAN clustering method is used for spatial clustering.

[0115] Suppose that R candidate clusters are obtained after clustering, denoted as: (14) in, This represents the number of candidate clusters obtained through spatial clustering methods; Indicates the first A candidate cluster is a set of indices corresponding to all candidate rectangles in the candidate cluster; Indicates the first candidate cluster; Indicates the second candidate cluster; Indicates the first There are 10 candidate clusters.

[0116] For the r-th candidate cluster ,calculate The sum of confidence scores of all candidate bounding boxes This is called the candidate cluster confidence score. (15) in, No. The detection confidence level corresponding to each candidate bounding box; The candidate cluster with the highest sum of candidate cluster confidence and that meets the preset fusion confidence threshold is determined as the target cluster. : (16) The target cluster must satisfy: (17) in, This represents the preset fusion confidence threshold, which is set to a value in this embodiment. .

[0117] When no target cluster meets the preset fusion confidence threshold requirement according to step 4.3.3, the voting model outputs the fusion rectangle. If the result is empty, it means no spindle was detected; if there is a target cluster that meets the preset fusion confidence threshold, continue to step 4.3.4.

[0118] Step 4.3.4: Calculate the integrated bounding box based on the bounding boxes within the target cluster.

[0119] For target clustering Calculate the confidence-weighted average center point of each candidate bounding box within the target cluster. : (18) in, For target cluster The Middle The center point of each candidate rectangle; Subsequently, within the target cluster, the true candidate bounding box whose center point is closest to the weighted average center point is selected as the final spindle-shaped positioning box. : (19) (20) in, This represents the integrated rectangle output by the voting model.

[0120] Step 5, Model Training. Based on the conditional image generation model obtained in Step 4. Target detection model Voting Model The training set and validation set obtained in step 3 are used to train the spindle-shaped conditional image generation model. Spindle body detection model Furthermore, the hyperparameters of the voting model were tuned to obtain the optimal hyperparameters. Specifically: Step 5.1: Input the standard input images from the first training set. Input the conditional image generation model from step 4.1 A polarized light style image was obtained. Enhanced polarized light images from the first training set (standard enhancement). As a supervised objective, the parameters of the conditional image generation model are iteratively updated to enable the model to learn the conditional mapping relationship for generating polarized style images from ordinary bright-field images or Huffman images, thereby obtaining the spindle-shaped conditional image generation model. .

[0121] In this embodiment, the ControlNet model is trained for 200 epochs on an RTX 5090 GPU, and saved every 20 epochs. Standard image pair set. There are 1113 standard image pairs, meaning there are 779 standard image pairs in the first training set.

[0122] Step 5.1.1, as follows Figure 2 As shown, a spindle-shaped region-weighted loss is constructed.

[0123] Since the spindle structure typically occupies only a local area in a standard polarized light image, while the background area occupies a large proportion, directly using the traditional pixel-level mean square error loss function for training makes the model optimization process susceptible to the influence of a large number of background pixels. This causes the generated model to focus more on the overall image structure and ignore the detailed features of the spindle region. Therefore, this invention introduces a weighted loss function based on the spindle region to improve the spindle conditional image generation model. The ability to focus on the spindle structure region.

[0124] Construct a region weight map based on the region where the spindle is located in the standard polarized light image. It is represented as: (twenty one) in, In this embodiment, the loss weights for the spindle region are represented. .

[0125] Set pixel points The generation error at that location is The weighted loss of the spindle region Represented as: (twenty two) in, Represents a standard intensified polarized light image All pixel regions in; Represents pixels in a spindle-style image The corresponding value; Represents pixels in a standard polarized light image The corresponding value; In this embodiment, pixel The generation error at that location is The calculation method is the same as that of the consistent ControlNet model.

[0126] Step 5.1.2: Verify the trained spindle-shaped conditional image generation model. .

[0127] The performance of the conditional image generation model during training was evaluated using the validation set of the spindle dataset. Based on the structural consistency, image quality, spindle region representation, and subsequent spindle detection performance between the generated polarized light style images and standard enhanced polarized light images, model parameters that meet the preset performance conditions were selected, resulting in the trained conditional image generation model for spindles. .

[0128] The preset performance conditions are: the structural similarity index (SSIM) between the generated image and the standard enhanced polarized light image is not lower than a first preset threshold, and the peak signal-to-noise ratio (PSNR) is not lower than a second preset threshold, thereby ensuring that the generated image meets the requirements in terms of structural information preservation and image quality. In this embodiment, the first preset threshold is 0.80; the second preset threshold is 30dB.

[0129] Step 5.2: Train the spindle-shaped target detection model. .

[0130] Step 5.2.1: Construct the spindle detection dataset Spindle detection dataset The dataset will be divided into a second training set, a second validation set, and a second test set according to a certain ratio.

[0131] For each standard image pair in the first training set Standard input image Input the spindle-shaped conditional image generation model trained in step 5.1. Inference is performed by changing the generation conditions, inference parameters, or random perturbation parameters. Multiple polarized light style images are generated. In this embodiment, the spindle detection dataset... There are 6232 images in total, consisting of 779 images in each of the 8 different diffusion step sizes.

[0132] Let the training set be... The standard input image corresponding to each sample is: , No. The generation conditions are: The corresponding polarization style image It can be represented as: (twenty three) in, This represents the conditional image generation model that has been trained. This represents the j-th generation condition, inference parameter, or random perturbation parameter; Step 5.2.2: Obtain the spindle-shaped rectangular annotation frame.

[0133] Manually annotate the corresponding rectangular annotation box of the spindle in the polarized light style image obtained in step 5.2.1.

[0134] Let the e-th spindle detection dataset be... The corresponding spindle-shaped rectangular bounding box in the image is Then the rectangular annotation box is represented as: (twenty four) in, This indicates the coordinates of the top left corner of the rectangular annotation box of the spindle shape. This indicates the coordinates of the lower right corner of the rectangular label box for the spindle shape.

[0135] Furthermore, the spindle-shaped rectangular annotation box is also represented by the coordinates of the geometric center point of the spindle, the width of the rectangle, and the height of the rectangle: (25) in, Here are the coordinates of the geometric center point of the rectangular annotation frame for the spindle shape. and These represent the width and height of the rectangular annotation box, respectively.

[0136] Step 5.2.3: Train the spindle-shaped target detection model .

[0137] Polarized light style image Input object detection model The corresponding spindle-shaped rectangular label box As supervisory information, the parameters of the target detection model are iteratively updated to enable the model to learn the correspondence between the spindle-shaped region features and the position of the spindle rectangle in the polarized light style image, thereby obtaining the spindle target detection model. In this embodiment, the target detection model is the YOLOv11 model.

[0138] Step 5.2.4: Verify the trained spindle detection model.

[0139] The spindle detection model during training was evaluated using the second validation set. Based on the accuracy of the spindle candidate bounding box position, detection confidence, and spindle region detection performance, model parameters that meet the preset performance conditions were selected, resulting in the trained spindle target detection model. In this embodiment, the preset performance conditions are as follows: in the second validation set, the intersection-over-union (IoU) between the spindle detection results and the actual labeled regions is not less than 0.5, the mean precision (mAP@0.5) of the detection model is not less than 0.80, the precision is not less than 0.85, the recall is not less than 0.70, and the confidence level of the detection results is not less than 0.25.

[0140] Step 5.3: Adjust the hyperparameters of the voting model to obtain the optimal hyperparameters for the voting model.

[0141] Step 5.3.1: Input the first validation set into the spindle conditional image generation model obtained in Step 5.1. By changing the generation conditions, inference parameters, or random perturbation parameters Multiple polarized light style images are generated to obtain a set of polarized light style images. Polarized light style image collection Depend on subset of composition.

[0142] (27) (28) in, For the first verification set The image is under the condition The polarized light style image obtained by inference from the spindle-shaped conditional image generation model. The number of standard image pairs in the first validation set Number of types of conditions when generating model inference for spindle conditional images In this embodiment, the inference parameters or random perturbation parameters are the diffusion step size of the inference parameters of the Diffusion model, and the set of diffusion step sizes used is as follows: ,Right now The first validation set contains 167 standard image pairs, namely... .

[0143] Step 5.3.2: Combine all the polarization style images obtained in step 5.3.1. Input the spindle target detection model obtained in step 5.2 The detection process is performed to obtain candidate bounding boxes for each polarized style image. This results in a set of detected polarized style images. , Depend on subset of composition.

[0144] (29) (30) in, Polarized light style image The corresponding candidate rectangle, This represents the detection confidence level corresponding to the candidate detection box.

[0145] Step 5.3.3: Using the detected set of polarized light style images All subsets Get the set of candidate rectangles .

[0146] (31) Set of candidate rectangles Input the voting model obtained in step 4.3 The integrated rectangular frame is obtained. .

[0147] Step 5.3.4: Obtain the truth rectangle of the first validation set sample.

[0148] The first verification set A standard enhanced polarized light image Input the spindle target detection model obtained in step 5.2 Inference is performed to obtain the corresponding truth rectangle. .

[0149] Step 5.3.5: The hyperparameters of the grid traversal voting model are used to obtain multiple sets of candidate hyperparameter combinations. The integrated bounding box of each standard enhanced polarized light image in the first validation set is then calculated. With truth rectangle The Euclidean distance between the center points is the hyperparameter of the optimal voting model among all candidate hyperparameter combinations, and the one with the smallest average center point distance is the hyperparameter of the optimal voting model.

[0150] In this embodiment, the optimal detection confidence threshold is obtained. The optimal preset fusion confidence threshold is obtained. .

[0151] Step 6: Based on the image acquisition and preprocessing methods in Steps 1.1 and 2.2, the spindle conditional image generation model obtained in Step 5 is used. Spindle-shaped target detection model Voting Model Optimal hyperparameters were determined to construct a complete spindle detection system. Specifically: Step 6.1: After completing model training and determining the optimal model parameters, save and deploy the obtained spindle-shaped object detection model. In subsequent practical applications, there is no need to retrain the model for each detection task; the already trained spindle-shaped conditional image generation model can be directly called. Spindle-shaped target detection model Voting Model Perform inference on the input image.

[0152] When the detection environment, sample type, or application requirements change, new labeled data is collected, and a model based on the existing spindle conditional image generation method is used. Spindle-shaped target detection model Voting Model The parameters are transferred or incrementally fine-tuned to improve the spindle-shaped conditional image generation model. Spindle-shaped target detection model Voting Model Adapt to the new data distribution. During fine-tuning, only the new data is needed to modify the spindle conditional image generation model. Spindle-shaped target detection model Voting Model Optimizing and updating the parameters can improve the spindle conditional image generation model. Spindle-shaped target detection model Voting Model Improve the detection accuracy in specific scenarios without retraining the model from scratch, thereby enhancing the conditional image generation model for spindle bodies. Spindle-shaped target detection model Voting Model Efficiency and flexibility in practical applications.

[0153] Step 6.2: Obtain the original microscopic image of the oocyte to be tested based on Step 1.1. ,Will Input 2.2 to get ,Will enter Polarized light style image collection subset of ,Will Input spindle target detection model The resulting set of polarized light style images subset of Finally, the voting model Output integrated rectangle .

[0154] in, for The x-coordinate of the top-left vertex; for The ordinate of the top-left vertex; for The x-coordinate of the bottom right vertex; for The y-coordinate of the bottom right vertex; Step 6.3, as follows Figure 5 As shown, the integrated rectangular frame obtained in step 6.2 The original integrated rectangular frame representing the position of the spindle of the oocyte under test on the original microscope image was calculated. .

[0155] (32) Finally, the spindle position detection box mapped to the original microscope image coordinate system is obtained. , is used to indicate the spatial position of the spindle of the oocyte under test in the original microscope image.

[0156] This embodiment also provides a cell spindle localization system based on ordinary bright field or Huffman imaging, which implements the above-mentioned cell spindle localization method. It includes an image acquisition module, an image preprocessing module, a polarized light style image generation module, a spindle candidate detection module, a multi-result fusion localization module, and a coordinate mapping module; specifically: The image acquisition module is used to acquire raw microscopic images of the oocytes to be detected and transmit these raw microscopic images to the image preprocessing module. The raw microscopic images are either ordinary bright-field images or Hoffman images, and include one or more structural information from the oocyte outline, zona pellucida, cytoplasmic region, and first polar body. The images acquired by the image acquisition module are obtained by a microscopic imaging device.

[0157] The image preprocessing module is connected to the image acquisition module and is used to extract the oocyte center point, crop, scale, and normalize the pixels of the original microscope image to obtain a standard input image that meets the model input requirements. The image preprocessing module is used to record the height and width of the original microscope image, the coordinates of the upper left corner vertex of the cropped area, and the scaling ratio to obtain spatial transformation parameters.

[0158] The polarized light style image generation module is connected to the image preprocessing module and is used to input a standard input image into a pre-trained spindle conditional image generation model to generate a polarized light style image with polarized light imaging characteristics. The polarized light style image is used to enhance or express spindle-related information that is weakly visible or not directly observable in ordinary bright-field or Huffman images. The polarized light style image generation module is also used to generate one or more polarized light style images based on the same standard input image under different generation conditions, inference parameters, or random perturbation parameters, resulting in a polarized light style image set.

[0159] The spindle candidate detection module is connected to the polarized light style image generation module. It is used to input each polarized light style image in the polarized light style image set into the pre-trained spindle target detection model, and to perform target detection on the spindle region in the polarized light style image to obtain a set of candidate rectangles.

[0160] The multi-result fusion localization module is connected to the spindle candidate detection module and is used to perform spatial clustering and confidence fusion on multiple candidate rectangles in the candidate rectangle set to determine the integrated rectangle as the reference position of the spindle.

[0161] The coordinate mapping module is connected to both the image preprocessing module and the multi-result fusion localization module. Based on the spatial transformation parameters recorded by the image preprocessing module, it transforms the integrated rectangular box representing the spindle reference position obtained by the multi-result fusion localization module from the standard input image coordinate system to the original microscope image coordinate system, thus obtaining the spindle localization result in the original ordinary bright-field or Huffman image. The coordinate mapping module also overlays the predicted spindle box onto the original ordinary bright-field or Huffman image, thereby providing the operator with spindle localization prompts to assist in ICSI microscopy operations.

[0162] The embodiments described above are merely illustrative of the implementation methods of the present invention, but should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the protection scope of the present invention.

Claims

1. A method for locating cell spindles based on ordinary bright-field or Huffman imaging, characterized in that, The cell spindle localization method includes the following steps: Step 1: Obtain raw microscopic images of oocytes using a microscope. and the original polarized light image The original image pairs Obtain the original image pair set consisting of the original image pairs. ; Step 2, process the original image pairs. Image preprocessing is performed to obtain a standard image pair set. ; Step 3: Divide the standard image pair set into a first training set, a first validation set, and a first test set; Step 4: Construct a spindle detection model, including constructing a conditional image generation model. Generate one or more corresponding polarization style images based on the standard input image; construct an object detection model. Detect spindle-shaped regions in the generated polarized light style images; construct a voting model. The final spindle region is determined from one or more detected spindle regions; Step 5, Model Training; Based on the conditional image generation model obtained in step 4 Target detection model Voting Model The first training set and the first validation set obtained in step 3 are used to train the spindle-shaped conditional image generation model. Spindle body detection model Furthermore, the hyperparameters of the voting model were tuned to obtain the optimal hyperparameters; Step 6: Based on Steps 1 and 2, and the spindle conditional image generation model obtained in Step 5. Spindle-shaped target detection model Voting Model Optimal hyperparameters were determined to construct a complete spindle detection system.

2. The cell spindle localization method based on ordinary bright-field or Huffman imaging according to claim 1, characterized in that, Step 1 specifically involves: Step 1.1: Use a camera to acquire a normal bright-field image of the oocyte in the microscope's normal bright-field imaging mode or an Huffman image of the oocyte in the microscope's Huffman imaging mode, as the original microscope image. Original microscope image The width of the middle is Original microscope image The height in is ; Step 1.2: Use a camera in the polarized light imaging mode of a microscope to acquire corresponding images of the same oocyte. During the image capture, ensure the oocyte's position and orientation are completely consistent with those in Step 1.

1. The captured image is called the original polarized light image. Original polarized light image Compared with the original microscope image Composition of original image pairs ; （1） in, This is the oocyte sample's identification number within the entire sample set. Original image pairs The set of images is called the original image pair set. .

3. The cell spindle localization method based on ordinary bright-field or Huffman imaging according to claim 2, characterized in that, Step 2 specifically involves: Step 2.1, for each original image pair On the original microscope image Mark the coordinates of the geometric center point of the oocyte; Step 2.2, using the geometric center coordinates of the oocyte as the center The original microscope image is cropped and scaled, and the pixel values ​​are normalized to obtain a standard input image. ; （2） in, This represents the spatial transformation determined by the coordinates of the oocyte's geometric center point, the cropping size, and the scaling ratio, as well as the numerical transformation based on pixel normalization. Indicates spatial transformation parameters; Step 2.3, process the original image pair The original polarized light image in With the corresponding Using the geometric center coordinates of the oocyte as the center, the original microscope image is cropped and scaled to the same resolution as in step 2.2, and the image pixel values ​​are normalized to obtain a standard polarized light image. ; （3） Step 2.4, for the standard polarized light image The region containing the spindle is intensified to obtain a standard intensified polarized light image. Manually label the region where the spindle is located in the standard polarized light image. Let the number of pixels in a standard polarized light image be... The original pixel value The enhanced pixel value The enhanced standard polarized light image is represented as: （4） in, Indicates the enhancement value of the spindle region; For standard image pairs, by and composition; Standard image pairs The set of these is called the standard image pair set. .

4. The cell spindle localization method based on ordinary bright-field or Huffman imaging according to claim 3, characterized in that, Step 4 specifically involves: Step 4.1, Construct a conditional image generation model These include image-to-image conversion models based on conditional generative adversarial networks, conditional image generation models based on encoder-decoder networks or autoencoder networks, conditional image generation models based on diffusion probability models or latent space diffusion models, conditional image generation models based on Transformer networks, and conditional image generation models composed of convolutional neural networks, Transformer networks, or a combination of both. The constructed conditional image generation model uses a standard input image. The input is a polarized light style image, and the output is a polarized light style image. ; （5） in, This represents the conditional image generation model that has been trained. Indicates the first A generation condition, inference parameter, or random perturbation parameter; By changing the generation conditions, inference parameters, or random perturbation parameters Obtain a set of polarized light style images ; （6） in, Indicates the first Zhang polarized light style image, Indicates the number of polarized style images; Step 4.2, Construct the object detection model The spindle detection model is selected from YOLO series target detection models, other convolutional neural networks suitable for small target detection, Transformer detection networks, or medical image target detection models. Set up a target detection model For the input polarized light style image The output of the spindle detection model is expressed as: （7） （8） in, Indicates the first One candidate rectangle; This represents the detection confidence level corresponding to the candidate bounding box; Indicates the number of candidate rectangles output; In order to be in The first detected The x-coordinate of the top-left vertex of each candidate rectangle; In order to be in The first detected The ordinate of the top-left vertex of each candidate rectangle; In order to be in The first detected The x-coordinate of the bottom right vertex of each candidate rectangle; In order to be in The first detected The ordinate of the top right corner vertex of each candidate rectangle; Step 4.3, Construct the voting model A set of candidate bounding boxes detected from multiple images of the same egg cell. An integrated rectangle was obtained through a voting process. .

5. The cell spindle localization method based on ordinary bright-field or Huffman imaging according to claim 4, characterized in that, Step 4.3 specifically involves: Step 4.3.1: Obtain the set of valid candidate rectangles; The polarized light style image set obtained in step 4.1 Each polarized style image is input into the target detection model obtained in step 4.

2. Obtain the corresponding candidate bounding boxes and detection confidence scores; For each polarized light style image Target detection model Output at most one candidate bounding box and its corresponding detection confidence; all candidate bounding boxes constitute the candidate bounding box set. ; （9） in, For the first One candidate rectangle; The detection confidence level corresponding to the candidate bounding box; This represents the number of candidate rectangles to be output. The prediction results with a detection confidence level not lower than the preset detection confidence threshold are used to form a set of valid candidate bounding boxes: （10） in, This represents the set of valid candidate rectangles; The confidence threshold for detection is set in the range of 0.6–0.

8. Step 4.3.2: Calculate the coordinates of the geometric center point of each candidate rectangle in the set of valid candidate rectangles; For the set of valid candidate rectangles The first in Candidate rectangles : （11） in, For the first The x-coordinate of the top-left vertex of each candidate rectangle; For the first The ordinate of the top-left vertex of each candidate rectangle; For the first The x-coordinate of the bottom right vertex of each candidate rectangle; For the first The ordinate of the top right corner vertex of each candidate rectangle; Geometric center coordinates of the candidate rectangle Represented as: （12） Normalize the center point coordinates to the range of zero to one to obtain the normalized center point coordinates. : （13） in, and These represent the width and height of the polarized light style image, respectively; The coordinates of the geometric center point of the candidate rectangle The x-coordinate; The coordinates of the geometric center point of the candidate rectangle The ordinate; Step 4.3.3: Perform spatial clustering and confidence fusion on the geometric center points of the candidate rectangles; Based on the spatial distance between the geometric center points of the candidate rectangles, spatial clustering is performed on the candidate regions in the spindle-shaped candidate region set; let R candidate clusters be obtained after clustering, denoted as: （14） in, This represents the number of candidate clusters obtained through spatial clustering methods; Indicates the first A candidate cluster is a set of indices corresponding to all candidate rectangles in the candidate cluster; Indicates the first candidate cluster; Indicates the second candidate cluster; Indicates the first 1 candidate cluster; For the r-th candidate cluster ,calculate The sum of confidence scores of all candidate bounding boxes This is called the candidate cluster confidence score. （15） in, No. The detection confidence level corresponding to each candidate bounding box; The candidate cluster with the highest sum of candidate cluster confidence and that meets the preset fusion confidence threshold is determined as the target cluster. : （16） The target cluster must satisfy: （17） in, This indicates the preset fusion confidence threshold, ranging from 1.5 to 2.0; When no target cluster meets the preset fusion confidence threshold requirement according to step 4.3.3, the voting model outputs the fusion rectangle. Empty, meaning no spindle was detected. If a target cluster that meets the preset fusion confidence threshold exists, continue with step 4.3.4; Step 4.3.4: Calculate the integrated bounding box based on the bounding boxes within the target cluster; For target clustering Calculate the confidence-weighted average center point of each candidate bounding box within the target cluster. : （18） in, For target cluster The Middle The center point of each candidate rectangle; Within the target cluster, select the true candidate rectangle whose center point is closest to the center point of the weighted average, and use it as the final spindle-shaped positioning box. : （19） （20） in, The ensemble rectangle represents the output of the voting model; The detection confidence threshold Preset fusion confidence threshold These are the hyperparameters of the voting model.

6. The cell spindle localization method based on ordinary bright-field or Huffman imaging according to claim 5, characterized in that, Step 5 specifically involves: Step 5.1, based on the conditional image generation model The spindle-shaped conditional image generation model is obtained by training on the first training set and the first validation set. : Standard input images from the first training set Input the conditional image generation model from step 4.1 A polarized light style image was obtained. Enhanced polarized light images from the first training set (standard enhancement). As a supervised objective, the parameters of the conditional image generation model are iteratively updated to enable the model to learn the conditional mapping relationship for generating polarized style images from ordinary bright-field images or Huffman images, thereby obtaining the spindle-shaped conditional image generation model. And it is verified using the first verification set; specifically: Step 5.1.1: Construct the spindle region weighted loss; We introduce a weighted loss function based on the spindle region, and construct a region weight map based on the region where the spindle is located in the standard polarized light image. It is represented as: （21） in, This represents the loss weight of the spindle region, and ; Set pixel points The generation error at that location is The weighted loss of the spindle region Represented as: （22） in, Represents a standard intensified polarized light image All pixel regions in; Represents pixels in a spindle-style image The corresponding value; Represents pixels in a standard polarized light image The corresponding value; Step 5.1.2: Verify the trained spindle-shaped conditional image generation model. ; The performance of the conditional image generation model during training was evaluated using the validation set of the spindle dataset. Based on the structural consistency, image quality, spindle region representation, and subsequent spindle detection performance between the generated polarized light style images and standard enhanced polarized light images, model parameters that meet the preset performance conditions were selected, resulting in the trained conditional image generation model for spindles. ; Step 5.2: Train the spindle-shaped target detection model. Specifically: Step 5.2.1: Construct the spindle detection dataset They were divided into a second training set, a second validation set, and a second test set. For each standard image pair in the first training set Standard input image Input the spindle-shaped conditional image generation model trained in step 5.

1. Inference is performed by changing the generation conditions, inference parameters, or random perturbation parameters. This generates multiple polarized light style images; Let the first training set be... The standard input image corresponding to each sample is: , No. The generation conditions are: The corresponding polarization style image Represented as: （23） in, This is the completed conditional image generation model. For the j-th generation condition, inference parameter, or random perturbation parameter; Step 5.2.2: Obtain the rectangular annotation frame of the spindle shape; Label the polarized light style image obtained in step 5.2.1 with the rectangular label box corresponding to the spindle shape; Let the e-th spindle detection dataset be... The corresponding spindle-shaped rectangular bounding box in the image is Then the rectangular annotation box is represented as: （24） in, This indicates the coordinates of the top left corner of the rectangular annotation box of the spindle shape. This indicates the coordinates of the lower right corner of the rectangular annotation box of the spindle shape; The spindle-shaped rectangular annotation box also represents the coordinates of the geometric center point of the spindle, the width of the rectangle, and the height of the rectangle: （25） in, This indicates the coordinates of the geometric center point of the rectangular annotation box of the spindle shape. and These represent the width and height of the rectangular annotation box, respectively. Step 5.2.3: Train the spindle-shaped target detection model ; Polarized light style image Input object detection model The corresponding spindle-shaped rectangular label box As supervisory information, the parameters of the target detection model are iteratively updated to enable the model to learn the correspondence between the spindle-shaped region features and the position of the spindle-shaped bounding box in the polarized light style image, thus obtaining the spindle-shaped target detection model. ; The training loss of the spindle-shaped target detection model is: （26） in, This represents the regression loss between the predicted bounding box and the bounding box. This indicates the loss for predicting the spindle type. This represents the confidence loss for predicting the bounding box target; , and These represent the weight coefficients of the corresponding loss terms. These are the weighting coefficients of the regression loss between the rectangular bounding box and the spindle-shaped rectangular annotation box. The weighting coefficients for predicting the loss based on the spindle shape category. Weighting coefficients for predicting the confidence loss of bounding box targets; Step 5.2.4: Verify the trained spindle detection model; The spindle detection model during training was evaluated using the second validation set. Based on the accuracy of the spindle candidate bounding box position, detection confidence, and spindle region detection performance, model parameters that meet the preset performance conditions were selected, resulting in the trained spindle target detection model. ; Step 5.3: Adjust the hyperparameters of the voting model to obtain the optimal hyperparameters of the voting model; Step 5.3.1: Input the first validation set into the spindle conditional image generation model obtained in Step 5.

1. By changing the generation conditions, inference parameters, or random perturbation parameters Multiple polarized light style images are generated to obtain a set of polarized light style images. Polarized light style image collection From its subsets composition; （27） （28） in, For the first verification set The image is under the condition Polarized light style image obtained by inference from the lower spindle body conditional image generation model; The number of standard image pairs in the first validation set; The number of types of conditions when generating the model for inference using the conditional image of the spindle body; Step 5.3.2, generate all polarized light style images. Input the spindle target detection model obtained in step 5.2 The detection process is performed to obtain candidate bounding boxes for each polarized style image; the resulting set of detected polarized style images is then obtained. , From its subsets composition; （29） （30） in, Polarized light style image The corresponding candidate rectangle, The detection confidence level corresponding to the candidate detection box; Step 5.3.3: Using the detected set of polarized light style images All subsets Get the set of candidate rectangles ; （31） Set of candidate rectangles Input the voting model obtained in step 4.3 The integrated rectangular frame is obtained. ; Step 5.3.4: Obtain the truth rectangle of the first validation set sample; The first verification set A standard enhanced polarized light image Input the spindle target detection model obtained in step 5.2 Inference is performed to obtain the corresponding truth rectangle. ; Step 5.3.5: The hyperparameters of the grid traversal voting model are used to obtain multiple sets of candidate hyperparameter combinations. The integrated bounding box of each standard enhanced polarized light image in the first validation set is then calculated. With truth rectangle The Euclidean distance between the center points is the hyperparameter of the optimal voting model among all candidate hyperparameter combinations, and the one with the smallest average center point distance is the hyperparameter of the optimal voting model.

7. The cell spindle localization method based on ordinary bright-field or Huffman imaging according to claim 6, characterized in that, In step 5: In step 5.1.2, the preset performance conditions are: the structural similarity index between the generated image and the standard enhanced polarized light image is not lower than the first preset threshold, and the peak signal-to-noise ratio is not lower than the second preset threshold, ensuring that the generated image meets the requirements in terms of structural information preservation and image quality; wherein, the first preset threshold is 0.70 to 0.90; and the second preset threshold is 20 to 40 dB. In step 5.2.4, the preset performance conditions are as follows: in the second validation set, the intersection-union ratio (IoU) between the spindle detection results and the actual labeled regions is not less than 0.5; the mean precision (mAP@0.5) of the detection model is not less than 0.80; the detection precision is not less than 0.85, the recall is not less than 0.70, and the confidence level of the detection results is not less than 0.

25.

8. The cell spindle localization method based on ordinary bright-field or Huffman imaging according to claim 7, characterized in that, Step 6 specifically involves: Step 6.1: After completing model training and determining the optimal model parameters, save and deploy the obtained spindle target detection model. When the detection environment, sample type, or application requirements change, new labeled data is collected, and a model is generated based on the existing spindle conditional image. Spindle-shaped target detection model Voting Model Parameters can be transferred or incrementally fine-tuned. Step 6.2: Obtain the original microscopic image of the oocyte to be tested based on Step 1.

1. ,Will Input 2.2 to get ,Will enter Obtain a set of polarized light style images subset of ,Will Input spindle target detection model The resulting set of polarized light style images subset of Finally, the voting model Output integrated rectangle ,in, for The x-coordinate of the top-left vertex; for The ordinate of the top-left vertex; for The x-coordinate of the bottom right vertex; for The y-coordinate of the bottom right vertex; Step 6.3, based on the integrated rectangular frame obtained in step 6.2 The original integrated rectangular frame representing the position of the spindle of the oocyte under test on the original microscope image was calculated. ; （32） Finally, the spindle position detection box mapped to the original microscope image coordinate system is obtained. , is used to indicate the spatial position of the spindle of the oocyte under test in the original microscope image.

9. A cell spindle localization system based on ordinary bright-field or Hoffman imaging, characterized in that, The cell spindle localization method according to any one of claims 1-8 is implemented using a cell spindle localization system; the cell spindle localization system includes an image acquisition module, an image preprocessing module, a polarized light style image generation module, a spindle candidate detection module, a multi-result fusion localization module, and a coordinate mapping module.

10. The cell spindle localization system based on ordinary bright-field or Huffman imaging according to claim 9, characterized in that, Specifically, the cell spindle positioning system includes: The image acquisition module is used to acquire the original microscope image of the oocyte to be detected and transmit the original microscope image to the image preprocessing module; The image preprocessing module is connected to the image acquisition module and is used to extract the oocyte center point, crop the size, scale and normalize the pixels of the original microscope image to obtain a standard input image that meets the model input requirements. The polarized light style image generation module is connected to the image preprocessing module and is used to input the standard input image into the pre-trained spindle conditional image generation model to generate polarized light style images with polarized light imaging features, thereby obtaining a polarized light style image set. The spindle candidate detection module is connected to the polarized light style image generation module. It is used to input each polarized light style image in the polarized light style image set into the pre-trained spindle target detection model, and to perform target detection on the spindle region in the polarized light style image to obtain a set of candidate rectangles. The multi-result fusion localization module is connected to the spindle candidate detection module and is used to perform spatial clustering and confidence fusion on multiple candidate rectangles in the candidate rectangle set to determine the integrated rectangle as the reference position of the spindle. The coordinate mapping module is connected to the image preprocessing module and the multi-result fusion positioning module, respectively. It is used to obtain the spindle positioning result in the original ordinary bright field or Huffman image by transferring the integrated rectangular box representing the reference position of the spindle obtained by the multi-result fusion positioning module from the standard input image coordinate system to the original microscope image coordinate system, based on the spatial transformation parameters recorded by the image preprocessing module. The coordinate mapping module is also used to overlay and display the spindle prediction box on the original ordinary bright field or Huffman image and output spindle positioning prompts to assist ICSI microscopy operations.

Images