Method, device and readable storage medium of automatic cell detection of fluorescence in situ hybridization image

Abstract

The application provides a kind of fluorescence in situ hybridization image automatic cell detection method, device and its readable storage medium, the method fuses the fluorescence signal classification detection ability of Yolov12 and the fine boundary segmentation advantage of Cellpose, realizes collaborative decision-making through data preparation and labeling, double model targeted training, segmentation prediction, post-processing optimization and mask logic and operation, solves the problems such as cell overlapping segmentation difficulty, fluorescence signal attenuation interference and strong subjectivity of artificial interpretation in existing FISH image analysis. It can significantly improve the segmentation accuracy and edge recognition ability of overlapping cells, realize fully automated efficient processing, provide objective and reliable basis for the diagnosis of related diseases, and has high precision, high efficiency and strong objectivity.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing, and particularly relates to an automatic cell detection method and device for a fluorescence in situ hybridization image and a readable storage medium thereof. BACKGROUND

[0002] Fluorescence in situ hybridization (FISH) technology is an important means of molecular pathological diagnosis. The technology can observe the distribution of fluorescent signals under a fluorescence microscope by specific hybridization of fluorescently labeled nucleic acid probes with target DNA sequences in the cell nucleus, and can accurately determine the gene amplification or deletion (such as HER2 gene amplification), thereby providing an important basis for the typing and treatment guidance of related cancers.

[0003] However, the analysis of the FISH image currently mainly depends on manual interpretation by professionals: the technicians need to identify cells one by one under a microscope and count fluorescent signals, which not only has a huge workload and low efficiency, but also is difficult to meet the rapid processing needs of large-scale clinical samples; and the manual interpretation is affected by subjective experience, fatigue state and other factors, and the interpretation results of different observers or even the same observer at different times may have significant differences, thereby leading to poor diagnostic consistency.

[0004] In addition, the FISH image itself has many technical challenges: on the one hand, cells often form overlapping or adherent cell clusters due to dense distribution, and traditional segmentation methods are difficult to accurately outline the boundaries; on the other hand, the fluorescent signal is easily affected by imaging conditions to appear attenuation and unevenness, which further interferes with cell recognition and signal counting. These problems jointly restrict the automation level and accuracy of FISH diagnosis, and an efficient, accurate and objective automatic cell detection method is urgently needed. SUMMARY

[0005] The present application provides an automatic cell detection method, device and readable storage medium for a fluorescence in situ hybridization image, which aims to solve the problems of large workload, strong subjectivity and low efficiency of the existing FISH image analysis relying on manual interpretation, and the insufficient accuracy of traditional automatic methods under the conditions of difficult cell overlap segmentation and fluorescent signal attenuation interference.

[0006] The core technology of the present application mainly fuses the fluorescent signal detection capability of the Yolov12 model and the fine boundary segmentation advantage of the Cellpose model, constructs a double-model collaborative decision mechanism, and realizes the automatic cell classification and segmentation of the FISH image by using mask logic and operation.

[0007] In a first aspect, the present application provides an automatic cell detection method for a fluorescence in situ hybridization image, which comprises the following steps:

[0008] S1, data preparation: obtain FISH images and perform labeling, divide the cells in the image into a first type of cells that need to be counted for fluorescent spots and a second type of cells that are not analyzed at present, the first type of cells are complete and exist independently in the field of view, uniform in size and consistent in shape, and the second type of cells are cell clusters with incomplete boundaries or adhesion;

[0009] S2, generating a data set: converting the labeling results into a standard format required by the first model, the conversion includes coordinate normalization, class encoding and quality control;

[0010] S3, first model training: loading the pre-training parameters of the first model, parameter fine-tuning based on the data set, the first model is a deep learning model with classification detection capability, the training process uses a combined loss function containing multiple loss functions, and combines data enhancement and regularization strategies;

[0011] S4, second model training: loading the pre-training parameters of the second model, adapting the input channel to adapt to the features of the FISH image, the second model is a deep learning model with fine boundary segmentation capability, the training process uses a combined loss function containing mask loss, boundary loss and regularization loss, and combines targeted data enhancement and dynamic training strategies;

[0012] S5, double model segmentation prediction: using the trained first model and second model to segment the FISH image respectively, outputting the segmentation mask, class identification and confidence of the first model, and the segmentation mask of the second model;

[0013] S6, first model segmentation result optimization: post-processing the segmentation result of the first model, including confidence sorting, mask scaling, morphological operation and overlapping area removal;

[0014] S7, double model collaborative decision: logical AND operation on the segmentation mask of the second model and the optimized first model segmentation mask, determining the cell class according to the number and distribution of connected domains in the operation result, and generating the final detection result.

[0015] Further, in step S1, the labeling adopts a double labeling verification method, and the third party expert interprets the ambiguous samples, and the labeling tool is an image labeling tool.

[0016] Further, in step S2, the quality control includes filtering invalid labeling, checking polygon closure and verifying vertex coordinate range.

[0017] Further, in step S3, the first model is Yolov12 model, the combined loss function includes a weighted combination of Dice loss, Focal loss, Jaccard loss, mutual exclusion loss and soft sampling weighted loss, the data enhancement includes rotation, flip and color space adjustment, and the regularization strategy includes label smoothing, weight decay and random dropout.

[0018] Further, in step S4, the second model is Cellpose model, the mask loss is a joint loss of Dice loss and cross-entropy loss, the boundary loss is a multi-scale matching loss, the regularization loss is a morphological consistency loss, and the targeted data enhancement includes elastic deformation, fluorescence channel separation and incorporation of adversarial samples.

[0019] Further, in step S4, the dynamic training strategy includes preheating of the learning rate, cosine annealing decay and restart mechanism, and early stopping mechanism based on the boundary evaluation index.

[0020] Further, in step S6, the morphological operation includes dilation operation and closing operation on the binary mask, and the overlapping area removal is achieved by logical AND operation.

[0021] Further, in step S7, when the logical AND operation result exists single block connected domain, the class label predicted by the first model is directly used; when there are multiple connected domains, the first model class label corresponding to the connected domain with the most pixel points is used as the result; when there is no connected domain, the second model segmentation mask is expanded to overlap with the first model segmentation mask, and the first model class label corresponding to the overlapping area is used as the result.

[0022] Further, in step S3, the weights of the combined loss function are determined by genetic algorithm optimization, and the average intersection over union of the verification set is used as the fitness index.

[0023] Further, in step S6, the mask scaling adjusts the mask image to the source image size by interpolation method, and the interpolation method includes quadratic spline interpolation.

[0024] In a second aspect, the present application provides an automatic cell detection device for fluorescence in situ hybridization images, comprising:

[0025] A data preparation module acquires FISH images and performs labeling, and divides the cells in the images into first class cells that need to be counted for fluorescent points and second class cells that are not analyzed at present, the first class cells are complete and exist independently, uniform in size and consistent in shape, and the second class cells are incomplete in boundary or exist as cell clusters.

[0026] A data set module converts the labeling results into a standard format required by the first model, and the conversion includes coordinate normalization, class encoding and quality control.

[0027] The first model module loads pre-training parameters of the first model, fine-tunes parameters based on a data set, the first model is a deep learning model with classification detection capability, a combined loss function containing multiple loss functions is used in the training process, and data enhancement and regularization strategies are combined;

[0028] The second model module loads pre-training parameters of the second model, adapts the input channel to adapt to FISH image features, the second model is a deep learning model with fine boundary segmentation capability, a combined loss function containing mask loss, boundary loss and regularization loss is used in the training process, and targeted data enhancement and dynamic training strategies are combined;

[0029] The double model segmentation prediction module uses the trained first model and the second model to segment the FISH image respectively, and outputs the segmentation mask, class label and confidence of the first model, and the segmentation mask of the second model;

[0030] The result optimization module performs post-processing on the segmentation result of the first model, including confidence sorting, mask scaling, morphological operation and overlapping region removal;

[0031] The collaborative decision module performs logical AND operation on the segmentation mask of the second model and the optimized first model segmentation mask, determines the cell class according to the number and distribution of connected domains in the operation result, and generates the final detection result.

[0032] In a third aspect, the present application provides an electronic device comprising a memory and a processor, the memory storing a computer program, and the processor being configured to run the computer program to execute the automatic cell detection method for fluorescence in situ hybridization images described above.

[0033] In a fourth aspect, the present application provides a readable storage medium, the readable storage medium storing a computer program, the computer program comprising program code for controlling a process to execute the process, the process comprising the automatic cell detection method for fluorescence in situ hybridization images described above.

[0034] The main contributions and innovations of the present application are as follows:

[0035] 1. Classification and segmentation accuracy is significantly improved: through double model collaboration (classification detection capability of Yolov12 and fine segmentation advantage of Cellpose) and mask logical AND operation, the problems of cell overlap segmentation difficulty and inaccurate boundary drawing are effectively solved, the edge signal recognition capability is improved, and the processing accuracy of overlapping cells and weak fluorescent signal cells is better than that of traditional methods.

[0036] 2. Processing efficiency is greatly improved: a fully automated processing flow is constructed, and double models are used to realize high-speed parallel operation, which greatly shortens the analysis time while ensuring accuracy, and can meet the efficient processing needs of large-scale samples in clinical practice.

[0037] 3. Diagnosis objectivity and consistency are enhanced: the subjectivity of manual interpretation is avoided, and stable and repeatable analysis results are output through standardized algorithms, providing objective and reliable basis for the diagnosis of related cancers and improving the diagnosis consistency between different institutions and different personnel.

[0038] The details of one or more embodiments of the present application are presented in the following drawings and description to make other features, objects and advantages of the present application more apparent. BRIEF DESCRIPTION OF DRAWINGS

[0039] The drawings described herein are intended to provide further understanding of the present application, and form a part of the present application. The illustrative embodiments of the present application and their description serve to explain the present application, and do not constitute an improper limitation of the present application. In the drawings:

[0040] Figure 1 is a flowchart of an automatic cell detection method for fluorescence in situ hybridization images according to an embodiment of the present application;

[0041] Figure 2 is a Cellpose model segmentation result map according to an embodiment of the present application;

[0042] Figure 3 is a YOLO model segmentation result map according to an embodiment of the present application;

[0043] Figure 4 is a YOLO model overlap elimination and boundary optimization result map according to an embodiment of the present application;

[0044] Figure 5 is a segmentation result map of the Cellpose model combined with the YOLO model according to an embodiment of the present application;

[0045] Figure 6 is a hardware structure schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0046] The exemplary embodiments will be described in detail herein with reference to the accompanying drawings. Unless otherwise indicated, the same numbers on different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all the implementations consistent with one or more embodiments of the present specification. Rather, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of the present specification, as detailed in the appended claims.

[0047] It should be noted that the steps of the corresponding method are not necessarily performed in the order shown and described in the present specification in other embodiments. In some other embodiments, the steps included in the method thereof can be more or less than described in the present specification. In addition, a single step described in the present specification can be divided into multiple steps for description in other embodiments, and multiple steps described in the present specification can be combined into a single step for description in other embodiments.

[0048] The existing FISH image analysis relies on manual interpretation, and has problems of large workload, strong subjectivity and low efficiency. In addition, under the conditions of difficult cell overlap segmentation and interference of fluorescent signal decay, the precision of traditional automatic method is insufficient.

[0049] Based on this, the present application solves the problems existing in the prior art based on the Yolov12 model and the Cellpose model.

[0050] Embodiment one

[0051] The present application aims to propose an automatic cell detection method for fluorescence in situ hybridization images. Specifically, referring to Figure 1 , the method comprises the following steps:

[0052] S1, data preparation: obtaining FISH images and labeling, dividing the cells in the images into first-class cells that need to be counted for fluorescent points and second-class cells that are not analyzed, the first-class cells are complete and exist independently in the field of view, uniform in size and consistent in shape, and the second-class cells are incomplete or have cell clusters that are not suitable for subsequent fluorescent point counting.

[0053] In the present embodiment, labelme 5.4.1 version is used for labeling. The labeling process needs to classify the cells in the image into cell category and other category, wherein the cell category is the cell that needs to be counted for fluorescent points, these cells are complete and exist independently in the field of view, relatively large in the whole image, uniform in size and consistent in shape; the other category is the cell that is not analyzed, these cells are incomplete in the field of view, or have cell clusters that cannot be used for subsequent fluorescent point counting.

[0054] The labeling process adopts the mode of double labeling verification, and if there is ambiguity in the interpretation results of the two persons, a third expert is invited to interpret, and the result of the third expert is taken as the final interpretation result.

[0055] For example, in 500 labeled images, the ratio of cell and other categories is about 1:1, and the samples do not contain different imaging quality and staining intensity, and the cells in the images are clear in the field of view and uniform in staining.

[0056] S2, generating data set: converting the annotation results into the standard format required by the first model, the conversion includes coordinate normalization, class encoding and quality control;

[0057] In this embodiment, the json file annotated by experts using Labelme is converted into the standard txt format required by Yolo model, the specific steps are as follows:

[0058] S2.1, data preparation and input: obtaining the json annotation file generated by Labelme, each file corresponds to a medical image, reading the polygon vertex coordinates and cell class label from the json file, and reading the original image size (width, height) for normalization operation.

[0059] S2.2, coordinate conversion and normalization: convert absolute pixel coordinates to normalized coordinates, and encode the class, where the "cell" class corresponds to the number 0, and the "other" class corresponds to the number 1, the specific vertex coordinate (x, y) conversion process is as follows:

[0060]

[0061]

[0062] Finally, the result is kept to 6 decimal places, reducing the floating point error.

[0063] S2.3, quality control of converted data set: during the process of converting json file into standard txt format required by Yolo model, strict quality control is needed, which includes the following three points:

[0064] Filter invalid annotation: polygons with less than 3 vertices are automatically discarded.

[0065] Polygon closure check: when the first and last point number is greater than 1 pixel, the polygon is automatically closed.

[0066] Vertex verification: ensure that the coordinate values of all points are located in the range of [0, width] and [0, height].

[0067] S3, first model training: loading the pre-training parameters of the first model, parameter fine-tuning based on the data set, the first model is a deep learning model with classification detection ability, the training process uses a combination of loss functions containing multiple loss functions, combined with data enhancement and regularization strategy;

[0068] In this embodiment, the yolov12 model (first model) is first loaded with pre-trained model parameters on a large-scale general dataset, without modifying the specific architecture of the model, and the parameter fine-tuning is performed using the annotation results of the FISH image, and the number of classes of the model is set to 2. In the training, the related training parameters are set, the training rounds (Epoch) are set to 200, the batch size (Batch Size) is set to 96, and the image size (Image Size) is set to 1024, so that the yolov12 model gradually adapts to the features of the FISH image.

[0069] In addition, the following training strategy adjustments are added during the training process:

[0070] S3.1, loss function optimization: to improve the performance of the model, a combination of multiple loss functions is used, including Dice loss, Focal loss, Jaccard loss, mutual exclusion loss and soft sampling weighted loss, which are combined into a custom combination loss, and the genetic algorithm is used to design the weights of the multiple loss functions, and the average intersection over union (mIOU) result of the segmentation result in the validation set is used as the fitness of the genetic algorithm to determine the weights of the multiple loss function combination. The final weights of Dice loss, Focal loss, Jaccard loss, mutual exclusion loss and soft sampling weighted loss are 0.3:0.2:0.2:0.2:0.1.

[0071] wherein the average intersection over union (mIOU) is the average value of the intersection over union (IOU) of all classes, and the calculation formula is as follows:

[0072]

[0073] And the calculation formula of mIOU is as follows:

[0074]

[0075] wherein c is the number of classes; is the intersection over union of the i-th class.

[0076] In the loss function, the Dice loss is used to measure the overlap between the predicted result and the true label, and its expression is as follows:

[0077]

[0078] In the above formula, is the segmentation mask predicted by the model; is the true label; represents the intersection of the prediction and the true label; is a smoothing term, which is generally 1 , used to avoid the case where the denominator is 0.

[0079] In addition, Focal loss is used to solve the problem of class imbalance in classification tasks, and its standard formula is as follows:

[0080]

[0081] In the above formula, represents the predicted probability of the model for the true class, which is used to measure the correctness of sample classification. represents the balance factor, which adjusts the weight of positive and negative samples. represents the adjustment factor, which is used to reduce the weight of easily classified samples, and generally takes the value of 2.

[0082] Jaccard loss is used to measure the degree of overlap between the predicted result and the true label, which is defined as the complement of the Jaccard coefficient, and the calculation process of the Jaccard coefficient is as follows:

[0083]

[0084] In the above formula, represents the segmentation mask predicted by the model, represents the true label, represents the intersection size of the prediction and the true label, represents the union size of the prediction and the true label, represents a small constant, which is used to ensure that the denominator is not zero.

[0085] The calculation formula of Jaccard loss is:

[0086]

[0087] Exclusive loss refers to the loss function that forces the model to assign a unique class label to each pixel, and its calculation form is cross-entropy loss:

[0088]

[0089] where, represents the pixel i in the class c of the true label, represents the predicted probability of pixel i belonging to class c output by Softmax, represents the total number of classes, represents the total number of image pixels.

[0090] Soft sample weighting loss is to dynamically adjust the weight according to the classification difficulty of the sample, so that the model pays more attention to the hard samples, thereby improving the recognition ability of the model to the boundary or minority class samples. The basis is the cross-entropy loss (Cross-Entropy Loss), and after adding the weight modulation factor, the form is as follows:

[0091]

[0092] In the above formula, represents the number of pixels in the image, represents the probability that the i-th sample belongs to its true class predicted by the model, represents the weight of the i-th sample, which is composed of two parts:

[0093]

[0094] represents the class weight, which is used to solve the class imbalance problem. represents the difficulty weight, which is used to strengthen the difficult samples. represents the adjustment factor, which is used to reduce the weight of easy classification samples. Generally, the value is 2.

[0095] S3.2, training strategy optimization: in the process of model training, a series of optimization strategies are adopted.

[0096] In terms of data augmentation, rotation, up-down flipping, left-right flipping, translation and HSV space adjustment data augmentation operations are used to expand the input FISH image data and improve the generalization performance of the model.

[0097] In terms of learning rate adjustment, the cosine annealing strategy is adopted to adjust the learning rate, the initial learning rate is adjusted to 0.001, the final learning rate is set to 0.1, the learning rate preheating parameter is set to 5, and the learning rate preheating stage momentum is set to 0.8.

[0098] At the same time, label smoothing, weight decay and random dropout regularization methods are set to prevent model overfitting and improve the stability and accuracy of the model. AdamW optimizer is used, the label smoothing probability is set to 0.1, the weight decay rate is 0.05, and the classifier dropout rate is set to 0.5.

[0099] S4, second model training: load the pre-training parameters of the second model, adapt the input channel to adapt to the FISH image features, the second model is a deep learning model with fine boundary segmentation capability, the training process uses a combination loss function containing mask loss, boundary loss and regularization loss, and combines targeted data augmentation and dynamic training strategy;

[0100] In this embodiment, the model parameters of Cellpose (the second model) pre-trained on the biomedical image dataset (cpsam) are loaded, and the original model architecture is kept unchanged, with the input channel adapted to 3 to adapt to the features of the FISH image.

[0101] The training-related parameters are set: the training epoch is set to 200, the batch size is set to 32, the image size is set to 1024*1024 high-resolution image, and GPU acceleration and mixed precision training (AMP) are abandoned.

[0102] S4.1, the loss function used in the training of the Cellpose model:

[0103] Core loss function:

[0104]

[0105] wherein, L mask represents the mask loss, and the weight is 0.6. L boundary represents the boundary loss, and the weight is 0.3. L regularization represents the regularization loss, and the weight is 0.1.

[0106] Mask loss ( ) uses an improved Dice-CE combined loss:

[0107]

[0108] wherein, represents the Dice loss, which is more sensitive to the segmentation boundary and small target by calculating the overlap ratio of the predicted area and the real area; represents the coefficient of the Dice loss, which is usually taken as 0.7; represents the cross-entropy loss, which solves the class imbalance problem through a logarithmic penalty mechanism; represents the coefficient of the cross-entropy loss, which is usually taken as 0.3.

[0109] The standard calculation method of the cross-entropy loss here is as follows:

[0110] 1. Binary classification scenario (such as the cells in the present application are divided into "first type" and "second type"):

[0111] For a single sample, let the true label be , and the probability of the model predicting that the sample belongs to class 1 is p, then the cross-entropy loss is:

[0112]

[0113] 2. For a batch of samples, take the average of all sample losses:

[0114] The core logic is to penalize the "deviation between the predicted probability and the true label" through a logarithmic function (the less accurate the prediction, the greater the loss value). Especially for scenarios with class imbalance (such as when the proportion of samples in a certain class is extremely low), further optimization can be achieved through weighting (such as introducing class weight α), namely:

[0115]

[0116] In actual operation, you can choose a specific formula (such as the above-mentioned binary cross entropy or weighted cross entropy) according to your needs and adjust it according to the final effect.

[0117] Boundary optimization loss ( ) uses multi-scale matching loss:

[0118]

[0119] in, represents the L1 norm, Indicates the calculation of the sum of the pixel-by-pixel absolute errors of the two gradient maps; Represents the downsampling scale, that is, the prediction mask ( ) and the true mask (M) are downsampled by 1 times (original scale), 2 times, and 4 times respectively to obtain the predicted masks at different sampling scales and the true mask ; It is the Sobel gradient operator, which calculates the gradient amplitude of the image at each pixel through convolution. The formula is:

[0120]

[0121] is the input image, represents the gradient of the image in the x direction, Represents the gradient of the image in the y direction.

[0122] Regularization constraints ( ) using morphological consistency loss:

[0123]

[0124] in, Cell masks predicted by the model; Prediction mask The area (unit: pixel²); is the preset mean cell area (unit: pixel²); To predict the circularity of cells (morphological parameter); is the preset mean of cell circularity. The first term is the area term, divided by the standard deviation of the area of the model training set (σ ), to obtain "(Area deviation) / σ " (dimensionless); the second term is the circularity term, divided by the standard deviation of the circularity of the model training set (σ cire ), to obtain "(circularity deviation) / σ cire " (dimensionless). In this way, after normalization, both terms are converted into standardized deviations without dimension, which can be directly added.

[0125] S4.2, data enhancement strategy:

[0126] In terms of morphological enhancement, elastic deformation is used to simulate cell extrusion (a = 120, σ = 8, application probability 40%), and directional deformation is used to stretch along the long axis of the cell, with a deformation rate of 15%. Among them, a here is the deformation intensity parameter, which is used to control the amplitude (intensity) of elastic deformation. The larger the a value, the more significant the extrusion / twisting of the cell morphology; otherwise, the deformation is lighter. In the present application, a = 120, indicating that the true state of the cell being extruded is simulated by moderate intensity deformation. And σ is the standard deviation of the Gaussian kernel, which is used to control the smoothness of the deformation field. The larger the σ value, the smoother the generated deformation field (the deformation transition is more natural); otherwise, local severe deformation may occur. In the present application, σ = 8, which ensures that the cell morphology after deformation still meets the biological characteristics (avoiding morphological abnormalities caused by excessive twisting).

[0127] In terms of fluorescence characteristic enhancement, channel separation enhancement is used to independently process the HER2 / CEP17 channel. And signal attenuation is simulated to randomly reduce the fluorescence intensity of the edge region.

[0128] In terms of generating adversarial samples, 15% of adversarial samples are mixed into the training set, and pre-trained GAN is used to generate samples such as adherent cells and blurred boundaries.

[0129] S4.3, dynamic training strategy:

[0130] This step integrates learning rate scheduling, early stopping mechanism and model saving strategy, and the specific process is as follows:

[0131] S4.3.1, dynamic adjustment of learning rate:

[0132] Preheating phase: the first 5 training rounds use a linear warming-up strategy to increase the learning rate from the initial value to the target value, and the target value is set to .

[0133] Cosine annealing decay, from the 6th round, adjust the learning rate according to the formula to achieve smooth decay.

[0134] wherein the cosine annealing is to make the learning rate smooth decay with the training epoch by the periodic change of the cosine function, to avoid the gradient mutation and improve the model convergence stability. denotes the learning rate; is the initial learning rate; epoch is the current iteration round (integer, gradually increasing with the training, such as the first round, the second round, …) in the training process; π is the circular constant, used to construct the phase change of the cosine function; 100 denotes the circular adjustment parameter, used to define the “half cycle length” of the cosine decay.

[0135] Restart mechanism: if the verification loss decreases by less than 0.1% for 10 consecutive rounds, reset the learning rate to , to break the optimization stagnation.

[0136] S4.3.2, early stopping and checkpoint control:

[0137] Monitoring indicators: take the boundary F1 score ( ) as the core indicator, and comprehensively consider the precision and recall of cell contour segmentation.

[0138] Early stopping mechanism: if the F1 score increases by less than 0.5% for 15 consecutive rounds, , terminate the training to avoid overfitting.

[0139] Best model saving: record the peak value of throughout the process, save the corresponding model and optimizer state, and ensure that the final output is the optimal model.

[0140] S5, double model segmentation prediction: use the first model and the second model trained to segment the FISH image respectively, output the segmentation mask, class label and confidence of the first model, and the segmentation mask of the second model;

[0141] In this embodiment, the FISH image to be detected and segmented is input, and the Cellpose model trained is called to disable segmentation prediction, and the specific process is as follows:

[0142] S5.1, model calling: use the eval method of the Cellpose model trained to segment the input FISH image, as shown in Figure 2 .

[0143] S5.2, segmentation parameter adjustment: adjust the segmentation parameters according to the characteristics of the FISH image, and the specific parameters are as follows:

[0144] Cell diameter: This parameter is used to specify the approximate diameter of the cells. Setting a reasonable cell diameter can help improve the accuracy of segmentation. In this case, it is set to None, which means that the model will automatically estimate the cell diameter, adapting to the size variations of cells in different images, and improving the universality and accuracy of segmentation.

[0145] Flow threshold: This threshold is used to control the sensitivity of the flow field. In images with high cell density, appropriately reducing this threshold can detect more cells, but it may also introduce excessive noise. Here it is set to 0.4, balancing between detecting more nuclei and controlling noise, detecting most cells without introducing excessive noise interference.

[0146] Cell probability threshold: This parameter is used to filter the probability values of cells. Setting this threshold to 0.0 means detecting as many cells as possible.

[0147] Min cell size: This parameter is used to filter out cells that are too small, avoiding mistaking noise or small disturbances as cells. Here it is set to 15.

[0148] Batch size: The batch size determines the number of images processed at a time. Here it is set to 8, balancing between processing speed and memory resources.

[0149] Normalize: Setting normalize to True means normalizing the input images. Normalization can adjust the pixel values of the images to a uniform range, helping to improve the stability and segmentation effect of the model.

[0150] Augment: Data augmentation operations such as rotation and flipping can increase data diversity, but here it is set to False, as no data augmentation is needed.

[0151] In this embodiment, the trained YOLO model is called, and the predict method of the model is used to segment and predict the input FISH image, as shown in Figure 3 The red box represents the "cell" class, and the green box represents the "other" class. This method receives image paths, save options, device selection, whether to output bounding boxes, and confidence threshold parameters as input.

[0152] Output results: After the segmentation processing of the YOLO model, the segmentation results will be obtained, mainly including the following information:

[0153] Predicted masks: A three-dimensional array, each channel represents a predicted segmentation region. Through the mask, the location and range of each prediction region in the image can be determined.

[0154] Class IDs (cls_ids): Represents the category to which each predicted region belongs, different class IDs correspond to different cell categories.

[0155] Confidence (confs): Reflects the reliability of each prediction result, the higher the value, the more reliable the prediction result.

[0156] S5.3, Output results: After processing by the Cellpose model, three main output results are obtained:

[0157] Cell masks: A two-dimensional array, each different integer value represents a segmented cell, and the background area value is 0. Through the mask, the location and range of each cell in the image can be clearly seen.

[0158] Flow field: Contains the flow information of the cells, which can be used for further analysis of the morphology and motion characteristics of the cells.

[0159] Style information: Reflects the overall characteristics of the image, which can be used for image classification or clustering analysis.

[0160] S6, First model segmentation result optimization: Post-processing of the segmentation results of the first model (YOLO), including confidence sorting, mask scaling, morphological operation and overlapping region removal, as shown in Figure 4 .

[0161] In this embodiment, the specific steps are as follows:

[0162] S6.1, Confidence sorting, get sorting index. According to the confidence of the segmentation mask, sort from large to small, and rearrange the segmentation mask and class ID according to the sorting index.

[0163] S6.2, Initialize the saved mask array: Initialize the saved mask data, all values are 0, and the size is consistent with the real height and width of the original image.

[0164] S6.3, Traverse the prediction results and process individual masks: Traverse each segmentation mask and its corresponding class ID according to the confidence sorting, and perform the following processing procedures on each mask in turn:

[0165] S6.3.1, Mask image scaling: Use the method of quadratic spline interpolation to scale the mask image to the source image size.

[0166] S6.3.2, Mask dilation: morphological dilation operation is performed on the binary mask, and the structure element of the dilation process is a 3*3 circular element, which increases the area of the binary mask.

[0167] S6.3.3, Single cell image extraction: the input FISH image is converted from RGB color space to HSV color space, the intensity (V) component is extracted, and the intensity (V) component is normalized. The intensity V component is subjected to logical AND operation with the processed binary mask image to obtain a refined segmentation mask of the single cell region.

[0168] S6.3.4, Threshold segmentation of cell region: Otsu thresholding is used to further segment the intensity (V) component of the real cell region, to more accurately distinguish the cell and background regions, and to obtain a refined segmentation mask of the cell region.

[0169] S6.3.5, Morphological closing operation: morphological closing operation is performed on the refined segmentation mask, and a 3*3 rectangular result element is used for closing operation to eliminate holes in the binary region and make the cell more complete.

[0170] S6.3.6, Overlapping region removal: the saved mask array is subjected to logical NOT operation, and logical AND operation is performed with the processed refined segmentation mask to obtain a non-overlapping part mask.

[0171] S6.3.7, Saved mask update: logical OR operation is performed between the non-overlapping part mask and the saved mask array to update the saved mask array.

[0172] S7, Dual-model collaborative decision: logical AND operation is performed between the segmentation mask of the second model (Cellpose) and the optimized segmentation mask of the first model, and the cell category is determined according to the number and distribution of connected domains in the operation result to generate the final detection result.

[0173] In this embodiment, the segmentation result of the FISH image by the Cellpose model is subjected to logical operation with the segmentation and classification result of the Yolo model to obtain the final segmentation result, as shown in Figure 5 The red frame represents the "cell" category, and the green frame represents the "other" category. The specific process is as follows:

[0174] S7.1, Extract the segmentation result of the Cellpose model, and perform logical AND operation between the segmentation result of the Yolo model and the cells segmented by the Cellpose model, and count the results after logical AND operation.

[0175] If there is only one connected domain after the operation, the classification of the cell is determined according to the classification result predicted by the YOLO model, and the category ID=0 represents "cell", and the category ID=1 represents "other".

[0176] If there are multiple connected domains after the logical AND operation, the classification of the cell is determined according to the number of pixel points of the connected domain. The Yolo model prediction result with the largest number of pixel points is taken as the classification of the cell, and the class ID = 0 represents "cell" and the class ID = 1 represents "other".

[0177] If there is no connected domain after the logical AND operation, the class of the cell is determined by the following method:

[0178] The result of the Cellpose model is dilated by using a square structural element, the size of the square structural element is 1*1, and the dilated cell segmented by the Cellpose model is continuously dilated. After each dilation, the image is logically ANDed with the Yolo model result. If there is an overlapping area, the dilation is stopped, otherwise, the dilation is continued until an overlap with the Yolo result appears. Finally, according to the class of the YOLO model in the overlapping area, the classification result of the Cellpose model is determined.

[0179] S7.2, the classification of all cells segmented by the Cellpose model is confirmed, and the classification result of the whole image is generated.

[0180] Example Two

[0181] Based on the same idea, the application also provides an automatic cell detection device for fluorescence in situ hybridization images, comprising:

[0182] A data preparation module acquires FISH images and performs labeling, and divides the cells in the image into first-class cells that need to be counted for fluorescent points and second-class cells that are not analyzed for the time being. The first-class cells are complete and exist independently in the field of view, uniform in size and consistent in shape. The second-class cells are cell clusters with incomplete boundaries or adhesion.

[0183] A data set module converts the labeling results into a standard format required by the first model, including coordinate normalization, class encoding and quality control.

[0184] A first model module loads the pre-training parameters of the first model, and fine-tunes the parameters based on the data set. The first model is a deep learning model with classification and detection capabilities. The training process uses a combined loss function containing multiple loss functions, combined with data augmentation and regularization strategies.

[0185] A second model module loads the pre-training parameters of the second model, and adapts the input channel to adapt to the features of the FISH image. The second model is a deep learning model with fine boundary segmentation capability. The training process uses a combined loss function containing mask loss, boundary loss and regularization loss, combined with targeted data augmentation and dynamic training strategies.

[0186] The dual-model segmentation prediction module respectively utilizes the trained first model and the second model to segment the FISH image, and outputs the segmentation mask, the category identifier and the confidence of the first model, and the segmentation mask of the second model;

[0187] The result optimization module performs post-processing on the segmentation result of the first model, including confidence sorting, mask scaling, morphological operation and overlapping region removal;

[0188] The collaborative decision module performs logical AND operation on the segmentation mask of the second model and the optimized segmentation mask of the first model, determines the cell category according to the number and distribution of connected domains in the operation result, and generates the final detection result.

[0189] Embodiment three

[0190] The embodiment also provides an electronic device, referring to Figure 6 The electronic device includes a memory 404 and a processor 402, the memory 404 stores a computer program, and the processor 402 is configured to run the computer program to perform the steps in any of the above method embodiments.

[0191] Specifically, the processor 402 can include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0192] The memory 404 can include a mass storage that stores data or instructions. For example, and without limitation, the memory 404 can include a Hard Disk Drive (HDD), a floppy disk drive, a Solid State Drive (SSD), a flash drive, a Compact Disc Read Only Memory (CD-ROM), a magneto-optical disk, a magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. The memory 404 can be removable and / or non-removable (or fixed) as appropriate. The memory 404 can be internal or external as appropriate. In particular embodiments, the memory 404 is a Non-Volatile memory. In particular embodiments, the memory 404 includes a Read-Only Memory (ROM) and a Random Access Memory (RAM). The ROM can be a mask-programmed ROM, a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), an Electrically Alterable ROM (EAROM), or a FLASH memory, or a combination of two or more of these, as appropriate. The RAM can be a Static Random-Access Memory (SRAM) or a Dynamic Random Access Memory (DRAM), which can be a Fast Page Mode Dynamic Random Access Memory (FPMDRAM), an Extended Data Output Dynamic Random Access Memory (EDODRAM), a Synchronous Dynamic Random-Access Memory (SDRAM), or the like, as appropriate.

[0193] The memory 404 can be used to store or cache various data files needed for processing and / or communication, and possible computer program instructions executed by the processor 402.

[0194] The processor 402 implements the automated cell detection method of the fluorescence in situ hybridization image in any of the above embodiments by reading and executing the computer program instructions stored in the memory 404.

[0195] Optionally, the electronic device described above can further include a transmission device 406 connected to the processor 402 and an input / output device 408 connected to the processor 402.

[0196] The transmission device 406 can be used to receive or send data via a network. Specific examples of the network can include a wired or wireless network provided by a communication provider of the electronic device. In one example, the transmission device includes a network adapter (NIC) which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 406 can be a radio frequency (RF) module which is used to communicate with the Internet in a wireless manner.

[0197] The input / output device 408 is used to input or output information.

[0198] Embodiment Four

[0199] The embodiment also provides a readable storage medium, and the readable storage medium stores a computer program. The computer program includes program codes for controlling a process to execute the process. The process includes the automated cell detection method of the fluorescence in situ hybridization image according to the first embodiment.

[0200] It should be noted that the specific examples in the embodiment can refer to the examples described in the above embodiments and optional implementation manners, and the embodiment will not be described here.

[0201] Generally, various embodiments can be implemented in hardware or special-purpose circuitry, software, logic or any combination thereof. Some aspects of the application can be implemented in hardware, while other aspects can be implemented by firmware or software executed by a controller, microprocessor or other computing device, but the application is not limited thereto. Although various aspects of the application can be illustrated and described as block diagrams, flow charts, or using some other pictorial representation, it is well understood that these blocks, apparatus, systems, techniques or methods described herein can be implemented in hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.

[0202] Embodiments of the application can be implemented by computer software executable by a data processor of the mobile device such as in the processor entity, or by hardware, or by a combination of software and hardware. Computer software or program, also called program product, including software routines, applets and / or macros, can be stored in any apparatus-readable data storage medium and they include program instructions to implement certain tasks. The program product can include one or more computer-executable components tangibly embodied in a computer- readable medium, when executed, for implementing one or more embodiments of the present application. The one or more computer-executable components can be one or more of: a process; a function; a routine; a sag; a subroutine; a plug-in; an app; or a program. The one or more computer-executable components can include a computer program that is written in any form of programming language, including code.

[0203] Those skilled in the art should clearly understand that each technical feature in the above embodiments can be combined with any other technical feature, and for the sake of brevity, each technical feature in the above embodiments has not been described in all possible combinations, however, as long as the combinations of the technical features do not exist contradictions, it should be considered that they are within the scope of the present disclosure.

[0204] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the scope of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. An automated cell detection method using fluorescence in situ hybridization images, characterized in that: The following steps are involved: S1. Data preparation: Acquire and annotate FISH images, and divide the cells in the images into the first category of cells for which fluorescence spot counting is required, and the second category of cells that are not yet analyzed. The first category of cells are complete and individual cells in the field of view with uniform size and consistent morphology. The second category of cells are cell clusters with incomplete boundaries or adhesions. S2. Generate a dataset: Convert the annotation results into the standard format required by the first model. The conversion includes coordinate normalization, category coding, and quality control. The quality control includes filtering invalid annotations, checking polygon closure, and verifying vertex coordinate ranges. S3. First model training: loading pre-trained parameters of the first model and fine-tuning the parameters based on the dataset. The first model is a Yolov12 model with classification and detection capabilities. The training process uses a combination loss function containing multiple loss functions, combined with data enhancement and regularization strategies. S4. Second model training: Load the pre-trained parameters of the second model and adapt the input channels to suit the FISH image characteristics. The second model is a Cellpose model with fine boundary segmentation capabilities. The training process uses a combined loss function including mask loss, boundary loss, and regularization loss, combined with targeted data augmentation and dynamic training strategies. S5. Dual-model segmentation prediction: Use the trained first model and second model to segment the FISH image, output the segmentation mask, category identifier and confidence of the first model, and the segmentation mask of the second model; S6. Optimizing the segmentation results of the first model: performing post-processing on the segmentation results of the first model, including confidence ranking, mask scaling, morphological operations, and overlapping area removal; S7, dual-model collaborative decision-making: performing a logical AND operation on the segmentation mask of the second model and the optimized segmentation mask of the first model, determining the cell type based on the number and distribution of connected domains in the operation result, and generating a final detection result; Among them, when the result of the logical AND operation contains a single connected domain, the category identifier predicted by the first model is directly used; when there are multiple connected domains, the category identifier of the first model corresponding to the connected domain with the largest number of pixels is used as the result; when there is no connected domain, the segmentation mask of the second model is expanded until it overlaps with the segmentation mask of the first model, and the category identifier of the first model corresponding to the overlapping area is used as the result.

2. The automated cell detection method of fluorescence in situ hybridization images according to claim 1, wherein: In step S1, the annotation is carried out by a double-person annotation verification method, and the ambiguous samples are finally interpreted by a third-party expert, and the annotation tool is an image annotation tool.

3. The automated cell detection method of fluorescence in situ hybridization images according to claim 1, wherein: In step S3, the combined loss function includes a weighted combination of Dice loss, Focal loss, Jaccard loss, mutual exclusion loss and soft sampling weighted loss, the data enhancement includes rotation, flipping and color space adjustment, and the regularization strategy includes label smoothing, weight decay and random dropout.

4. The automated cell detection method of fluorescence in situ hybridization images according to claim 1, wherein: In step S4, the mask loss is a joint loss of Dice loss and cross entropy loss, the boundary loss is a multi-scale matching loss, the regularization loss is a morphological consistency loss, and the targeted data enhancement includes elastic deformation, fluorescence channel separation, and incorporation of adversarial samples.

5. The automated cell detection method of fluorescence in situ hybridization images according to claim 1, wherein: In step S4, the dynamic training strategy includes a learning rate warm-up, a cosine annealing decay and restart mechanism, and an early stopping mechanism based on a boundary evaluation index.

6. An automated cell detection device for fluorescence in situ hybridization images, characterized in that: include: The data preparation module acquires and annotates FISH images, dividing the cells in the images into the first category of cells for which fluorescence point counting is required, and the second category of cells that are not yet analyzed. The first category of cells are complete and single cells in the field of view with uniform size and consistent morphology, while the second category of cells are cell clusters with incomplete boundaries or adhesions. The dataset module converts the annotation results into the standard format required by the first model. The conversion includes coordinate normalization, category coding and quality control. The quality control includes filtering invalid annotations, checking polygon closure and verifying vertex coordinate ranges. The first model module loads the pre-trained parameters of the first model and fine-tunes the parameters based on the dataset. The first model is a Yolov12 model with classification and detection capabilities. The training process uses a combination of multiple loss functions and combines data enhancement and regularization strategies. The second model module loads the pre-trained parameters of the second model and adapts the input channels to suit the FISH image characteristics. The second model is a Cellpose model with fine boundary segmentation capabilities. The training process uses a combined loss function including mask loss, boundary loss, and regularization loss, combined with targeted data augmentation and dynamic training strategies; The dual-model segmentation prediction module segments the FISH image using the trained first model and the second model, and outputs the segmentation mask, category identifier, and confidence score of the first model, as well as the segmentation mask of the second model. The result optimization module performs post-processing on the segmentation results of the first model, including confidence-based sorting, mask scaling, morphological operations, and overlapping area removal; The collaborative decision-making module performs a logical AND operation on the segmentation mask of the second model and the optimized segmentation mask of the first model, determines the cell category based on the number and distribution of connected domains in the operation result, and generates the final detection result. When the logical AND operation result contains a single connected domain, the category identifier predicted by the first model is directly used; when there are multiple connected domains, the category identifier of the first model corresponding to the connected domain with the largest number of pixels is used as the result; when there is no connected domain, the segmentation mask of the second model is expanded until it overlaps with the segmentation mask of the first model, and the category identifier of the first model corresponding to the overlapping area is used as the result.

7. An electronic device comprising a memory and a processor, characterized in that: The memory stores a computer program, and the processor is configured to run the computer program to execute the automated cell detection method based on fluorescence in situ hybridization images according to any one of claims 1 to 5.

8. A readable storage medium, characterized in that: The readable storage medium stores a computer program, which includes a program code for controlling a process to execute a process, wherein the process includes the automated cell detection method based on fluorescence in situ hybridization images according to any one of claims 1 to 5.

Images