Training Method and System for Circulating Tumor Cell Recognition Model Based on Deep Learning

Abstract

The present invention provides a deep learning-based circulating tumor cell recognition model training method and system. By acquiring a collection of multi-channel pathology sample images of circulating tumor cells, cross-modal feature extraction is performed on the multi-channel pathology sample images to generate a fused feature vector containing cell morphology features and microenvironment features. A deep convolutional network with a multi-stage training strategy is constructed. A layered cross-validation mechanism is used to verify and adjust the network parameters of the deep convolutional network during training. Finally, a final recognition model is generated based on the verified and adjusted network parameters. The final recognition model is configured to receive a clinical pathology image stream and output the location coordinates and classification confidence of circulating tumor cells. The present invention can ensure that circulating tumor cell recognition results are both consistent with the spatial distribution patterns of cells and interpretable for clinical pathology diagnosis. This overcomes the problems of traditional methods that ignore microenvironmental interactions, resulting in high false positive rates and insufficient differentiation of cell subtypes.

Description

Technical Field

[0001] The present invention relates to the field of deep learning, and in particular to a method and system for training a circulating tumor cell recognition model based on deep learning. Background Art

[0002] In the field of medical image analysis, the identification of circulating tumor cells is of great significance for cancer metastasis monitoring. The current mainstream method extracts nuclear morphological features based on single-channel pathological images and achieves tumor cell detection by training a classification model through a convolutional neural network. Existing technologies mostly focus on the calculation of isolated morphological parameters such as nuclear size and nuclear-cytoplasmic ratio deviation, and use end-to-end deep learning models to directly output classification results. They fail to effectively integrate the spatial interaction characteristics of tumor cells and the microenvironment, resulting in feature misjudgment when the model faces complex pathological scenarios such as stain differences and immune cell infiltration. Traditional training strategies are insufficient in recognizing behavioral patterns unique to circulating tumor cells, such as loose intercellular connections and matrix penetration. In addition, existing model compression methods mostly achieve lightweighting through global channel pruning, which destroys the local feature response path that is sensitive to tumor diagnosis and affects the pathological interpretability during clinical deployment. Based on this, existing technologies have problems such as large positioning deviation in real clinical samples, high missed detection rate of undifferentiated cells, and difficulty in filtering false positive samples. Summary of the Invention

[0003] The present invention provides a deep learning-based circulating tumor cell recognition model training method and system.

[0004] In a first aspect, an embodiment of the present invention provides a deep learning-based circulating tumor cell recognition model training method, the method comprising: obtaining a multi-channel pathology sample image set of circulating tumor cells, the multi-channel pathology sample image set comprising cell microscopic images treated with different stains, each sample image carrying corresponding cell type annotation information; performing cross-modal feature extraction on the multi-channel pathology sample images to generate a fused feature vector comprising cell morphology features and microenvironment features, wherein the microenvironment features include matrix component distribution features and immune cell infiltration features; constructing a deep convolutional network with a multi-stage training strategy, wherein the first stage utilizes a cross-channel feature attention mechanism to optimize convolution kernel parameters, the second stage enhances feature discriminability by generating adversarial samples, and the third stage utilizes knowledge distillation to compress the network size; utilizing a hierarchical cross-validation mechanism to verify and adjust the network parameters of the deep convolutional network during training, wherein the hierarchical cross-validation mechanism divides the network into multiple validation subsets according to the biological characteristics corresponding to the cell type annotation information, and adjusts the network parameters based on the classification error of each validation subset; generating a final recognition model based on the verified and adjusted network parameters, wherein the final recognition model is configured to receive a clinical pathology image stream and output the location coordinates and classification confidence of circulating tumor cells.

[0005] In a second aspect, an embodiment of the present invention provides a computer system, comprising: a memory storing a computer program; and a processor for loading the computer program to implement the deep learning-based circulating tumor cell recognition model training method as described above.

[0006] The circulating tumor cell recognition model training method and system provided by the present invention significantly improves the accuracy and clinical applicability of recognition results by integrating the cell morphological features and microenvironmental features of multi-channel pathological images and adopting a multi-stage optimization strategy. A spatial position correlation alignment mechanism is established in cross-modal feature extraction to form a biomedical semantic association between cell nuclear atypia parameters and matrix fiber orientation characteristics, effectively capturing the morphological-microenvironmental coordinated changes in tumor cell invasion and metastasis. Adversarial samples are used to generate co-occurrence patterns that interfere with the density of intercellular connections and immune cell aggregation density, enhancing the model's ability to discriminate cell variation characteristics in real complex pathological scenarios. Knowledge distillation technology is combined to inherit the teacher network's sensitive response path to the nuclear atypia index, compressing the model size while retaining key biometric feature recognition capabilities. The network parameters are further dynamically adjusted based on the hierarchical verification mechanism of tumor biological characteristics to ensure the balanced generalization performance of the model on epithelial, mesenchymal and undifferentiated tumor cell subsets. The final output positioning coordinates and classification confidence are deeply integrated with the immune infiltration density and cell connection density indicators, so that the recognition results are consistent with the spatial distribution of cells and have clinical pathological diagnostic interpretability, overcoming the problems of high false positive rate and insufficient cell subtype differentiation caused by ignoring microenvironment interactions in traditional methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0008] Figure 1 This is a flowchart of a circulating tumor cell recognition model training method based on deep learning provided by an embodiment of the present invention.

[0009] Figure 2 It is a schematic diagram of the composition of a computer system provided by an embodiment of the present invention.

[0010] Reference numerals: 101 - processor; 102 - communication interface; 103 - memory. DETAILED DESCRIPTION

[0011] See also Figure 1 , Figure 1 A flowchart of a deep learning-based circulating tumor cell recognition model training method provided in an embodiment of the present invention is provided. The deep learning-based circulating tumor cell recognition model training method can be executed by a computer system. The deep learning-based circulating tumor cell recognition model training method may include the following steps:

[0012] Step S100: Acquire a multi-channel pathology sample image set of circulating tumor cells, wherein the multi-channel pathology sample image set includes cell microscopic images processed with different stains, and each sample image carries corresponding cell type annotation information.

[0013] In an embodiment of the present invention, a multi-channel pathology sample image set is composed of cell microscopic images processed with different stains. The staining is to better display the different structures and components of cells, and different stains can highlight the specific characteristics of cells. Cell microscopic images are images obtained by observing and photographing cell samples under a microscope, which contain information such as cell morphology and structure. Cell type annotation information is a clear identification of the cell type in each sample image, such as epithelial tumor cells, mesenchymal tumor cells, etc. The annotation information is the basis for supervised learning in subsequent model training.

[0014] This invention does not restrict the source of samples in the multi-channel pathology sample image collection. For example, as permitted by laws and regulations, circulating tumor cell samples treated with different stains can be collected from a hospital's pathology database. These samples are then observed and photographed using specialized microscope equipment to produce cell microscopic images. Each sample image can also be annotated with the cell type by domain experts.

[0015] Step S200: performing cross-modal feature extraction on the multi-channel pathological sample image to generate a fusion feature vector including cell morphology features and microenvironment features, wherein the microenvironment features include matrix component distribution features and immune cell infiltration features.

[0016] Cross-modal feature extraction involves extracting features from data from different modalities (in this embodiment, multi-channel pathology sample images treated with different stains). Cell morphological features refer to the morphological and structural characteristics of the cell itself, such as the size, shape, and nuclear-cytoplasmic ratio of the nucleus. These features can reflect the cell's biological properties and pathological state. Microenvironmental features refer to the characteristics of the cell's surrounding environment, including matrix component distribution and immune cell infiltration. Matrix component distribution describes the distribution of the extracellular matrix, such as the orientation and density of matrix fibers; immune cell infiltration reflects the distribution and aggregation of immune cells in tumor tissue, helping to understand the tumor's immune response and prognosis. A fused feature vector is a vector obtained by integrating cell morphological features and microenvironmental features, containing comprehensive information about the cell and its microenvironment.

[0017] Cross-modal feature extraction from multi-channel pathology sample images can be performed using, for example, deep learning-based convolutional neural networks (CNNs), such as ResNet and VGG. The extracted cell morphology features and microenvironmental features are then fused to generate a fused feature vector.

[0018] As an embodiment, step S200 performs cross-modal feature extraction on a multi-channel pathological sample image to generate a fusion feature vector including cell morphological features and microenvironmental features. Specifically, the following steps S210-S260 may be included:

[0019] Step S210: performing superpixel segmentation on the multi-channel pathological sample image to generate structural data covering the cell nucleus region, cytoplasm region and extracellular matrix. The structural data includes cell nucleus morphology parameters, cell membrane curvature parameters and extracellular matrix distribution parameters of each region.

[0020] Superpixel segmentation is used to divide an image into several small regions with similar characteristics, also known as superpixels. In this embodiment of the present invention, superpixel segmentation is performed on a multi-channel pathology sample image to separate the cell nucleus, cytoplasm, and extracellular matrix regions in the image and generate structural data.

[0021] Nuclear morphological parameters include the size, shape, and nuclear-to-cytoplasmic ratio of the nucleus. These parameters can be obtained by geometrically analyzing the segmented nuclear region. For example, the area, perimeter, and lengths of the major and minor axes of the nucleus can be calculated, and then morphological indicators such as the nuclear-to-cytoplasmic ratio can be calculated based on these parameters. Cell membrane curvature parameters describe the degree of curvature of the cell membrane and can be obtained by analyzing the curvature of the cell membrane boundary. Extracellular matrix distribution parameters include the orientation and density of matrix fibers. These parameters can be obtained by analyzing the image features of the extracellular matrix region. For example, texture analysis methods can be used to calculate the orientation and density of matrix fibers.

[0022] Various algorithms can be used to perform superpixel segmentation, such as the Simple Linear Iterative Clustering (SLIC) algorithm. The SLIC algorithm segments an image into superpixels by clustering them in both the color and coordinate spaces of the image. For example, cluster centers are randomly initialized in the image. Pixels are then assigned to clusters based on their distance from the cluster centers, with the locations of the cluster centers continuously updated until convergence conditions are met.

[0023] Step S220: Perform a two-branch feature extraction based on the structural data, where the first branch extracts the nuclear atypia index and the intercellular connection density index as cell morphological features through geometric topological analysis, and the second branch extracts the matrix fiber orientation characteristics and immune cell aggregation density as microenvironment features through spatial frequency transformation.

[0024] Dual-branch feature extraction involves inputting structural data into two separate branches for feature extraction, each extracting a different type of feature. The first branch primarily focuses on the morphological characteristics of the cells themselves, extracting the nuclear atypia index and intercellular connectivity density index through geometric topological analysis. In this embodiment of the present invention, the boundaries between the cell nucleus and the intercellular connectivity region are analyzed to extract relevant feature indices.

[0025] The nuclear atypia index is an indicator that measures the degree of abnormal nuclear morphology, which reflects the malignancy of tumor cells. The intercellular connection tightness index describes the tightness of the connection between cells, which helps to understand the growth and migration characteristics of cells. The second branch focuses on the microenvironment characteristics of the cells, and extracts the matrix fiber direction characteristics and immune cell aggregation density through spatial frequency transformation. Spatial frequency transformation is used to convert images from the spatial domain to the frequency domain, which can reveal the distribution of different frequency components in the image. In an embodiment of the present invention, the direction of the matrix fibers and the aggregation density of immune cells are extracted by performing spatial frequency transformation on the image of the extracellular matrix area.

[0026] In the first branch, geometric topology analysis can be performed using the following specific steps. First, edge detection is performed on the structural data to generate nuclear contour boundary data and intercellular connection region boundary data. The boundary data contains pixel-level coordinate sequences and curvature change information. Edge detection can use the Canny algorithm, which calculates the gradient values of pixels in the image to identify edge points. Next, a polygon fitting operation is performed on the nuclear contour boundary data to generate a polygon vertex distribution describing the nuclear morphology. Based on this vertex distribution, the nuclear-to-cytoplasmic ratio deviation, contour concavity index, and main axis deflection angle are calculated. These parameters are then integrated to generate the nuclear atypia index. Polygon fitting can use the least squares method to approximate the nuclear contour by fitting the vertices of the polygon. For the intercellular connection region boundary data, topological structure analysis is performed to extract the statistical distribution of the overlap length and contact angle of adjacent cell contact surfaces. Combined with the area percentage of the connection region, an intercellular connection density index is generated. Topological structure analysis can extract relevant features by analyzing the topological relationships of the boundary data, such as connectivity and the number of holes.

[0027] In the second branch, spatial frequency transformation can be performed using either a discrete Fourier transform (DFT) or a fast Fourier transform (FFT). First, a spatial frequency transform is performed on the extracellular matrix regions in the structural data, converting the pixel distribution into a frequency-domain energy distribution. The dominant frequency direction is extracted as a characteristic of the matrix fiber orientation. Next, regional clustering analysis is performed on the frequency-domain energy distribution to identify the spatial density distribution of immune cell clusters. The immune cell density is calculated by combining the area of the clustered regions with the contact boundary length of adjacent matrix regions. Regional clustering analysis can use the K-means clustering algorithm to identify immune cell clusters by clustering points in the frequency-domain energy distribution into distinct regions. For example, for a multi-channel pathology sample image, structural data is obtained through superpixel segmentation and then fed into the first and second branches for feature extraction. In the first branch, the Canny algorithm is used to detect the boundaries of cell nuclei and intercellular junctions. Polygonal fitting is then performed using the least squares method to calculate the nuclear atypia index and intercellular junction density index. In the second branch, FFT was used to perform spatial frequency transformation on the extracellular matrix area to extract the matrix fiber orientation characteristics, and then the K-means clustering algorithm was used to identify the immune cell aggregation area and calculate the immune cell aggregation density.

[0028] As an embodiment, in step S220, the first branch extracts the nuclear atypia index and the intercellular connection density index as cell morphological features through geometric topological analysis, which may specifically include the following steps S221-S223:

[0029] Step S221: performing edge detection on the structural data to generate cell nucleus contour boundary data and intercellular connection region boundary data. The boundary data includes pixel-level coordinate sequences and curvature change information.

[0030] In this embodiment of the present invention, edge detection is performed on structural data to extract boundary information between the cell nucleus and intercellular junctions. The pixel-level coordinate sequence, i.e., the coordinate values of each pixel on the boundary, accurately describes the location of the boundary. Curvature variation information reflects the degree of boundary curvature, assisting in analyzing the morphology of the cell nucleus and the structure of intercellular junctions.

[0031] Various algorithms can be used to perform edge detection, such as the aforementioned Canny algorithm. During this process, the image is first Gaussian smoothed to reduce noise. The gradient value and direction of each pixel in the image are then calculated. Non-maximum suppression is then performed on the gradient values to refine the edges. Finally, a double-thresholding method is used to screen the edges and identify true edge points. For example, the Canny algorithm is used to detect edges in a structural data image obtained through superpixel segmentation. First, the image is Gaussian smoothed to remove noise. The gradient value and direction of each pixel are then calculated. Non-maximum suppression is then performed, setting pixels whose gradient values are not local maxima to zero. Finally, a double-thresholding method is used to identify pixels with gradient values greater than a high threshold as edge points. Pixels with gradient values between the high and low thresholds and connected to edge points are also identified as edge points. This yields the boundary data for the cell nucleus outline and the boundary data for the intercellular connection region. This boundary data contains pixel-level coordinate sequences and curvature change information.

[0032] Step S222: polygon fitting is performed on the nucleus contour boundary data to generate a polygon vertex distribution describing the nucleus morphology, and the nucleus-cytoplasm ratio deviation, contour concavity index, and main axis deflection angle are calculated based on the vertex distribution. The parameters are fused to generate the nucleus atypia index.

[0033] Polygon fitting can approximate a discrete set of points using polygons. In an embodiment of the present invention, a polygon fitting operation is performed on the nuclear outline boundary data to describe the morphology of the cell nucleus using the distribution of polygonal vertices. The nuclear-cytoplasmic ratio deviation, which is the degree of difference between the nuclear-cytoplasmic ratio of a cell nucleus and that of a normal cell, can reflect the degree of nuclear abnormality. The contour concavity index describes the concavity and convexity of the nuclear outline, reflecting the degree of irregularity of the cell nucleus. The principal axis deflection angle, which is the angle of deflection of the principal axis of the cell nucleus relative to a reference direction, can reflect the orientation of the cell nucleus. Specifically, polygon fitting can use the least squares method to find the optimal function matching the data by minimizing the sum of squared errors. In an embodiment of the present invention, the least squares method is used to fit the nuclear outline boundary data to determine a set of polygon vertices such that the error between the polygon formed by these vertices and the nuclear outline is minimized. To calculate the nuclear-cytoplasmic ratio deviation, the area of the nucleus and the area of the cytoplasm are first calculated, followed by the nuclear-cytoplasmic ratio, which is then compared with the nuclear-cytoplasmic ratio of a normal cell to obtain the nuclear-cytoplasmic ratio deviation. The contour concavity index can be obtained by calculating the concavity of polygon vertices, for example, by counting the number of convex and concave vertices in a polygon and calculating their ratio. The principal axis deflection angle can be obtained by calculating the principal axis orientation of the polygon, for example, using principal component analysis (PCA). Finally, the nuclear-cytoplasmic ratio deviation, contour concavity index, and principal axis deflection angle are combined to generate the nuclear atypia index. This fusion method can be weighted summation, for example, by assigning a weight to each parameter, multiplying them, and then adding them together to generate the nuclear atypia index. For example, for a nucleus outline boundary data set, polygon fitting is performed using the least squares method to obtain a set of polygon vertex distributions. The nuclear-cytoplasmic ratio deviation, contour concavity index, and principal axis deflection angle of the nucleus are then calculated, and these parameters are weighted summed to generate the nuclear atypia index.

[0034] Step S223: performing topological structure analysis on the boundary data of the intercellular connection area, extracting the statistical distribution of the overlapping length and contact angle of the adjacent cell contact surfaces, and generating an intercellular connection tightness index based on the area ratio of the connection area.

[0035] Topological structure analysis can reveal topological features of an object, such as its connectivity and the number of holes. In an embodiment of the present invention, topological structure analysis is performed on the boundary data of the intercellular connection region to extract the statistical distribution of the overlap length and contact angle of adjacent cell contact surfaces. The overlap length is the length of the adjacent cell contact surface, and the contact angle is the angle between adjacent cell contact surfaces. The area ratio of the connection region is the ratio of the area of the intercellular connection region to the total cell area.

[0036] Topological structure analysis can be achieved by analyzing the topological relationships of boundary data. For example, using graph theory, boundary data can be represented as a graph, where nodes represent points on the boundary and edges represent connections between adjacent nodes. By analyzing the graph's connectivity and the distances between nodes, the statistical distribution of the overlap length and contact angle of adjacent cell contacts can be extracted. Then, combined with the area percentage of the connected region, an intercellular connectivity tightness index is generated. This intercellular connectivity tightness index can be generated by comprehensively considering the overlap length, the statistical distribution of the contact angle, and the area percentage of the connected region, for example, by combining these parameters using a function. For example, given the boundary data of an intercellular connection region, graph theory can be used to represent it as a graph. By analyzing the graph's connectivity and the distances between nodes, the statistical distribution of the overlap length and contact angle of adjacent cell contacts can be obtained. Assuming the mean overlap length is L, the standard deviation of the contact angle is A, and the area percentage of the connected region is S, a function f(L,A,S) is used to generate the intercellular connectivity tightness index, where f can be a linear combination function, such as f(L,A,S) = 0.5L + 0.3(1 / A) + 0.2S. It is understood that before combining, normalization or standardization can be used to unify or eliminate the dimensions of each parameter (e.g., maximum-minimum normalization, normalizing the values to [0, 1]) before performing subsequent calculations. This is common knowledge and will not be described in detail. Similarly, for subsequent similar fusion calculations of parameters of different dimensions, those skilled in the art can use common knowledge to unify or eliminate conflicts in the relevant dimensions and align the dimensions to meet the calculation requirements.

[0037] In one embodiment, the second branch extracts matrix fiber orientation characteristics and immune cell aggregation density as microenvironment characteristics through spatial frequency transformation, which may specifically include the following steps S224-S226:

[0038] Step S224: performing a spatial frequency transformation operation on the extracellular matrix region in the structural data, converting the pixel distribution into a frequency domain energy distribution, and extracting the main frequency direction as the matrix fiber orientation feature.

[0039] Spatial frequency transform is used to convert an image from the spatial domain to the frequency domain, which can reveal the distribution of different frequency components in the image. In an embodiment of the present invention, a spatial frequency transform operation is performed on the extracellular matrix region in the structural data to convert the pixel distribution of the extracellular matrix region into a frequency domain energy distribution. The frequency domain energy distribution describes the energy of the different frequency components in the image, which can reflect the texture and structural information of the image. The dominant frequency direction is the direction with the highest energy in the frequency domain energy distribution, which can represent the direction of the matrix fibers.

[0040] As previously mentioned, the spatial frequency transform operation can be performed using a discrete Fourier transform (DFT) or a fast Fourier transform (FFT). The image of the extracellular matrix region is cropped and preprocessed, and then converted into a frequency domain energy distribution using the FFT algorithm. Next, the direction with the highest energy in the frequency domain energy distribution is determined and used as the dominant frequency direction, i.e., the matrix fiber orientation characteristic. For example, an image of an extracellular matrix region is converted into a frequency domain energy distribution using the FFT algorithm. In the frequency domain energy distribution, the dominant frequency direction is determined by searching for the location of the maximum energy. Assuming that the frequency domain energy distribution is a two-dimensional matrix, the location of the maximum energy is determined by traversing each element in the matrix, and the direction corresponding to this location is then calculated as the matrix fiber orientation characteristic.

[0041] Step S225: performing regional cluster analysis on the frequency domain energy distribution to identify the spatial density distribution of the immune cell aggregation region, and generating the immune cell aggregation density by combining the area of the cluster region and the contact boundary length of the adjacent matrix region.

[0042] Regional cluster analysis is used to divide data points into clusters, grouping data points with similar characteristics. Regional cluster analysis of the frequency domain energy distribution can identify the spatial density distribution of immune cell clusters. The cluster area is the area of the immune cell cluster, and the contact boundary length of the adjacent matrix region is the length of the contact boundary between the immune cell cluster region and the adjacent matrix region.

[0043] Regional cluster analysis can use the K-means clustering algorithm to divide data points into K clusters by continuously updating the position of the cluster center. For example, K cluster centers are randomly initialized, and then the data points are assigned to different clusters according to the distance between the data points and the cluster center. The position of the cluster center is then updated until the cluster center no longer changes or meets the convergence condition. In an embodiment of the present invention, the data points in the frequency domain energy distribution are used as input and divided into different clusters using the K-means clustering algorithm to identify immune cell aggregation areas. Then, the contact boundary length of the cluster area and the adjacent matrix area is calculated, and the immune cell aggregation density is generated by combining these two parameters. The method for generating immune cell aggregation density can be to comprehensively consider the contact boundary length of the cluster area and the adjacent matrix area, for example, using a function to combine these two parameters. For example, for a frequency domain energy distribution, it is divided into 3 clusters using the K-means clustering algorithm to identify immune cell aggregation areas. Assuming that the area of the cluster region is A and the contact boundary length of the adjacent matrix region is B, the function g(A,B) is used to generate the immune cell aggregation density, where g can be a linear combination function, such as g(A,B)=0.6A+0.4B.

[0044] Step S226: combining the nuclear atypia index and the intercellular connection tightness index into the cell morphology feature, and combining the matrix fiber orientation feature and the immune cell aggregation density into the microenvironment feature.

[0045] In an embodiment of the present invention, the nuclear atypia index and the intercellular connection tightness index extracted above are combined to form a cell morphological feature. The cell morphological feature combines the degree of morphological abnormality of the cell nucleus and the tightness of the intercellular connection, and can more comprehensively describe the morphological characteristics of the cell. The matrix fiber orientation feature and the immune cell aggregation density are combined to form a microenvironment feature. The combination method can be vector splicing. That is, the nuclear atypia index and the intercellular connection tightness index are used as elements of a vector and spliced into a new vector as the cell morphological feature. Similarly, the matrix fiber orientation feature and the immune cell aggregation density are used as elements of a vector and spliced into a new vector as the microenvironment feature.

[0046] Step S230: Input the cell morphological features and microenvironmental features into the cross-modal alignment module. The module generates a feature alignment matrix by calculating the positional correlation between the two types of features in the spatial coordinate system. Each element in the matrix represents the degree of matching between the cell morphological features and the microenvironmental features at the corresponding spatial position.

[0047] The cross-modal alignment module is used to handle the alignment problem between different modal features. In an embodiment of the present invention, the cell morphological features and microenvironmental features are input into the cross-modal alignment module to determine the positional correlation between the two types of features in the spatial coordinate system. The spatial coordinate system can be the pixel coordinate system of the image, and the correlation between the features is determined by associating the features with the pixel positions of the image. The feature alignment matrix is a two-dimensional matrix, and each element in the matrix represents the degree of matching between the cell morphological features and the microenvironmental features of the corresponding spatial position.

[0048] The cross-modal alignment module can be implemented, for example, using a neural network based on an attention mechanism. The attention mechanism focuses on the important parts of the features, thereby improving the accuracy of feature alignment. Specifically, the cell morphological features and microenvironmental features are encoded and converted into vectors of the same dimension. The attention mechanism is then used to calculate the correlation between the two types of features to generate a feature alignment matrix. For example, a multi-layer perceptron (MLP) is used to encode the cell morphological features and microenvironmental features and convert them into 128-dimensional vectors. The attention mechanism is then used to calculate the correlation between the two vectors to generate a 128×128 feature alignment matrix. Each element in the matrix represents the degree of match between a dimension of the cell morphological feature and a dimension of the microenvironmental feature, with larger values indicating a higher degree of match.

[0049] Step S240: A multi-level feature interaction mechanism is used to process the feature alignment matrix, wherein the primary interaction layer captures the morphology-microenvironment association pattern in the local area through convolution operations, and the high-level interaction layer establishes cross-regional feature dependencies through a self-attention mechanism.

[0050] A multi-level feature interaction mechanism processes interactions between features, extracting correlation information between features at different levels. The primary interaction layer focuses on feature interactions within a local region, capturing morphological-microenvironmental correlation patterns within that region through convolution operations. Convolution operations extract local features within the image. The advanced interaction layer focuses on cross-regional feature interactions, establishing cross-regional feature dependencies through a self-attention mechanism. The self-attention mechanism automatically focuses on important parts of a feature, thereby establishing long-range dependencies between features.

[0051] The primary interaction layer can use a convolutional neural network (CNN) to gradually extract features from the image through the convolution operation of the convolution layer and the downsampling operation of the pooling layer. In an embodiment of the present invention, a CNN is used to process the feature alignment matrix to capture the morphology-microenvironment association pattern in the local area, and the advanced interaction layer can use a Transformer model. In an embodiment of the present invention, the feature alignment matrix is regarded as a sequence and processed using a Transformer model to establish feature dependencies across regions. For example, for the feature alignment matrix, a CNN containing 3 convolutional layers and 2 pooling layers is first used to process it to capture the morphology-microenvironment association pattern in the local area. Then, the output of the CNN is input into the Transformer model, and the self-attention mechanism is used to establish feature dependencies across regions.

[0052] Step S250: Dynamic gate fusion is performed on the interactively processed features to generate a four-dimensional feature tensor including nuclear level morphological features, intercellular connection morphological features, matrix component microenvironment features, and immune infiltration microenvironment features.

[0053] Dynamic gated fusion dynamically adjusts feature weights based on their importance. In an embodiment of the present invention, dynamic gated fusion is performed on interactively processed features to fuse nuclear hierarchical morphological features, intercellular junction morphological features, matrix component microenvironment features, and immune infiltration microenvironment features to generate a four-dimensional feature tensor. This four-dimensional feature tensor contains comprehensive information about cells and their microenvironment, helping to improve the recognition accuracy of subsequent models. Dynamic gated fusion can specifically use a gated recurrent unit (GRU) to dynamically adjust the gating signal based on the input features, thereby controlling the feature fusion process. Specifically, the interactively processed features are first input into the GRU, which generates a gating signal based on the input features. The GRU then performs a weighted fusion of the features based on the gating signal to generate a four-dimensional feature tensor. For example, assume that the interactively processed features include nuclear hierarchical morphological features, intercellular junction morphological features, matrix component microenvironment features, and immune infiltration microenvironment features. These features are then input into a GRU. The GRU generates gating signals based on the input features. For example, for nuclear morphology, a weight coefficient of 0.4 is generated, for intercellular connection morphology, a weight coefficient of 0.3 is generated, for matrix component microenvironment features, a weight coefficient of 0.2 is generated, and for immune infiltration microenvironment features, a weight coefficient of 0.1 is generated. The features are then weighted and fused according to these weight coefficients to generate a four-dimensional feature tensor.

[0054] Step S260: Input the four-dimensional feature tensor into the compression excitation network, strengthen the morphological and microenvironmental feature channels that are sensitive to diagnosis through the channel attention mechanism, suppress redundant feature channels, and finally output the fusion feature vector after dimension compression.

[0055] The Squeeze-and-Excitation (SE) network is a network used for feature compression and feature selection. It uses a channel-attention mechanism to strengthen diagnostically sensitive feature channels and suppress redundant ones. The channel-attention mechanism automatically focuses on important portions of a feature channel and dynamically adjusts channel weights based on their importance. In this embodiment of the present invention, a four-dimensional feature tensor is input into the Squeeze-and-Excitation (SE) network to compress and select features, outputting a dimensionally compressed fused feature vector. The Squeeze-and-Excitation (SE) network can be implemented using the existing Squeeze-and-Excitation (SE) module. The SE module consists of two components: the Squeeze component and the Excitation component. The Squeeze component uses global average pooling to compress the spatial dimensions of the four-dimensional feature tensor into a scalar, obtaining global feature information for each channel. The Excitation component uses a fully connected layer and a sigmoid activation function to generate weight coefficients for each channel based on this global feature information. These weight coefficients are then multiplied by the original feature channels to strengthen diagnostically sensitive feature channels and suppress redundant ones. Finally, the processed features are dimensionally compressed, outputting a fused feature vector. For example, for a four-dimensional feature tensor, global average pooling is first used to compress its spatial dimensions into a scalar to obtain the global feature information of each channel. This global feature information is then input into a fully connected layer and the weight coefficient of each channel is obtained through the sigmoid activation function.

[0056] Step S300: Construct a deep convolutional network with a multi-stage training strategy, in which the first stage uses a cross-channel feature attention mechanism to optimize the convolution kernel parameters, the second stage enhances feature discriminability by generating adversarial samples, and the third stage uses knowledge distillation to compress the network size.

[0057] A deep convolutional network is a deep learning model based on a convolutional neural network that can automatically learn features in an image. A multi-stage training strategy divides the training process into multiple stages, with each stage using different training methods to achieve different training goals. In an embodiment of the present invention, a deep convolutional network with a multi-stage training strategy is constructed. In the first stage, a cross-channel feature attention mechanism is used to optimize the convolution kernel parameters, with the aim of making the convolution kernel pay more attention to feature channels that are sensitive to diagnosis. In the second stage, feature discriminability is enhanced by generating adversarial samples. Adversarial samples are samples that can mislead the model after being perturbed. By generating adversarial samples, the robustness and feature discrimination ability of the model can be improved. In the third stage, knowledge distillation is used to compress the network size. Knowledge distillation is a method of transferring knowledge from a large model to a small model. Knowledge distillation can reduce the size of the model without losing too much performance.

[0058] The cross-channel feature attention mechanism can be implemented using a method similar to the SE module. By calculating the spatial correlation between each feature channel and the nuclear atypia index, a channel enhancement coefficient matrix is generated. The convolution kernels are then weighted channel by channel based on this matrix to optimize the kernel parameters. Adversarial examples can be generated using gradient-based methods, such as the Fast Gradient Signed Method (FGSM), by perturbing the original examples. Knowledge distillation can be implemented using a teacher-student model, with a large model serving as the teacher and a small model as the student. Knowledge transfer is achieved by having the student learn from the teacher's output. For example, a deep convolutional network consisting of multiple convolutional and fully connected layers is constructed. In the first stage, a cross-channel feature attention mechanism is used to optimize the convolution kernel parameters, directing the kernels to focus more on diagnostically sensitive feature channels. In the second stage, adversarial examples are generated using the FGSM method and fed into the network for training along with the original examples to enhance feature discriminability. In the third stage, the large model serves as the teacher and the small model as the student. Knowledge distillation is used to transfer the teacher's knowledge to the student, compressing the network size.

[0059] As an implementation method, the multi-stage training strategy may specifically include the following steps S310 to S340:

[0060] Step S310: Input the initial parameters of the deep convolutional network into the first-stage channel optimization module, generate a channel enhancement coefficient matrix by calculating the spatial correlation between the activation map of each feature channel and the nuclear atypia index, weight the convolution kernel channel by channel based on the channel enhancement coefficient matrix, and output the first-stage optimization parameters.

[0061] In an embodiment of the present invention, the initial parameters of the deep convolutional network are input into the first-stage channel optimization module. The channel enhancement coefficient matrix is generated by calculating the spatial correlation between the activation maps of each feature channel and the nuclear atypia index. The feature channel activation map is the feature map output by the convolutional layer, which reflects the activation of each feature channel in the image. The nuclear atypia index is a metric extracted earlier to measure the degree of nuclear morphological abnormality. The channel enhancement coefficient matrix is a two-dimensional matrix, with each element representing the enhancement weight of the corresponding feature channel. The spatial correlation between the activation maps of each feature channel and the nuclear atypia index can be calculated using the mutual information method. Mutual information is a metric used to measure the statistical dependence between two random variables. In an embodiment of the present invention, the spatial correlation between the feature channel activation maps and the nuclear atypia index is measured by calculating the mutual information value between the feature channel activation maps and the nuclear atypia index. The specific implementation process includes the following steps: First, a set of activation maps output by each feature channel of the deep convolutional network is obtained. The activation map set contains the feature response strengths of different convolutional layers at different spatial locations. Then, a spatial distribution map corresponding to the nuclear atypia index is extracted. Each pixel position in the spatial distribution map stores the nuclear atypia index value at that location. Next, spatial normalization is performed on the activation map set, resizing the activation maps and spatial distribution maps of each feature channel to the same size and establishing a pixel-level position mapping relationship. Subsequently, the mutual information (MI) value of each feature channel in the spatial dimension is calculated based on the normalized activation maps and spatial distribution maps. The MI value represents the statistical dependence of the channel's activation intensity and heteromorphism index. A nonlinear transformation is performed on the MI values, setting MI values below a set threshold to zero while retaining positive responses. This generates preliminary weight coefficients for each feature channel. These preliminary weight coefficients are then smoothed and filtered based on the differences in weight coefficients between adjacent feature channels to eliminate isolated noise points and enhance spatial coherence, resulting in optimized weight coefficients. Based on these optimized weight coefficients, an initial channel enhancement coefficient matrix is generated, with each element in the matrix corresponding to the enhancement weight of a feature channel. Finally, coefficient values in the initial channel enhancement coefficient matrix that exceed a preset interval are clipped to the interval boundaries and rescaled to the target numerical range, resulting in a channel enhancement coefficient matrix. For example, for a deep convolutional network, we obtain a set of activation maps output by each feature channel and a spatial distribution map corresponding to the nuclear atypia index. We then spatially normalize the activation map to match the spatial distribution map size and establish a pixel-level position mapping. We then calculate the mutual information (MI) for each feature channel. For example, if the MI value for a particular feature channel is 0.6, after nonlinear transformation and thresholding, we obtain a preliminary weight coefficient of 0.6. This preliminary weight coefficient is then smoothed and filtered, resulting in an optimized weight coefficient of 0.65.Based on the optimized weight coefficients, an initial channel enhancement coefficient matrix is generated. Assuming that an element in the matrix is 1.2, which exceeds the preset interval [0, 1], it is clipped to 1 and rescaled to the target numerical range to obtain the channel enhancement coefficient matrix. Finally, the convolution kernel is weighted channel by channel according to the channel enhancement coefficient matrix, and the first-stage optimization parameters are output. Based on the above process, the first-stage channel optimization module may include a feature extraction unit, a correlation calculation unit, a weight generation unit, and a convolution kernel weighting unit. The feature extraction unit performs activation map set acquisition and spatial distribution map extraction. The correlation calculation unit performs spatial normalization and mutual information calculation. The weight generation unit performs nonlinear transformation, smoothing filtering, and matrix generation and adjustment. The convolution kernel weighting unit performs channel-by-channel weighting of the convolution kernel according to the channel enhancement coefficient matrix and outputs the first-stage optimization parameters. Regarding the actions performed by each unit, those skilled in the art can select a specific execution structure based on actual needs. For example, feature extraction can use convolutional layers, normalization can use bilinear interpolation, and smoothing filtering can use Gaussian filtering, etc., but the specific implementation is not limited.

[0062] As an embodiment, step S310, calculating the spatial correlation between each characteristic channel activation map and the nuclear atypia index to generate a channel enhancement coefficient matrix, may specifically include the following steps S311-S318:

[0063] Step S311: Obtain a set of activation maps output by each feature channel of the deep convolutional network. The activation map set contains the feature response strengths of different convolutional layers at different spatial positions.

[0064] The activation map is a feature map output by the convolution layer, which reflects the activation of each feature channel in the image. The activation maps of different convolutional layers contain feature information at different levels. The activation maps of shallow convolutional layers mainly reflect the local features of the image, while the activation maps of deep convolutional layers mainly reflect the global features of the image. In this embodiment of the present invention, the activation map set output by each feature channel of the deep convolutional network is obtained in order to collect the feature response strength of different convolutional layers at different spatial positions, in preparation for the subsequent calculation of spatial correlation.

[0065] To obtain a set of activation maps, an output layer is added after each convolutional layer in a deep convolutional network, and the output of each convolutional layer is stored to form a set of activation maps. For example, for a deep convolutional network with five convolutional layers, an output layer is added after each convolutional layer. When an image is input, the output of each convolutional layer is recorded to obtain five activation maps, which constitute the set of activation maps. Each activation map is a three-dimensional tensor consisting of three dimensions: feature channel, height, and width. The feature channel represents different feature types, and the height and width represent the spatial position of the image. Each element in the activation map represents the feature response strength of the corresponding feature channel at the corresponding spatial position.

[0066] Step S312: extracting the spatial distribution map corresponding to the cell nuclear atypia index, and storing the atypia index value of the cell nucleus at each pixel position in the spatial distribution map.

[0067] The nuclear atypia index is a previously extracted indicator used to measure the degree of nuclear morphological abnormality. The spatial distribution map is a two-dimensional image, where each pixel stores the nuclear atypia index value for that location. In this embodiment of the present invention, the spatial distribution map corresponding to the nuclear atypia index is extracted to correlate the nuclear atypia index with the spatial location of the image, providing a basis for subsequent calculation of spatial correlation.

[0068] The spatial distribution map can be extracted by mapping the nuclear atypia index to each pixel position in the image based on the previously extracted nuclear atypia index and the pixel position information of the image. For example, for a multi-channel pathology sample image, the atypia index of each nucleus has been calculated through the aforementioned steps, and the position information of each nucleus has also been determined. The atypia index of each nucleus is assigned to the spatial distribution map of the pixel position at which it is located. For pixel positions without nuclei, the atypia index value can be set to 0. This results in a spatial distribution map corresponding to the nuclear atypia index.

[0069] Step S313: performing a spatial normalization operation on the activation map set, adjusting the activation map and the spatial distribution map of each feature channel to the same size and establishing a pixel-level position mapping relationship.

[0070] Spatial normalization resizes images of different sizes to the same size, ensuring that the activation maps and spatial distribution maps of each feature channel are spatially consistent. In this embodiment of the present invention, spatial normalization is performed on the activation map set to resize the activation maps and spatial distribution maps of each feature channel to the same size and establish a pixel-level position mapping relationship to facilitate the subsequent calculation of spatial correlation.

[0071] As mentioned above, the spatial normalization operation can use the bilinear interpolation method to calculate the grayscale value of the scaled pixel by linearly interpolating the grayscale values of adjacent pixels. Specifically, for each activation map and spatial distribution map, the scaling ratio is calculated according to the target size, and then the activation map and the spatial distribution map are adjusted to the target size using the bilinear interpolation method. At the same time, a pixel-level position mapping relationship is established, that is, the correspondence between the pixel positions in each activation map and the spatial distribution map is recorded. For example, for an activation map and a spatial distribution map, the target size is 256×256 pixels. Calculate the scaling ratio of the activation map and the spatial distribution map, assuming that the original size of the activation map is 512×512 pixels and the scaling ratio is 0.5, and use the bilinear interpolation method to adjust the activation map to 256×256 pixels. Similarly, the spatial distribution map is also adjusted to 256×256 pixels. At the same time, a correspondence between the pixel positions in the activation map and the spatial distribution map is established, such that the pixel at position (100, 100) in the activation map corresponds to the pixel at position (100, 100) in the spatial distribution map.

[0072] Step S314: Based on the normalized activation map and spatial distribution map, the mutual information value of each feature channel in the spatial dimension is calculated. The mutual information value represents the statistical dependence of the activation intensity of the channel and the heteromorphism index.

[0073] Mutual information is a metric used to measure the statistical dependence between two random variables. In this embodiment of the present invention, the mutual information value of each feature channel in the spatial dimension is calculated based on the normalized activation map and spatial distribution map. The purpose is to measure the statistical dependence between the activation intensity of each feature channel and the nuclear atypia index. The larger the mutual information value, the stronger the correlation between the activation intensity of the feature channel and the nuclear atypia index.

[0074] Mutual information can be calculated using the mutual information formula from information theory (a general formula not detailed here). Specifically, the normalized activation map and spatial distribution map are treated as two random variables. Their pixel values are discretized into several intervals. The number of pixels in each interval is then counted, and the joint probability distribution and marginal probability distribution are calculated. Finally, the mutual information value is calculated using the mutual information formula.

[0075] Step S315: performing a nonlinear transformation operation on the mutual information values, setting the mutual information values whose absolute values are lower than the set threshold to zero and retaining the positive responses, and generating preliminary weight coefficients for each feature channel.

[0076] Nonlinear transformations can alter the distribution and characteristics of data. In this embodiment of the present invention, a nonlinear transformation is performed on the mutual information values to remove noise information whose absolute value falls below a set threshold, retain positive responses that are important for diagnosis, and generate preliminary weight coefficients for each feature channel.

[0077] Nonlinear transformations can be performed using thresholding, for example. A threshold is set, and the mutual information values below the threshold are set to zero, retaining positive responses. For example, if the threshold is set to 0.1, a feature channel with a mutual information value of 0.05 is set to zero, while a feature channel with a mutual information value of 0.2 is retained. After thresholding, preliminary weight coefficients are obtained for each feature channel.

[0078] Step S316: For the preliminary weight coefficients, smoothing filtering is performed according to the differences in weight coefficients of adjacent feature channels to eliminate isolated noise points and enhance spatial coherence, thereby obtaining optimized weight coefficients.

[0079] Smoothing filtering is used to remove noise from data, and can make the data smoother by averaging or weighted averaging adjacent data points. In an embodiment of the present invention, for the preliminary weight coefficient, smoothing filtering is performed according to the difference in weight coefficients of adjacent feature channels, with the aim of eliminating isolated noise points in the preliminary weight coefficient, enhancing spatial coherence, and obtaining optimized weight coefficients. Smoothing filtering can adopt the Gaussian filtering method to perform weighted averaging according to the distance and weight of adjacent data points. Specifically, for the preliminary weight coefficient of each feature channel, the weighted average is calculated using a Gaussian function according to the difference in weight coefficients of its adjacent feature channels to obtain the optimized weight coefficient.

[0080] Step S317: Generate an initial channel enhancement coefficient matrix based on the optimized weight coefficients, where each element in the initial channel enhancement coefficient matrix corresponds to the enhancement weight of the feature channel.

[0081] In an embodiment of the present invention, an initial channel enhancement coefficient matrix is generated based on the optimized weight coefficients. The purpose is to organize the optimized weight coefficients into a matrix. The method for generating the initial channel enhancement coefficient matrix can be to arrange the optimized weight coefficients into a matrix in the order of feature channels. For example, assuming that the deep convolutional network has 100 feature channels, the optimized weight coefficients are w1, w2, ..., w100 respectively. These weight coefficients are arranged into a 1×100 matrix as the initial channel enhancement coefficient matrix.

[0082] Step S318: The coefficient values of the elements in the initial channel enhancement coefficient matrix that exceed the preset interval are clipped to the interval boundary and rescaled to the target value range to obtain the channel enhancement coefficient matrix.

[0083] The preset interval is used to limit the value of the channel enhancement coefficient. In an embodiment of the present invention, the element coefficient values in the initial channel enhancement coefficient matrix that exceed the preset interval are clipped to the interval boundary and rescaled to the target numerical range. The purpose is to ensure that the element coefficient values in the channel enhancement coefficient matrix are within a reasonable range and avoid the occurrence of excessively large or small coefficient values that affect the weighting effect of the convolution kernel. The method of clipping and rescaling can be: first determine a preset interval, such as [0, 1]. For the element coefficient values in the initial channel enhancement coefficient matrix that exceed the interval, they are clipped to the interval boundary, such as element coefficient values greater than 1 are clipped to 1, and element coefficient values less than 0 are clipped to 0. Then, the clipped matrix is rescaled to the target numerical range, such as linearly mapping the element coefficient values in the matrix to the range of [0.1, 0.9]. For example, for an initial channel enhancement coefficient matrix, where a certain element coefficient value is 1.2, it is clipped to 1. Assuming the target numerical range is [0.1, 0.9], the element coefficient values in the clipped matrix are linearly mapped to this range to obtain the channel enhancement coefficient matrix.

[0084] Step S320: Input the first-stage optimization parameters into the second-stage adversarial training module, and generate a perturbation sample set based on the optimized convolution kernel. The perturbation samples in the perturbation sample set are constructed by adjusting the spatial co-occurrence pattern of the intercellular connection density index and the immune cell aggregation density, and maintaining the continuity constraint of the matrix fiber orientation characteristics.

[0085] The second-stage adversarial training module is used to generate adversarial samples and conduct adversarial training. In this embodiment of the present invention, the first-stage optimized parameters are input into the second-stage adversarial training module, and a set of perturbed samples is generated based on the optimized convolution kernel. Perturbed samples are samples that, after perturbation, can mislead the model. Generating perturbed samples can improve the model's robustness and feature discrimination capabilities.

[0086] Adjusting the spatial co-occurrence pattern of the intercellular connectivity density index and immune cell aggregation density means adjusting the spatial distribution of the intercellular connectivity density index and immune cell aggregation density in the image to make them different from the original sample. Maintaining the continuity constraint of the matrix fiber orientation characteristics means maintaining the continuity of the matrix fiber orientation characteristics during the process of generating the perturbed sample to avoid unreasonable changes.

[0087] The method for generating perturbed samples can be a gradient-based method, such as the fast gradient sign method (FGSM). Specifically, the original sample is first input into a deep convolutional network, and the gradient of the loss function with respect to the input sample is calculated. Then, according to the sign of the gradient and the preset perturbation strength, the original sample is perturbed to generate a perturbed sample. During the perturbation process, the spatial co-occurrence pattern of the intercellular connection density index and the immune cell aggregation density is adjusted, and the continuity constraint of the matrix fiber orientation characteristics is maintained. For example, for an original sample, it is input into a deep convolutional network, and the gradient of the loss function with respect to the input sample is calculated. Assuming the perturbation strength is 0.01, the original sample is perturbed according to the sign of the gradient, and the spatial co-occurrence pattern of the intercellular connection density index and the immune cell aggregation density is adjusted, and the continuity constraint of the matrix fiber orientation characteristics is maintained to generate a perturbed sample. This process is repeated to generate a set of perturbed samples.

[0088] It can be understood that based on the above description, those skilled in the art can choose the corresponding implementation method to implement the architecture of the second-stage adversarial training module according to actual needs. For example, a generator-discriminator structure can be adopted. The generator is based on the convolution kernel optimized in the first stage, and uses the gradient ascent method to generate a set of perturbation samples, adjust the spatial co-occurrence pattern of the intercellular connection density index and the immune cell aggregation density, and maintain the continuity of the matrix fiber direction characteristics; the discriminator is a deep convolutional network, which updates the fully connected layer parameters using the stochastic gradient descent method by comparing the distribution offset of the original sample and the perturbation sample in the microenvironment feature space.

[0089] Step S330: Alternately input the perturbation sample set and the original sample into the deep convolutional network, update the fully connected layer parameters by comparing the distribution offset of the original features and the perturbation features in the microenvironment feature space, and output the second stage optimization parameters.

[0090] In this embodiment of the present invention, a set of perturbed samples is alternately fed into a deep convolutional network alongside the original samples. This allows the model to simultaneously learn the features of both the original and perturbed samples, improving the model's robustness and feature discrimination capabilities. The distribution offset of the features is measured by comparing the distribution offsets of the original and perturbed features in the microenvironment feature space. This distance is calculated between the feature vectors of the original and perturbed samples in the microenvironment feature space.

[0091] The parameters of the fully connected layer can be updated using the stochastic gradient descent (SGD) algorithm, which continuously updates the model parameters to gradually reduce the loss function. Specifically, a set of perturbed samples is first fed alternately into the deep convolutional network with the original samples, and the distribution offset between the original and perturbed features in the microenvironment feature space is calculated. The loss function is then calculated based on this distribution offset, and the fully connected layer parameters are updated using the SGD algorithm. For example, for a deep convolutional network, a set of perturbed samples is fed alternately into the network with the original samples, and the distribution offset between the original and perturbed features in the microenvironment feature space is calculated. Assuming the distribution offset is 0.5, the loss function is calculated based on this offset, and the fully connected layer parameters are updated using the SGD algorithm. This process is repeated until the loss function converges, and the optimized parameters for the second stage are output.

[0092] Step S340: Input the second-stage optimization parameters into the third-stage compression module, match the feature activation path differences between the teacher network and the student network during the nuclear atypia index extraction process, use the channel importance ranking algorithm to remove redundant channels, and output the lightweight third-stage optimization parameters.

[0093] The third-stage compression module is used to compress the network size. In an embodiment of the present invention, the second-stage optimization parameters are input into the third-stage compression module. By matching the differences in feature activation paths between the teacher network and the student network during the extraction of the nuclear atypia index, a channel importance ranking algorithm is used to remove redundant channels. The purpose is to reduce the size of the network without sacrificing too much performance. The teacher network is a large deep convolutional network with high performance. The student network is a small deep convolutional network with a smaller scale. The channel importance ranking algorithm is an algorithm for evaluating channel importance. It can sort channels according to their contribution to model performance, thereby identifying redundant channels.

[0094] A method for matching the differences in feature activation paths between the teacher and student networks during nuclear atypia index extraction can be to calculate the difference between the feature activation maps of the two networks under the same input. A channel importance ranking algorithm can employ a gradient-based approach, such as calculating the gradient of each channel with respect to the loss function and ranking the channels based on the magnitude of the gradient. Specifically, the second-stage optimization parameters are first loaded into the teacher and student networks, respectively. The same sample is fed into the network, and the difference between the feature activation maps of the two networks during nuclear atypia index extraction is calculated. Then, the student network's channels are ranked using a channel importance ranking algorithm to identify redundant channels. Finally, redundant channels are removed, and lightweight third-stage optimization parameters are output. For example, for a teacher network and a student network, the second-stage optimization parameters are loaded into the two networks, a multi-channel pathology sample image is fed into the network, and the difference between the feature activation maps of the two networks during nuclear atypia index extraction is calculated. The student network's channels are ranked using a gradient-based channel importance ranking algorithm. Assuming that the last 20% of channels contribute little to model performance after ranking, these channels are removed, and the lightweight third-stage optimization parameters are output. It is understandable that, based on the above description, those skilled in the art can select a corresponding implementation method to implement the architecture of the third-stage compression module according to actual needs, such as using a teacher-student network structure.

[0095] The above three stages adopt a progressive parameter transfer mechanism, in which the convolution kernel of the second stage inherits the channel weighted result of the optimized parameters in the first stage, and the fully connected layer of the third stage inherits the spatial distribution pattern of the optimized parameters in the second stage. The training samples in each stage dynamically adjust the sampling ratio according to the recognition error of the cell membrane curvature parameters in the previous stage.

[0096] The progressive parameter transfer mechanism is a mechanism in which, during a multi-stage training process, the optimized parameters of the previous stage are transferred to the next stage, serving as the initial parameters for the next stage's training. In this embodiment of the present invention, the convolution kernels of the second stage inherit the channel-weighted results of the optimized parameters of the first stage, i.e., the convolution kernels of the second stage use the convolution kernel parameters optimized in the first stage. The fully connected layer of the third stage inherits the spatial distribution pattern of the optimized parameters of the second stage, i.e., the fully connected layer of the third stage uses the fully connected layer parameters optimized in the second stage.

[0097] The sampling ratio of training samples in each stage is dynamically adjusted based on the recognition error of cell membrane curvature parameters in the previous stage. Specifically, the sampling ratio of different sample types in the current stage's training samples is adjusted based on the recognition error of the cell membrane curvature parameters in the previous stage. If the recognition error of the cell membrane curvature parameters for a certain type of sample in the previous stage is large, the sampling ratio of that type of sample in the current stage's training samples is increased to improve the model's recognition ability for that type of sample. For example, after the first stage of training, if the model's recognition error for the cell membrane curvature parameters of epithelial tumor cells is large, the sampling ratio of epithelial tumor cells in the training samples is increased in the second stage of training.

[0098] As an implementation method, the process of generating the adversarial sample may specifically include the following steps S21-S26:

[0099] Step S21: extracting the feature gradient distribution of the deep convolutional network on the multi-channel pathology sample image set, and identifying the cross-modal gradient correlation pattern between the nuclear atypia index and the immune cell aggregation density based on the feature gradient distribution.

[0100] The feature gradient distribution is the gradient distribution of the loss function of a deep convolutional network when it is fed a set of multi-channel pathology sample images. The cross-modal gradient correlation pattern is the correlation pattern between the nuclear atypia index and immune cell aggregation density within the feature gradient distribution. In this embodiment of the present invention, the feature gradient distribution of a deep convolutional network on a set of multi-channel pathology sample images is extracted with the goal of identifying the cross-modal gradient correlation pattern between the nuclear atypia index and immune cell aggregation density by analyzing the feature gradient distribution.

[0101] Extracting feature gradient distributions involves inputting a multi-channel set of pathology sample images into a deep convolutional network, calculating a loss function, and then using an automatic differentiation tool to calculate the gradient of the loss function with respect to the input image. Based on the feature gradient distributions, cross-modal gradient correlation patterns are identified by analyzing the relationship between the gradient values corresponding to the nuclear atypia index and the immune cell aggregation density. For example, by calculating the correlation between the gradient values corresponding to the nuclear atypia index and the immune cell aggregation density, correlation patterns between them can be identified.

[0102] Step S22: generating a perturbation direction template according to the cross-modal gradient correlation pattern, wherein the perturbation direction template includes pixel offset directions that produce synergistic gradient changes in cell nucleus morphological parameters and matrix fiber orientation characteristics.

[0103] The perturbation direction template is used to guide the generation of perturbation samples. It contains pixel offset directions that produce synergistic gradient changes in nuclear morphology parameters and matrix fiber orientation characteristics. In this embodiment of the present invention, the perturbation direction template is generated based on the cross-modal gradient correlation pattern to provide guidance for the subsequent generation of perturbation samples, enabling the perturbation samples to maintain the synergistic change relationship between nuclear morphology parameters and matrix fiber orientation characteristics while changing them.

[0104] A method for generating a perturbation direction template can be to identify pixel locations and offset directions that produce synergistic gradient changes in nuclear morphological parameters and matrix fiber orientation characteristics based on cross-modal gradient correlation patterns. Specifically, the cross-modal gradient correlation patterns are analyzed to identify pixel locations where the gradient values corresponding to nuclear morphological parameters and matrix fiber orientation characteristics simultaneously increase or decrease. The offset directions of these pixels are recorded to generate a perturbation direction template. For example, for a cross-modal gradient correlation pattern, when the gradient value of a pixel simultaneously affects both nuclear morphological parameters and matrix fiber orientation characteristics, the offset direction of the pixel is recorded. This process is repeated to generate a perturbation direction template.

[0105] Step S23: Input the multi-channel pathological sample image into the deep convolutional network, extract the spatial gradient response map of the fused feature vector, and generate an initial perturbation mask based on the distribution similarity between the spatial gradient response map and the perturbation direction template.

[0106] The spatial gradient response map is the gradient distribution of the fused feature vector in the spatial dimension. The initial perturbation mask is a binary mask that indicates which pixels need to be perturbed. In an embodiment of the present invention, a multi-channel pathology sample image is input into a deep convolutional network, and the spatial gradient response map of the fused feature vector is extracted. The initial perturbation mask is generated based on the distribution similarity between the spatial gradient response map and the perturbation direction template. The purpose is to determine which pixels need to be perturbated to generate valid perturbation samples.

[0107] The method for extracting the spatial gradient response map can be to use an automatic differentiation tool, such as an autograd block. Specifically, the multi-channel pathological sample image is input into a deep convolutional network, the fused feature vector is calculated, and then the gradient of the fused feature vector to the input image is calculated using the automatic differentiation tool to obtain the spatial gradient response map. Based on the distribution similarity between the spatial gradient response map and the perturbation direction template, by calculating the correlation between them, it is determined which pixels have a gradient distribution similar to the perturbation direction template, and the positions corresponding to these pixels are set to 1, and the other positions are set to 0 to generate an initial perturbation mask. For example, for a spatial gradient response map and a perturbation direction template, the correlation between them is calculated. Assuming the correlation threshold is 0.8, the pixel positions with a correlation greater than 0.8 are set to 1, and the other positions are set to 0 to generate an initial perturbation mask.

[0108] Step S24: performing a morphological boundary constraint operation on the initial perturbation mask, eliminating isolated perturbation points that have no intersection with the cytoplasm region boundary, and retaining the perturbation region that overlaps with the cell nucleus contour boundary data space.

[0109] Morphological boundary constraint operations are used to process binary masks, adjusting the mask's shape and boundaries through morphological transformations. In an embodiment of the present invention, a morphological boundary constraint operation is performed on the initial perturbation mask to eliminate isolated perturbation points that do not intersect with the cytoplasm region boundary, while retaining the perturbation region that overlaps with the nuclear outline boundary data space, making the perturbation sample more reasonable and effective. Morphological boundary constraint operations can employ dilation and erosion operations. Dilation can expand the white area in the binary mask, while erosion can reduce the white area. Specifically, a dilation operation is first performed on the initial perturbation mask to expand the perturbation region. An intersection operation is then performed with the cytoplasm region boundary to eliminate isolated perturbation points that do not intersect with the cytoplasm region boundary. Next, an erosion operation is performed on the processed mask to restore the perturbation region to its original size. Finally, an intersection operation is performed with the nuclear outline boundary data space to retain the perturbation region that overlaps with the nuclear outline boundary data space. For example, for an initial perturbation mask, a dilation operation is used to expand the perturbation region, followed by an intersection operation with the cytoplasm region boundary to eliminate isolated perturbation points. Next, the erosion operation is used to restore the disturbed area to its original size, and finally an intersection operation is performed with the cell nucleus contour boundary data to retain the disturbed area that overlaps with the cell nucleus contour boundary data space.

[0110] Step S25: The retained disturbance area is superimposed on the original multi-channel pathological sample image to generate a disturbance image and input it into the deep convolutional network to calculate the consistency index of the change direction of the nuclear atypia index before and after the disturbance and the change direction of the immune cell aggregation density.

[0111] The consistency index is the degree of consistency between the direction of change in the nuclear atypia index before and after the perturbation and the direction of change in the immune cell aggregation density. In this embodiment of the present invention, the retained perturbed region is superimposed on the original multi-channel pathology sample image to generate a perturbed image, which is then input into a deep convolutional network. The consistency index between the direction of change in the nuclear atypia index before and after the perturbation and the direction of change in the immune cell aggregation density before and after the perturbation is calculated. The purpose is to evaluate the effectiveness of the perturbed sample and ensure that the perturbed sample has the expected impact on the model.

[0112] The retained perturbed region is superimposed on the original multi-channel pathology sample image by adjusting the pixel values of the pixels corresponding to the retained perturbed region in the original image. For example, if the pixel values of the pixel positions corresponding to the perturbed region need to be increased by a fixed offset, the pixel values of these pixels in the original image are added to the offset to generate the perturbed image.

[0113] The perturbed image is fed into a deep convolutional network to extract the nuclear atypia index and immune cell aggregation density again. Calculating the consistency between the change direction of the nuclear atypia index before and after the perturbation and the change direction of the immune cell aggregation density can be achieved in the following way. First, calculate the difference between the nuclear atypia index before and after the perturbation. The difference between the density of immune cells and .like If it is positive, it means that the nuclear atypia index increases; if it is negative, it means that it decreases. There are also corresponding positive and negative values to indicate the increase or decrease of immune cell aggregation density. The consistency index can be defined as: and If the two values have the same sign (i.e. both are positive or both are negative), the consistency index is 1; if they have different signs, it is -1; if one of the differences is 0, a smaller consistency index value, such as 0, can be set according to the specific situation.

[0114] Step S26: Dynamically adjust the intensity distribution of the perturbation direction template based on the consistency index to generate an adversarial sample set, where the perturbation intensity of each sample in the adversarial sample set is inversely correlated with the local change rate of the cell membrane curvature parameter.

[0115] Dynamically adjusting the intensity distribution of the perturbation direction template is intended to make the generated perturbation samples more effective and better able to mislead the model. Based on the consistency index, if the consistency index is high, it means that the current perturbation direction template can effectively change the nuclear atypia index and immune cell aggregation density, and the changes in the two are consistent. In this case, the intensity of the perturbation direction template can be appropriately increased. If the consistency index is low, the intensity of the perturbation direction template needs to be reduced and its direction adjusted.

[0116] A specific method for adjusting the intensity distribution of the perturbation direction template is to set a base intensity value S0 and calculate the adjusted intensity value S based on the consistency index C. For example, a linear adjustment method can be used: S = S0 + k × C, where k is an adjustment coefficient. If C = 1, the intensity is increased; if C = -1, the intensity is decreased.

[0117] When generating an adversarial sample set, the perturbation intensity of each sample is inversely correlated with the local rate of change of the cell membrane curvature parameter. The local rate of change of the cell membrane curvature parameter can be calculated by calculating the curvature difference between adjacent pixels on the cell membrane. Regions with large local rates of change indicate more dramatic changes in the cell membrane morphology. In these cases, the perturbation intensity should be reduced to avoid excessive perturbation and sample distortion. Regions with smaller local rates of change can be appropriately increased. The adjusted perturbation intensity is multiplied by the perturbation direction template and superimposed on the original sample to generate an adversarial sample. This process is repeated until a sufficient number of adversarial samples are generated to form an adversarial sample set.

[0118] As an implementation method, the above-mentioned knowledge distillation and compression process may specifically include the following steps S31 to S36:

[0119] Step S31: Use the convolutional layer parameters of the deep convolutional network as the teacher network parameters, and initialize the convolutional layer parameters of the student network so that its channel dimension matches the teacher network parameters but the number of channels is reduced.

[0120] The convolutional layer parameters of the deep convolutional network are used as the parameters of the teacher network. Specifically, the weights and bias parameters of the trained deep convolutional network are saved and used as the parameters of the teacher network. When initializing the convolutional layer parameters of the student network, its channel dimensions must match those of the teacher network. This means that the student and teacher networks have the same dimension of feature channels, but the number of channels in the student network is reduced, which can reduce the size of the model to a certain extent. Random initialization can be used to initialize the convolutional layer parameters of the student network, such as using the Xavier initialization method, which can ensure good convergence of the network in the early stages of training.

[0121] Step S32: extracting a set of intermediate feature activation maps of the teacher network on the multi-channel pathological sample image, where the set of intermediate feature activation maps includes the spatial response distribution of cell morphological features and microenvironment features in different convolutional layers.

[0122] Intermediate feature activation maps are feature maps output by the teacher network at different convolutional layers. These feature maps reflect the spatial distribution of cell morphological and microenvironmental features across these layers. The purpose of extracting a set of intermediate feature activation maps is to provide learning targets for the student network, allowing it to learn the teacher network's feature representation when processing multi-channel pathology sample images. The process of extracting a set of intermediate feature activation maps involves inputting a multi-channel pathology sample image into the teacher network and recording the feature maps at the output of each convolutional layer. Because feature maps from different convolutional layers have different hierarchical and semantic information, feature maps from shallow convolutional layers primarily reflect local image features, such as edges and texture, while feature maps from deeper convolutional layers primarily reflect global image features, such as object shape and category. Therefore, the set of intermediate feature activation maps encompasses the spatial distribution of cell morphological and microenvironmental features at different hierarchical levels. For example, for a teacher network, a multi-channel pathology sample image is input, and the output feature maps from each convolutional layer are recorded. These feature maps constitute the set of intermediate feature activation maps.

[0123] Step S33: perform cross-layer spatial alignment on the intermediate feature activation map of the student network and the intermediate feature activation map of the teacher network, compensate for the resolution difference between the two through void convolution and generate a feature similarity distribution map.

[0124] Cross-layer spatial alignment involves spatially aligning the intermediate feature activation maps output by the student and teacher networks at different convolutional layers in order to compare their similarities. Because the student and teacher networks may have different structures and resolutions, dilated convolutions are needed to compensate for this resolution difference.

[0125] Atrous convolution is a convolution operation that introduces a hole in the convolution kernel. It can expand the receptive field of the convolution kernel without increasing the number of parameters, thereby improving the resolution of the feature map. Specifically, the intermediate feature activation maps of the student network and the teacher network are matched in the order of the convolution layer. For feature maps with different resolutions, the feature map of the student network is upsampled using atrous convolution so that its resolution is the same as that of the feature map of the teacher network. Then, the similarity between the corresponding feature maps of the student network and the teacher network is calculated to generate a feature similarity distribution map. The similarity can be calculated by the mean square error (MSE), that is, calculating the average of the squares of the difference between the corresponding pixel values of the two feature maps. For example, for the feature map F output by the student network and the teacher network at a certain convolution layer, s and F t , calculate their mean square error , where N is the number of pixels in the feature map. The mean square error of each pixel is taken as the value of the corresponding pixel in the feature similarity distribution map.

[0126] Step S34: Dynamically adjust the channel attention coefficient of the student network based on the feature similarity distribution map to suppress channels with significant differences in response patterns from the teacher network and enhance the weights of spatially consistent channels.

[0127] The channel attention coefficient is used to adjust the importance of different feature channels in the student network. The channel attention coefficient of the student network is dynamically adjusted based on the feature similarity distribution map. The purpose is to make the student network pay more attention to the channels that are consistent with the response pattern of the teacher network and suppress the channels that have significant differences from the response pattern of the teacher network.

[0128] The process of dynamically adjusting the channel attention coefficient is, for example, to first calculate the average similarity score of each channel based on the feature similarity distribution map. For each channel, the pixel values of the feature similarity distribution map on the channel are averaged to obtain the average similarity score of the channel. Then, the channel attention coefficient is adjusted according to the average similarity score. For channels with higher average similarity scores, it means that the response pattern of the channel is more consistent with that of the teacher network, and its channel attention coefficient is enhanced; for channels with lower average similarity scores, it means that there is a significant difference in the response pattern of the channel with that of the teacher network, and its channel attention coefficient is suppressed. For example, a threshold T is set, and for channels with an average similarity score greater than T, its channel attention coefficient is multiplied by a coefficient greater than 1, such as 1.2; for channels with an average similarity score less than T, its channel attention coefficient is multiplied by a coefficient less than 1, such as 0.8.

[0129] Step S35: Input the compressed feature vector of the student network into the classifier, calculate the difference loss between it and the classification probability distribution of the teacher network, and generate the adaptive distillation loss coefficient in combination with the cell type annotation information.

[0130] The compressed feature vector of the student network is input into the classifier to obtain the classification probability distribution of the student network. The classification probability distribution of the teacher network is obtained by inputting the same samples into the classifier of the teacher network.

[0131] The difference loss between the classification probability distribution of the student network and the teacher network can be calculated using the KL divergence (Kullback-Leibler divergence), which represents the information loss of one probability distribution relative to another. For example, for the classification probability distribution P of the student network s And the classification probability distribution P of the teacher network t ,calculate .

[0132] The purpose of generating the adaptive distillation loss coefficient by combining the cell type annotation information is to make the distillation loss more reasonable. The cell type annotation information is the true category label of each sample. The weight of the distillation loss can be dynamically adjusted according to the difference between the sample category and the classification probability distribution. For example, for samples that are misclassified, the weight of the distillation loss is increased; for samples that are correctly classified, the weight of the distillation loss is reduced. The adaptive distillation loss coefficient can be calculated by a function g, for example, , a is the preset adjustment coefficient.

[0133] Step S36: Iteratively update the fully connected layer parameters of the student network according to the adaptive distillation loss coefficient until the performance difference between the comprehensive performance index of the student network and the teacher network under the layered cross-validation mechanism is stabilized within a preset range, and the compression process is terminated.

[0134] Iteratively updating the parameters of the fully connected layer of the student network is achieved through optimization algorithms such as stochastic gradient descent (SGD) and adaptive moment estimation (Adam). In each iteration, the loss function is calculated based on the adaptive distillation loss coefficient, and then the optimization algorithm is used to update the weights and bias of the fully connected layer.

[0135] The stratified cross-validation mechanism is used to evaluate model performance. The dataset is stratified by category, and then divided into multiple subsets. Each time, one subset is used as the validation set, and the remaining subsets are used as the training set. This process is repeated multiple times to obtain the performance indicators of the model on different subsets. Finally, these indicators are combined to obtain the comprehensive performance indicators of the model.

[0136] The preset range is a pre-set performance difference threshold. When the performance difference between the student network's comprehensive performance indicators and the teacher network under the stratified cross-validation mechanism stabilizes within this range, it means that the student network has learned most of the knowledge of the teacher network, and the compression process can be terminated at this time. For example, the preset range can be set to a difference of less than 5% between the accuracy of the student network and the accuracy of the teacher network. After each iteration of updating the fully connected layer parameters of the student network, the stratified cross-validation mechanism is used to evaluate the performance of the student network and compare it with the performance of the teacher network. When the performance difference between the two stabilizes within the preset range, training is stopped, and the knowledge distillation compression process is completed.

[0137] Step S400: A hierarchical cross-validation mechanism is used to verify and adjust the network parameters of the deep convolutional network during the training process, wherein the hierarchical cross-validation mechanism divides the biological characteristics corresponding to the cell type annotation information into multiple validation subsets, and adjusts the network parameters based on the classification error of each validation subset.

[0138] The Stratified Cross-Validation mechanism can fully consider the category distribution of the data and avoid evaluation bias caused by data imbalance. In this embodiment of the present invention, the biological characteristics corresponding to the cell type annotation information are divided into multiple validation subsets because different cell types have different biological characteristics. For example, epithelial tumor cells and mesenchymal tumor cells differ in morphology and function. Stratifying the data by cell type can ensure that the data in each validation subset has similar biological characteristics, thereby more accurately evaluating the performance of the model on different cell types.

[0139] Adjusting network parameters based on the classification error of each validation subset is designed to ensure that the model performs well on different cell types. Classification error is the difference between the model's classification of a sample and the true label. By calculating the classification error for each validation subset, we can understand the model's performance on different cell types. If the classification error for a particular validation subset is large, it indicates that the model's performance on that cell type is poor, and network parameters need to be adjusted to improve the model's ability to recognize that cell type.

[0140] As an implementation method, step S400 uses a layered cross-validation mechanism to verify and adjust the network parameters of the deep convolutional network during training, which may specifically include the following steps S410-S450:

[0141] Step S410: dividing the multi-channel pathology sample image set into an epithelial-derived tumor cell subset, a mesenchymal-derived tumor cell subset, and a tumor cell subset with an undetermined degree of differentiation according to the biological characteristics corresponding to the cell type annotation information.

[0142] Epithelial tumor cells are tumor cells that originate from epithelial tissue and possess specific morphological and biological characteristics, such as regular cell arrangement and distinct polarity. Mesenchymal tumor cells are tumor cells that originate from mesenchymal tissue and have different morphological and biological characteristics from epithelial tumor cells, such as diverse cell morphology and rich mesenchymal components. Undifferentiated tumor cells are those whose degree of differentiation cannot be clearly determined and have more complex biological characteristics.

[0143] Classification is performed based on the biological characteristics corresponding to the cell type annotation information. Specifically, the cell type annotation information of each sample in the multi-channel pathology sample image set can be first analyzed to determine whether it belongs to epithelial tumor cells, mesenchymal tumor cells, or tumor cells with an undetermined degree of differentiation.

[0144] Step S420: In each round of training iteration, the verification operation is performed in sequence in the order of epithelial tumor cell subset, mesenchymal tumor cell subset, and tumor cell subset with undetermined differentiation degree. The verification operation of each subset includes extracting the nuclear atypia index classification error and immune cell aggregation density regression error of the current network parameters in the corresponding subset.

[0145] Validation operations are performed sequentially in each training iteration to comprehensively evaluate the model's performance on different cell type subsets. The nuclear atypia index classification error is the difference between the model's classification result for the nuclear atypia index and the true classification. The immune cell cluster density regression error is the difference between the model's predicted value for the immune cell cluster density and the true value. The nuclear atypia index classification error can be calculated by, for example, inputting samples from the corresponding subset into the current network, obtaining the model's classification result for the nuclear atypia index, and then comparing it with the true classification to calculate the classification error. For example, the cross-entropy loss function can be used to calculate the classification error, which measures the difference between the model's classification result and the true label. The immune cell cluster density regression error can be calculated by inputting samples from the corresponding subset into the current network, obtaining the model's predicted value for the immune cell cluster density, and then comparing it with the true value to calculate the regression error. For example, the mean squared error (MSE) can be used to calculate the regression error, which measures the average error between the predicted value and the true value.

[0146] Step S430: The weighted sum of the classification error and the regression error of the epithelial-derived tumor cell subset is used as the first verification indicator, the weighted sum of the classification error and the regression error of the mesenchymal-derived tumor cell subset is used as the second verification indicator, and the weighted sum of the classification error and the regression error of the tumor cell subset with an undetermined degree of differentiation is used as the third verification indicator.

[0147] Weighted summation involves multiplying the classification error and regression error by their respective weights and then adding them together to produce a composite metric. Setting different weights can adjust the importance of classification error and regression error in the validation metric based on actual needs. For example, for an epithelial tumor cell subset, assuming the classification error has a weight of 0.6 and the regression error has a weight of 0.4, we multiply the classification error by 0.6 and the regression error by 0.4, then add them together to produce the first validation metric. Similarly, the second and third validation metrics are calculated.

[0148] Step S440: Dynamically adjust the learning rate and loss function weight distribution ratio of the fully connected layer of the deep convolutional network based on the difference between the first verification indicator and the second verification indicator, and adjust the channel attention coefficient update step size of the convolutional layer based on the absolute value of the third verification indicator.

[0149] Dynamically adjusting the learning rate of the fully connected layer and the weight distribution ratio of the loss function is intended to enable the model to better learn on different cell subsets. If the difference between the first and second validation metrics is large, it indicates that the model's performance on the epithelial and mesenchymal tumor cell subsets differs significantly. Adjusting the learning rate of the fully connected layer and the weight distribution ratio of the loss function will be necessary to balance the model's performance on these two subsets. For example, if the first validation metric is greater than the second validation metric, it indicates that the model's performance on the epithelial tumor cell subset is poor. The learning rate of the fully connected layer on the epithelial tumor cell subset can be appropriately increased, while adjusting the weight distribution ratio of the loss function to ensure that the model focuses more on the classification and regression errors of the epithelial tumor cell subset.

[0150] Adjusting the channel attention coefficient update step size of the convolutional layer based on the absolute value of the third verification metric is intended to enable the model to better handle subsets of tumor cells with uncertain differentiation levels. If the absolute value of the third verification metric is large, it indicates that the model performs poorly on the subset of tumor cells with uncertain differentiation levels. It is necessary to increase the channel attention coefficient update step size of the convolutional layer to make the model pay more attention to feature channels related to tumor cells with uncertain differentiation levels. For example, an update step size adjustment function h is set, whose input is the absolute value of the third verification metric and whose output is the channel attention coefficient update step size of the convolutional layer. When the absolute value of the third verification metric is large, h outputs a larger update step size; when the absolute value of the third verification metric is small, h outputs a smaller update step size.

[0151] Step S450: When the difference direction between the first verification indicator and the second verification indicator remains consistent in three consecutive training iterations and the third verification indicator does not exceed the preset fluctuation range, stop adjusting the network parameters and lock the current fully connected layer weights and convolutional layer channel attention coefficients as the final verification parameters.

[0152] The difference between the first and second validation metrics remained consistent across three consecutive training iterations, indicating a stable trend in the performance differences between the model on epithelial and mesenchymal tumor cell subsets. The third validation metric remained within the preset fluctuation range, indicating relatively stable performance on tumor cell subsets with uncertain differentiation. When these two conditions are met, the model's performance has stabilized. At this point, network parameter adjustments cease, locking the current fully connected layer weights and convolutional layer channel attention coefficients as the final validation parameters.

[0153] The preset fluctuation range is a pre-set fluctuation threshold for the third verification indicator, for example, ±10% of the third verification indicator's average value. Over three consecutive training iterations, the direction of the difference between the first and second verification indicators and the value of the third verification indicator are monitored. Network parameter adjustment ceases when the direction of the difference between the first and second verification indicators remains consistent three times in a row, and the value of the third verification indicator is within the preset fluctuation range.

[0154] Step S500: generating a final recognition model based on the verified and adjusted network parameters, wherein the final recognition model is configured to receive a clinical pathology image stream and output the location coordinates and classification confidence of circulating tumor cells.

[0155] The final recognition model, obtained after the aforementioned training and validation adjustments, demonstrates good performance and stability. The model receives a stream of clinical pathology images, meaning it can continuously process multiple clinical pathology images, which can be obtained from a hospital's pathology equipment. It outputs the location coordinates and classification confidence of circulating tumor cells, meaning the model can determine the location of circulating tumor cells in the image and assign a confidence level that each cell belongs to a different type.

[0156] As an implementation manner, step S500, generating a final recognition model based on the verified and adjusted network parameters, may specifically include the following steps S510-S560:

[0157] Step S510: Load the verified and adjusted network parameters into the convolutional layer and fully connected layer of the deep convolutional network. Connect the spatial positioning head and the classification head in parallel after the last convolutional feature map of the deep convolutional network. The spatial positioning head uses three sets of deconvolution layers to generate pixel-level coordinate offset maps and bounding box size maps. The classification head uses a channel attention mechanism to generate multi-category probability distribution maps.

[0158] The validation-adjusted network parameters are loaded into the convolutional and fully connected layers of the deep convolutional network to apply the optimal parameters obtained from previous training and validation adjustments to the model, thereby achieving better performance. The spatial localization head and classification head are connected in parallel after the convolutional feature map of the final layer of the deep convolutional network to simultaneously localize and classify circulating tumor cells. The spatial localization head uses three sets of deconvolutional layers to generate a pixel-level coordinate offset map and a bounding box size map. Deconvolutional layers are a type of convolutional layer used for upsampling, which increases the size of the feature map. These three sets of deconvolutional layers gradually increase the size of the final convolutional feature map to the same size as the original image, while simultaneously generating a pixel-level coordinate offset map and a bounding box size map. The pixel-level coordinate offset map represents the coordinate offset of each pixel relative to the center point of the circulating tumor cell, and the bounding box size map represents the width and height of the bounding box of each circulating tumor cell. The classification head uses a channel-level attention mechanism to generate a multi-class probability distribution map. The channel-level attention mechanism automatically focuses on important portions of a feature channel, enhancing the response strength of feature channels relevant to the classification. Through the channel attention mechanism, the classification head can generate a probability distribution for each pixel belonging to different cell types based on the input feature map, forming a multi-class probability distribution map. For example, for a classification task involving three types of cells: epithelial tumor cells, mesenchymal tumor cells, and normal cells, the classification head can generate a probability distribution for each pixel belonging to each of the three types.

[0159] The specific implementation process of step S510, for example, involves loading the verified and adjusted network parameters into the convolutional and fully connected layers of the deep convolutional network to ensure that the model parameters are optimized. Then, a spatial localization head and a classification head are added after the last convolutional feature map layer. For the spatial localization head, three sets of deconvolutional layers are constructed. The parameters of each deconvolutional layer, such as the kernel size, stride, and padding, can be adjusted as needed. For example, the kernel size of the first set of deconvolutional layers can be 4×4, with a stride of 2 and a padding of 1. The parameters of the second and third sets of deconvolutional layers can be adjusted as needed to gradually enlarge the feature map size to the same size as the original image. During training, the parameters of the deconvolutional layers are updated using a backpropagation algorithm to accurately generate pixel-level coordinate offset maps and bounding box size maps. For the classification head, a channel attention mechanism is introduced, such as the Squeeze-and-Excitation (SE) module. The SE module generates channel attention coefficients through global average pooling, fully connected layers, and Sigmoid activation functions, and then multiplies the channel attention coefficients with the input feature map to enhance the response strength of the feature channels related to classification. During the training process, the parameters of the classification head are updated through the backpropagation algorithm, enabling it to accurately generate multi-category probability distribution maps.

[0160] Step S520: Input the fused feature vector into the spatial positioning head, upsample the feature map through the deconvolution layer and perform weighted fusion with the immune cell aggregation density in the microenvironment features, and output the horizontal and vertical coordinate offsets and the bounding box width and height of each pixel position.

[0161] After the fused feature vector is input into the spatial localization head, the deconvolution layer upsamples the feature map to the same size as the original image, generating coordinate offsets and bounding box dimensions for each pixel. Weighted fusion with the immune cell aggregation density in the microenvironmental features is performed to leverage this microenvironmental information to improve the accuracy of circulating tumor cell localization.

[0162] The upsampling process of the deconvolution layer is achieved through convolution operations, which convert low-resolution feature maps into high-resolution feature maps. During deconvolution, the convolution kernel slides over the input feature map, and the output feature map is generated by calculating the convolution result between the convolution kernel and the input feature map. For example, if the input feature map is 32×32, after a set of deconvolution layers, the output feature map can be 64×64.

[0163] The process of weighted fusion of immune cell aggregation density is, for example: assign a weight coefficient to the immune cell aggregation density, multiply it element-by-element with the feature map output by the deconvolution layer, and then add the multiplication result to the feature map output by the deconvolution layer to obtain the weighted fused feature map. Through the weighted fused feature map, the spatial positioning head outputs the horizontal and vertical coordinate offsets and the bounding box width and height of each pixel position. The specific implementation is that on the weighted fused feature map, each channel corresponds to an output information, for example, one channel corresponds to the horizontal coordinate offset, one channel corresponds to the vertical coordinate offset, one channel corresponds to the bounding box width, and one channel corresponds to the bounding box height. By decoding the values of these channels, the horizontal and vertical coordinate offsets and the bounding box width and height of each pixel position can be obtained.

[0164] Step S530: Input the fused feature vector into the classification head, enhance the feature channel response intensity related to the nuclear atypia index through the channel attention mechanism, and output the classification probability of different cell types corresponding to each pixel position.

[0165] After the fused feature vector is input into the classification head, the channel attention mechanism automatically focuses on feature channels related to the nuclear atypia index, enhancing the response strength of these channels and thus improving the accuracy of circulating tumor cell classification. The nuclear atypia index measures the degree of nuclear morphological abnormality and is closely related to the type of circulating tumor cells.

[0166] The channel attention mechanism can employ the aforementioned SE module. Specifically, global average pooling is first performed on the input fused feature vector, compressing the spatial dimensions of the feature map into a scalar to obtain global feature information for each channel. This global feature information is then input into a fully connected layer, where a sigmoid activation function is applied to obtain a channel attention coefficient for each channel. Finally, the channel attention coefficient is element-wise multiplied by the input fused feature vector to enhance the response strength of the feature channels associated with the nuclear atypia index.

[0167] Using the enhanced feature vector, the classification head outputs the classification probability of each pixel corresponding to a different cell type. Specifically, a fully connected layer is added to the classification head, the enhanced feature vector is input to the fully connected layer, and the output is converted into a probability distribution through a Softmax activation function. For example, for a classification task involving three cell types, the output of the fully connected layer is a three-dimensional vector. After the Softmax activation function, the classification probability of each pixel corresponding to each of the three cell types is obtained.

[0168] Step S540: Perform peak point detection on the coordinate offset map and the bounding box size map, extract the pixel point with the local response maximum value as the candidate cell center point, and generate an initial detection box set based on the coordinate offset and bounding box size of the candidate cell center point.

[0169] Peak detection is used to find local maxima in an image, which can help identify the center of circulating tumor cells. Peak detection is performed on the coordinate offset map and bounding box size map to identify pixels with the largest changes in coordinate offset and bounding box size. These pixels are likely to be the center of circulating tumor cells.

[0170] Peak point detection can be performed using the non-maximum suppression (NMS) algorithm. The NMS algorithm performs a local maximum search on the coordinate offset map and the bounding box size map to identify the maximum value pixel within each local region. Then, based on the coordinate offsets and bounding box sizes of these maximum value pixels, the bounding box corresponding to each pixel is calculated. These bounding boxes are then sorted, and the one with the highest score is selected as the candidate bounding box. Finally, the remaining bounding boxes are traversed, and those whose overlap with the candidate bounding box exceeds a preset threshold are removed to obtain the final candidate cell center point.

[0171] The method for generating an initial set of detection frames based on the coordinate offsets and bounding box sizes of the candidate cell centers is to calculate the true coordinates of each candidate cell center in the original image based on its coordinate offset, and then generate a corresponding bounding box based on the bounding box size. The bounding boxes corresponding to all candidate cell centers are combined to form the initial set of detection frames.

[0172] Step S550: Match the initial detection frame set with the probability values of the corresponding areas in the multi-category probability distribution map, and select the detection frames whose probability values exceed the dynamic threshold as valid detection frames. The dynamic threshold is dynamically adjusted according to the probability distribution variance output by the classification head.

[0173] The purpose of matching the initial detection frame set with the probability values of the corresponding regions in the multi-class probability distribution map is to determine the probability that the cells within each detection frame belong to different types. Each pixel position in the multi-class probability distribution map corresponds to the classification probability of a different cell type. By overlapping the initial detection frame with the multi-class probability distribution map, the classification probability of each pixel position within the detection frame is extracted, and then these probabilities are statistically analyzed to obtain the probability that the cells within the detection frame belong to different types.

[0174] Detection frames with probability values exceeding the dynamic threshold are selected as valid detection frames to improve detection accuracy by removing those with low classification probabilities. The dynamic threshold is dynamically adjusted based on the variance of the probability distribution output by the classification head to adapt the threshold to different images and classification situations. If the probability distribution variance is large, the uncertainty of the classification result is high, and the dynamic threshold can be appropriately lowered. If the probability distribution variance is small, the uncertainty of the classification result is low, and the dynamic threshold can be appropriately increased.

[0175] The method of dynamically adjusting the threshold is, for example, to set a basic threshold T0 and an adjustment coefficient k, and to adjust the probability distribution variance according to the output of the classification head. Calculating dynamic thresholds The maximum probability that cells in each detection frame in the initial detection frame set belong to different types is compared with the dynamic threshold, and the detection frame whose probability value exceeds the dynamic threshold is regarded as a valid detection frame.

[0176] Step S560: Confidence calibration is performed on the effective detection frame based on the channel attention coefficient in the verified and adjusted network parameters, and the immune cell aggregation density and the intercellular connection density index are used as spatial weight factors to calculate the final classification confidence of the effective detection frame and output the positioning coordinates.

[0177] Confidence calibration of valid detection boxes is performed based on the channel attention coefficients in the validated and adjusted network parameters. This is done to utilize the channel attention coefficients to adjust the classification confidence of the valid detection boxes, ensuring that the confidence more accurately reflects the reliability of the detection results. The channel attention coefficients indicate the importance of each feature channel. By weightedly combining the channel attention coefficients with the original classification confidence of the valid detection box, the confidence of the detection boxes associated with important feature channels can be enhanced. Immune cell cluster density and intercellular connectivity density indicators are used as spatial weighting factors to account for the impact of cellular microenvironmental information on classification confidence. Immune cell cluster density and intercellular connectivity density indicators reflect the microenvironmental state of the cell, and different microenvironmental states may affect the classification results. By using these indicators as spatial weighting factors and multiplying them with the classification confidence of the valid detection box, the classification confidence can more accurately reflect the true state of the cell.

[0178] As an embodiment, step S560 performs confidence calibration on the valid detection frame based on the channel attention coefficient in the verified and adjusted network parameters, uses the immune cell aggregation density and the intercellular connection density index as spatial weight factors, calculates the final classification confidence of the valid detection frame, and outputs the positioning coordinates. Specifically, the following steps S561-S566 may be included:

[0179] Step S561: Extract the channel weighted value of the channel attention coefficient in the verified and adjusted network parameters, and perform channel-by-channel weighted summation on the immune cell aggregation density distribution map and the intercellular connection density distribution map in the area covered by the effective detection frame according to the channel weighted value to generate a spatial weight factor map.

[0180] Verify that the channel weighted values of the channel attention coefficients in the adjusted network parameters represent the importance of each feature channel. Apply these channel weighted values to the immune cell aggregation density distribution map and the intercellular connection density distribution map in the effective detection frame coverage area, and perform channel-by-channel weighted summation to obtain a spatial weight factor map that comprehensively considers channel importance and microenvironment information. When performing channel-by-channel weighted summation, for each channel of the immune cell aggregation density distribution map and the intercellular connection density distribution map, multiply the channel weighted value by the pixel value of the corresponding channel, and then add the results of all channels to obtain a spatial weight factor map. For example, assuming that the immune cell aggregation density distribution map has C i channels, the distribution of cell-cell junction tightness is C c channels, and the channel weighted values of the channel attention coefficients are , the pixel value of the jth channel of the immune cell aggregation density distribution map is , the pixel value of the kth channel of the intercellular connection density distribution map is , then the pixel value of the spatial weight factor map .

[0181] Step S562: Multiply the spatial weight factor map by the original classification probability distribution map of the valid detection box pixel by pixel to generate a weighted probability distribution map.

[0182] The purpose of pixel-by-pixel multiplication of the spatial weight factor map and the original classification probability distribution map of the effective detection frame is to incorporate the microenvironment information into the classification probability, so that the classification probability can more accurately reflect the actual situation of the cell. The pixel-by-pixel multiplication method is to multiply the pixel values of the spatial weight factor map and the original classification probability distribution map of the effective detection frame at each pixel position to obtain a weighted probability distribution map. For example, assuming that the pixel value of the spatial weight factor map at the pixel position (x, y) is S(x, y), and the pixel value of the original classification probability distribution map of the effective detection frame at the pixel position is P(x, y), then the pixel value of the weighted probability distribution map at the pixel position is P w(x, y) =S(x, y)×P(x, y) .

[0183] Step S563: Perform a regional integration operation on the weighted probability distribution map, count the maximum and mean values of the weighted probabilities within the coverage area of each valid detection frame, and calculate the geometric mean of the maximum and mean values as the preliminary confidence level.

[0184] The regional integration operation is to integrate the weighted probability distribution map within the area covered by the valid detection frame, that is, to sum up all the pixel values within the area. By counting the maximum and mean values of the weighted probabilities within each valid detection frame coverage area, the highest and average levels of the classification probability within the area can be obtained. Calculating the geometric mean of the maximum and mean values as the preliminary confidence level can comprehensively consider the maximum and mean values of the classification probability, so that the preliminary confidence level can more accurately reflect the reliability of the detection results. The calculation formula for the geometric mean is, for example, , where M is the maximum value and A is the mean value.

[0185] Step S564: According to the density gradient direction of the corresponding detection frame area in the immune cell aggregation density distribution map, the spatial consistency weight of the preliminary confidence is adjusted to generate an intermediate correction confidence.

[0186] The density gradient direction of the corresponding detection frame area in the immune cell aggregation density distribution map can reflect the trend and distribution of immune cell aggregation. The spatial consistency weight of the preliminary confidence is adjusted according to the density gradient direction, so that the preliminary confidence is more consistent with the spatial distribution characteristics of the cells. The method of adjusting the spatial consistency weight can be to calculate the density gradient direction of the corresponding detection frame area in the immune cell aggregation density distribution map, and adjust the preliminary confidence according to the consistency of the density gradient direction. If the density gradient direction is relatively consistent within the detection frame area, it means that the aggregation of immune cells has good spatial consistency. At this time, the spatial consistency weight of the preliminary confidence can be appropriately increased; if the density gradient direction is relatively scattered within the detection frame area, it means that the spatial consistency of the aggregation of immune cells is poor. At this time, the spatial consistency weight of the preliminary confidence can be appropriately reduced. Multiply the preliminary confidence by the spatial consistency weight to obtain the intermediate corrected confidence.

[0187] Step S565: multiplying the density difference ratio between inside and outside the detection box in the intercellular connection density distribution map by the intermediate correction confidence to obtain the final classification confidence.

[0188] The ratio of the density difference between the inside and outside of the detection box in the intercellular connectivity density distribution map reflects the difference in connectivity density between cells within the detection box and surrounding cells. Multiplying this ratio with the intermediate corrected confidence further accounts for the impact of intercellular connectivity density on classification confidence.

[0189] The process of calculating the density difference ratio between inside and outside the detection frame in the intercellular connection density distribution map is, for example, to calculate the average value of the intercellular connection density within the detection frame and the average value of the intercellular connection density outside the detection frame respectively, and then divide the average value within the detection frame by the average value outside the detection frame to obtain the density difference ratio.

[0190] Step S566: Sort the valid detection frames according to the final classification confidence, remove the detection frames with confidence lower than the preset stability threshold, and retain the positioning coordinates of the remaining detection frames and the corresponding final classification confidence.

[0191] Valid detection frames are sorted based on their final classification confidence to prioritize those with higher confidence levels for easier analysis and processing. The preset stability threshold is a pre-set confidence threshold used to eliminate detection frames with lower confidence levels, improving the reliability of detection results. Detection frames with final classification confidence levels below the preset stability threshold are eliminated, retaining the location coordinates and corresponding final classification confidence levels of the remaining detection frames.

[0192] See also Figure 2 , Figure 2The following is a schematic diagram of the structure of a computer system provided in an embodiment of the present invention. The computer system can be a stand-alone system, such as a high-performance desktop computer or workstation, or a cluster system, such as a computing cluster or cloud computing platform. The computer system includes at least a processor 101, a communication interface 102, and a memory 103. The processor 101, communication interface 102, and memory 103 can be connected via a bus or other means. The processor 101 (also known as the central processing unit (CPU)) is the computing and control core of the computer system, capable of parsing various instructions within the computer system and processing various data within the computer system. The communication interface 102 can optionally include a standard wired interface or a wireless interface (such as Wi-Fi or a mobile communication interface), which can be used to send and receive data under the control of the processor 101. The communication interface 102 can also be used for data transmission and interaction within the computer system. The memory 103 is a storage device in the computer system, used to store programs and data. It is understood that the memory 103 here can include both the computer system's built-in memory and, of course, the extended memory supported by the computer system. The memory 103 provides storage space that stores the operating system of the computer system, which is not limited in the present invention. In one embodiment, the processor 101 executes the deep learning-based circulating tumor cell recognition model training method provided in the above embodiment of the present invention by running the computer program in the memory 103.

Claims

1. A deep learning-based circulating tumor cell recognition model training method, characterized in that: The method comprises: Acquire a multi-channel pathology sample image set of circulating tumor cells, wherein the multi-channel pathology sample image set includes cell microscopic images processed with different stains, and each sample image carries corresponding cell type annotation information; performing cross-modal feature extraction on the multi-channel pathological sample image to generate a fusion feature vector comprising cell morphology features and microenvironment features, wherein the microenvironment features include matrix component distribution features and immune cell infiltration features; Construct a deep convolutional network with a multi-stage training strategy. In the first stage, a cross-channel feature attention mechanism is used to optimize the convolution kernel parameters. In the second stage, adversarial sample generation is used to enhance feature discriminability. In the third stage, knowledge distillation is used to compress the network size. Using a stratified cross-validation mechanism to verify and adjust the network parameters of the deep convolutional network during training, wherein the stratified cross-validation mechanism divides the network into multiple validation subsets according to the biological characteristics corresponding to the cell type annotation information, and adjusts the network parameters based on the classification error of each validation subset; generating a final recognition model based on the verified and adjusted network parameters, wherein the final recognition model is configured to receive a clinical pathology image stream and output location coordinates and classification confidence of circulating tumor cells; The cross-modal feature extraction of the multi-channel pathological sample image to generate a fusion feature vector containing cell morphology features and microenvironment features includes: Performing a superpixel segmentation operation on the multi-channel pathological sample image to generate structural data covering the cell nucleus region, the cytoplasm region and the extracellular matrix, wherein the structural data includes cell nucleus morphology parameters, cell membrane curvature parameters and extracellular matrix distribution parameters of each region; Based on the structural data, a two-branch feature extraction is performed, wherein the first branch extracts the nuclear atypia index and the intercellular connection tightness index as cell morphological features through geometric topological analysis, and the second branch extracts the matrix fiber orientation characteristics and the immune cell aggregation density as microenvironment features through spatial frequency transformation; Inputting the cell morphological features and microenvironmental features into a cross-modal alignment module, the module generates a feature alignment matrix by calculating the positional correlation between the two types of features in a spatial coordinate system, wherein each element in the matrix represents the degree of matching between the cell morphological features and the microenvironmental features at the corresponding spatial position; A multi-level feature interaction mechanism is used to process the feature alignment matrix, wherein the primary interaction layer captures the morphology-microenvironment association pattern in the local area through convolution operations, and the high-level interaction layer establishes cross-region feature dependencies through a self-attention mechanism; Dynamic gated fusion is performed on the interactively processed features to generate a four-dimensional feature tensor containing nuclear level morphological features, intercellular connection morphological features, matrix component microenvironment features, and immune infiltration microenvironment features; The four-dimensional feature tensor is input into the compression excitation network, and the morphological and microenvironmental feature channels that are sensitive to diagnosis are strengthened through the channel attention mechanism, redundant feature channels are suppressed, and finally the fusion feature vector with dimension compression is output.

2. The method according to claim 1, characterized in that The first branch extracts the nuclear atypia index and the intercellular connection tightness index as cell morphological features through geometric topological analysis, including: Performing edge detection on the structural data to generate cell nucleus contour boundary data and intercellular connection region boundary data, wherein the boundary data includes pixel-level coordinate sequences and curvature change information; Performing a polygon fitting operation on the cell nuclear contour boundary data to generate a polygon vertex distribution describing the cell nuclear morphology, calculating the nucleus-cytoplasm ratio deviation, contour concavity index, and principal axis deflection angle based on the vertex distribution, and fusing the parameters to generate a cell nuclear atypia index; Performing topological structure analysis on the boundary data of the intercellular connection area, extracting the statistical distribution of the overlapping length and contact angle of the adjacent cell contact surfaces, and generating an intercellular connection tightness index based on the area ratio of the connection area; The second branch extracts matrix fiber orientation characteristics and immune cell aggregation density as microenvironment characteristics through spatial frequency transformation, including: performing a spatial frequency transformation operation on the extracellular matrix region in the structural data, converting the pixel distribution into a frequency domain energy distribution, and extracting the main frequency direction as a matrix fiber orientation feature; performing regional cluster analysis on the frequency domain energy distribution to identify the spatial density distribution of immune cell aggregation regions, and generating immune cell aggregation density by combining the area of the cluster region with the contact boundary length of the adjacent matrix region; The nuclear atypia index and the cell-to-cell connection density index are combined into cell morphology characteristics, and the matrix fiber orientation characteristics and the immune cell aggregation density are combined into microenvironment characteristics.

3. The method according to claim 1, characterized in that The multi-stage training strategy includes: Inputting the initial parameters of the deep convolutional network into the first-stage channel optimization module, generating a channel enhancement coefficient matrix by calculating the spatial correlation between each feature channel activation map and the nuclear atypia index, performing channel-by-channel weighting on the convolution kernel based on the channel enhancement coefficient matrix, and outputting the first-stage optimization parameters; Inputting the first-stage optimized parameters into the second-stage adversarial training module, generating a perturbation sample set based on the optimized convolution kernel, wherein the perturbation samples in the perturbation sample set are constructed by adjusting the spatial co-occurrence pattern of the intercellular connection density index and the immune cell aggregation density, while maintaining the continuity constraint of the matrix fiber orientation characteristics; Alternately inputting the perturbed sample set and the original sample into the deep convolutional network, updating the fully connected layer parameters by comparing the distribution offsets of the original features and the perturbed features in the microenvironment feature space, and outputting the second-stage optimization parameters; The second-stage optimization parameters are input into the third-stage compression module. By matching the feature activation path differences between the teacher network and the student network during the nuclear atypia index extraction process, a channel importance ranking algorithm is used to remove redundant channels and output lightweight third-stage optimization parameters.

4. The method according to claim 1, wherein The layered cross-validation mechanism is used to verify and adjust the network parameters of the deep convolutional network during the training process, including: Dividing the multi-channel pathology sample image set into an epithelial-derived tumor cell subset, a mesenchymal-derived tumor cell subset, and a tumor cell subset with an undetermined degree of differentiation according to the biological characteristics corresponding to the cell type annotation information; In each round of training iteration, validation operations are performed in the order of the epithelial tumor cell subset, the mesenchymal tumor cell subset, and the tumor cell subset with an undetermined degree of differentiation, wherein the validation operation for each subset includes extracting the nuclear atypia index classification error and the immune cell aggregation density regression error of the current network parameters on the corresponding subset; The weighted sum of the classification error and the regression error of the epithelial-derived tumor cell subset is used as a first validation indicator, the weighted sum of the classification error and the regression error of the mesenchymal-derived tumor cell subset is used as a second validation indicator, and the weighted sum of the classification error and the regression error of the tumor cell subset with an undetermined degree of differentiation is used as a third validation indicator; Dynamically adjust the learning rate and loss function weight distribution ratio of the fully connected layer of the deep convolutional network based on the difference between the first verification indicator and the second verification indicator, and adjust the channel attention coefficient update step size of the convolutional layer based on the absolute value of the third verification indicator; When the difference direction between the first verification indicator and the second verification indicator remains consistent in three consecutive training iterations and the third verification indicator does not exceed the preset fluctuation range, stop adjusting the network parameters and lock the current fully connected layer weights and convolutional layer channel attention coefficients as the final verification parameters.

5. The method according to claim 1, wherein The generating of the final recognition model based on the verified and adjusted network parameters includes: The verified and adjusted network parameters are loaded into the convolutional layer and fully connected layer of the deep convolutional network. A spatial localization head and a classification head are connected in parallel after the last convolutional feature map of the deep convolutional network. The spatial localization head uses three sets of deconvolution layers to generate a pixel-level coordinate offset map and a bounding box size map. The classification head uses a channel attention mechanism to generate a multi-category probability distribution map. Input the fused feature vector into the spatial positioning head, upsample the feature map through the deconvolution layer and perform weighted fusion with the immune cell aggregation density in the microenvironment feature, and output the horizontal and vertical coordinate offsets and the bounding box width and height of each pixel position; Inputting the fused feature vector into the classification head, enhancing the feature channel response intensity related to the nuclear atypia index through the channel attention mechanism, and outputting the classification probability of different cell types corresponding to each pixel position; Performing peak point detection on the coordinate offset map and the bounding box size map, extracting the pixel point with the local response maximum value as the candidate cell center point, and generating an initial detection box set based on the coordinate offset and bounding box size of the candidate cell center point; Matching the initial set of detection frames with the probability values of corresponding areas in the multi-class probability distribution map, and screening detection frames whose probability values exceed a dynamic threshold as valid detection frames, wherein the dynamic threshold is dynamically adjusted according to the variance of the probability distribution output by the classification head; Based on the channel attention coefficient in the verified and adjusted network parameters, confidence calibration is performed on the effective detection frame, the immune cell aggregation density and the intercellular connection density index are used as spatial weight factors, the final classification confidence of the effective detection frame is calculated, and the positioning coordinates are output.

6. The method according to claim 5, characterized in that The confidence calibration of the effective detection frame is performed based on the channel attention coefficient in the network parameters adjusted after verification, the immune cell aggregation density and the intercellular connection density index are used as spatial weight factors, the final classification confidence of the effective detection frame is calculated, and the positioning coordinates are output, including: Extracting the channel weighted value of the channel attention coefficient in the verified and adjusted network parameters, and performing channel-by-channel weighted summation on the immune cell aggregation density distribution map and the intercellular connection density distribution map in the area covered by the effective detection frame according to the channel weighted value to generate a spatial weight factor map; Multiplying the spatial weight factor map by the original classification probability distribution map of the valid detection box pixel by pixel to generate a weighted probability distribution map; Performing a regional integration operation on the weighted probability distribution map, counting the maximum and mean values of the weighted probabilities within the coverage area of each valid detection frame, and calculating the geometric mean of the maximum and mean values as a preliminary confidence level; Adjusting the spatial consistency weight of the preliminary confidence level according to the density gradient direction of the corresponding detection frame area in the immune cell aggregation density distribution map to generate an intermediate corrected confidence level; The final classification confidence is obtained by multiplying the density difference ratio between inside and outside the detection frame in the intercellular connection density distribution map by the intermediate correction confidence; The valid detection frames are sorted according to the final classification confidence, the detection frames with confidence lower than a preset stability threshold are eliminated, and the positioning coordinates of the remaining detection frames and the corresponding final classification confidence are retained.

7. The method according to claim 2, characterized in that The adversarial sample generation process includes: Extracting the characteristic gradient distribution of the deep convolutional network on the multi-channel pathological sample image set, and identifying the cross-modal gradient association pattern between the cell nuclear atypia index and the immune cell aggregation density based on the characteristic gradient distribution; generating a perturbation direction template according to the cross-modal gradient association pattern, wherein the perturbation direction template includes a pixel offset direction that produces a synergistic gradient change in the cell nucleus morphological parameters and the matrix fiber orientation characteristics; Inputting the multi-channel pathological sample image into the deep convolutional network, extracting the spatial gradient response map of the fused feature vector, and generating an initial perturbation mask based on the distribution similarity between the spatial gradient response map and the perturbation direction template; Performing a morphological boundary constraint operation on the initial perturbation mask to eliminate isolated perturbation points that have no intersection with the cytoplasm region boundary, and retaining the perturbation region that overlaps with the cell nucleus contour boundary data space; The retained perturbation region is superimposed on the original multi-channel pathology sample image to generate a perturbed image and input into the deep convolutional network to calculate the consistency index of the change direction of the nuclear atypia index before and after the perturbation and the change direction of the immune cell aggregation density; The intensity distribution of the perturbation direction template is dynamically adjusted based on the consistency index to generate an adversarial sample set, wherein the perturbation intensity of each sample in the adversarial sample set is inversely correlated with the local change rate of the cell membrane curvature parameter.

8. The method according to claim 1, characterized in that The process of knowledge distillation and compression includes: Using the convolutional layer parameters of the deep convolutional network as the teacher network parameters, initializing the convolutional layer parameters of the student network so that its channel dimension matches the teacher network parameters but with a reduced number of channels; Extracting a set of intermediate feature activation maps of the teacher network on the multi-channel pathology sample image, wherein the set of intermediate feature activation maps includes spatial response distributions of the cell morphology features and microenvironment features in different convolutional layers; Aligning the intermediate feature activation map of the student network with the intermediate feature activation map of the teacher network across layers, compensating for the resolution difference between the two through dilated convolution, and generating a feature similarity distribution map; Dynamically adjusting the channel attention coefficients of the student network based on the feature similarity distribution map to suppress channels with significantly different response patterns from the teacher network and enhance the weights of spatially consistent channels; Inputting the compressed feature vector of the student network into the classifier, calculating the difference loss between the student network and the teacher network classification probability distribution, and generating an adaptive distillation loss coefficient in combination with the cell type annotation information; The fully connected layer parameters of the student network are iteratively updated according to the adaptive distillation loss coefficient until the compression process is terminated when the difference between the comprehensive performance index of the student network and the performance of the teacher network under the layered cross-validation mechanism is stable within a preset range.

9. A computer system, characterized in that: include: a memory storing a computer program; A processor, configured to load the computer program to implement the deep learning-based circulating tumor cell recognition model training method according to any one of claims 1 to 8.

Images