Pathological image sjogren's syndrome automatic diagnosis method, device, equipment and storage medium

Abstract

The application discloses a pathological image dry syndrome automatic diagnosis method and device, equipment and storage medium, and relates to the technical field of medical data processing. The method comprises the following steps: performing image preprocessing on an original digital pathology section image to obtain an optimized image block; performing cell-tissue collaborative segmentation on the optimized image block to obtain a cell entity set and a tissue entity set; mapping the cell entity set and the tissue entity set into a heterogeneous graph structure, wherein the heterogeneous graph structure takes cell graph nodes and tissue graph nodes as vertices and takes spatial adjacency relationships as edges; and determining a dry syndrome state based on the heterogeneous graph structure. The application abstracts cells and tissues into nodes and establishes spatial adjacency relationship edges by constructing a heterogeneous graph structure, models the global relationship between cells and the cross-scale relationship between cells and tissues, realizes global topology perception, improves diagnosis accuracy, and forms an end-to-end intelligent analysis process from pathological images to dry syndrome diagnosis.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical data processing, and particularly relates to a pathological image Sjogren's syndrome automatic diagnosis method and device, equipment and a storage medium. BACKGROUND

[0002] Sjogren's syndrome (SS) is a rare chronic autoimmune disease that mainly affects adult women, characterized by long-term inflammation and dysfunction of salivary and lacrimal glands. The disease is often associated with other diseases such as systemic lupus erythematosus, rheumatoid arthritis and kidney disease, which can increase mortality. Early and accurate diagnosis is crucial for improving patient outcomes.

[0003] However, the current diagnosis method relies on histopathological evaluation of labial gland tissue and deep learning methods based on convolutional neural networks (CNN), and these traditional methods have certain limitations, which limit their widespread application and effectiveness.

[0004] Therefore, how to improve the accuracy of Sjogren's syndrome diagnosis based on pathological images has become a problem to be solved. SUMMARY

[0005] The main purpose of the present application is to provide a pathological image Sjogren's syndrome automatic diagnosis method, device, equipment and storage medium, which aims to solve the technical problem of how to improve the accuracy of Sjogren's syndrome diagnosis based on pathological images.

[0006] To achieve the above-mentioned purpose, the present application provides a pathological image Sjogren's syndrome automatic diagnosis method, which comprises the following steps:

[0007] Image preprocessing is performed on the original digital pathology section image to obtain an optimized image block;

[0008] Cell-tissue collaborative segmentation is performed on the optimized image block to obtain a cell entity set and a tissue entity set;

[0009] The cell entity set and the tissue entity set are mapped into a heterogeneous graph structure, and the heterogeneous graph structure takes cell graph nodes and tissue graph nodes as vertices and spatial adjacency relationships as edges;

[0010] Sjogren's syndrome state determination is performed based on the heterogeneous graph structure.

[0011] In an embodiment, the cell entity set comprises single cell instances and cell multi-modal feature vectors corresponding to the single cell instances, and the tissue entity set comprises tissue region instances and tissue multi-modal feature vectors corresponding to the tissue region instances; the step of performing cell-tissue collaborative segmentation on the optimized image block to obtain a cell entity set and a tissue entity set comprises:

[0012] performing cell contour prediction on the optimized image block by a preset instance segmentation model to obtain a cell instance mask, and extracting single cell instances from the cell instance mask;

[0013] performing feature fusion on the cell geometric features, color texture features and spatial position features extracted for each single cell instance to obtain the cell multi-modal feature vectors;

[0014] performing tissue region prediction on the optimized image block by a general segmentation model to obtain a tissue region mask, and obtaining tissue region instances corresponding to the tissue region mask;

[0015] performing feature fusion on the tissue geometric features, staining texture features and position context features extracted for each tissue region instance to obtain the tissue multi-modal feature vectors.

[0016] In an embodiment, the step of mapping the cell entity set and the tissue entity set into a heterogeneous graph structure comprises:

[0017] taking each single cell instance as a cell graph node, and establishing a cell-cell edge set between the cell graph nodes that meet preset interval requirements;

[0018] taking each tissue region instance as a tissue graph node, and establishing a cell-tissue edge set between the cell graph nodes and the tissue graph nodes that meet preset coverage requirements;

[0019] integrating the cell-cell edge set and the cell-tissue edge set to generate a heterogeneous graph structure, wherein the nodes of the heterogeneous graph structure carry corresponding cell multi-modal feature vectors or tissue multi-modal feature vectors.

[0020] In an embodiment, the step of determining the dry syndrome state based on the heterogeneous graph structure comprises:

[0021] optimizing the cell multi-modal feature vectors in the heterogeneous graph structure by a self-supervised graph attention network to obtain optimized cell feature vectors; the self-supervised graph attention network is obtained through self-supervised pre-training by node masking reconstruction and contrast learning with structure disturbance;

[0022] perform immune cell classification according to the optimized cell feature vector, to obtain a plurality of immune cell graph nodes;

[0023] perform preset clustering analysis based on the plurality of immune cell graph nodes, and output a Sjogren's syndrome state determination result.

[0024] In an embodiment, the step of performing preset clustering analysis based on the plurality of immune cell graph nodes, and outputting a Sjogren's syndrome state determination result, comprises:

[0025] construct an immune cell graph node subgraph based on the plurality of immune cell graph nodes;

[0026] perform clustering on the immune cell graph node subgraph by running a preset community detection algorithm, to obtain immune cell clusters;

[0027] output a Sjogren's syndrome state determination result according to spatial distribution information of the immune cell clusters and the number of immune cells in the immune cell clusters.

[0028] In an embodiment, the step of performing image preprocessing on the original digital pathology slice image, to obtain optimized image blocks, comprises:

[0029] perform multi-scale tissue region extraction on the original digital pathology slice image, to obtain a target tissue mask;

[0030] perform region division on the original digital pathology slice image based on the target tissue mask, to obtain initial image blocks;

[0031] perform data enhancement processing on the initial image blocks, to obtain optimized image blocks.

[0032] In an embodiment, the step of performing multi-scale tissue region extraction on the original digital pathology slice image, to obtain a target tissue mask, comprises:

[0033] perform down-sampling on the original digital pathology slice image, to obtain a low-resolution optimized digital pathology slice image;

[0034] perform tissue region coarse positioning on the optimized digital pathology slice image, to obtain an initial tissue mask;

[0035] map the initial tissue mask back to the coordinate system of the original digital pathology slice image, to obtain a target tissue mask.

[0036] In addition, to achieve the above-mentioned purposes, the present application further provides a pathological image Sjogren's syndrome automatic diagnosis device, which comprises:

[0037] an image preprocessing module configured to perform image preprocessing on an original digital pathology slice image, to obtain optimized image blocks;

[0038] an image segmentation module configured to perform cell-tissue collaborative segmentation on the optimized image block to obtain a cell entity set and a tissue entity set;

[0039] a graph learning module configured to map the cell entity set and the tissue entity set into a heterogeneous graph structure, the heterogeneous graph structure having cell graph nodes and tissue graph nodes as vertices and spatial adjacency relations as edges;

[0040] a recognition module configured to determine a Sjogren's syndrome state based on the heterogeneous graph structure.

[0041] In addition, to achieve the above object, the present application further provides a pathological image Sjogren's syndrome automatic diagnosis device, which comprises a memory, a processor, and a pathological image Sjogren's syndrome automatic diagnosis program stored in the memory and executable on the processor, and the pathological image Sjogren's syndrome automatic diagnosis program is configured to implement the steps of the pathological image Sjogren's syndrome automatic diagnosis method mentioned above.

[0042] In addition, to achieve the above object, the present application further provides a storage medium, which stores a pathological image Sjogren's syndrome automatic diagnosis program, and the pathological image Sjogren's syndrome automatic diagnosis program implements the steps of the pathological image Sjogren's syndrome automatic diagnosis method mentioned above when executed by a processor.

[0043] The present application provides a pathological image Sjogren's syndrome automatic diagnosis method, device, equipment and storage medium. The method comprises: performing image preprocessing on an original digital pathology slice image to obtain an optimized image block; performing cell-tissue collaborative segmentation on the optimized image block to obtain a cell entity set and a tissue entity set; mapping the cell entity set and the tissue entity set into a heterogeneous graph structure, the heterogeneous graph structure having cell graph nodes and tissue graph nodes as vertices and spatial adjacency relations as edges; and determining a Sjogren's syndrome state based on the heterogeneous graph structure. The present application provides a more efficient, accurate and strongly explanatory digital pathology image automatic diagnosis solution, which abstracts cells and tissues into nodes and establishes spatial adjacency relation edges by constructing a heterogeneous graph structure, realizes modeling of global relationships between cells and cross-scale relationships between cells and tissues, effectively captures complex interactions between cells, and solves the problem of local field of view limitation of CNN. Therefore, the present application can realize global topology perception by establishing global relationships between cells, improve diagnosis accuracy, and is particularly suitable for early detection and diagnosis of complex diseases such as Sjogren's syndrome, forming an end-to-end intelligent analysis process from pathological images to Sjogren's syndrome diagnosis. BRIEF DESCRIPTION OF DRAWINGS

[0044] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate embodiments consistent with the application and, together with the description, further serve to explain the principles of the application.

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the accompanying drawings required by the embodiments or the prior art description will be briefly introduced as follows. Obviously, those skilled in the art can obtain other drawings from these drawings without any creative effort.

[0046] Figure 1 A flowchart of a first embodiment of the pathological image Sjogren's syndrome automatic diagnosis method of the present application;

[0047] Figure 2 A flowchart of a second embodiment of the pathological image Sjogren's syndrome automatic diagnosis method of the present application;

[0048] Figure 3 A cell segmentation schematic diagram of the second embodiment of the pathological image Sjogren's syndrome automatic diagnosis method of the present application;

[0049] Figure 4 A tissue segmentation schematic diagram of the second embodiment of the pathological image Sjogren's syndrome automatic diagnosis method of the present application;

[0050] Figure 5 A Sjogren's syndrome state determination process schematic diagram of the second embodiment of the pathological image Sjogren's syndrome automatic diagnosis method of the present application;

[0051] Figure 6 A process schematic diagram of the pathological image Sjogren's syndrome automatic diagnosis method of the present application;

[0052] Figure 7 A module structure schematic diagram of the pathological image Sjogren's syndrome automatic diagnosis device of the embodiment of the present application;

[0053] Figure 8 A device structure schematic diagram of the hardware running environment involved in the pathological image Sjogren's syndrome automatic diagnosis method in the embodiment of the present application.

[0054] The purpose implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0055] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application, and are not used to limit the present application.

[0056] In order to better understand the technical solutions of the present application, the following will be described in detail with reference to the drawings and specific embodiments in the description.

[0057] The main solution of the present application is: performing image preprocessing on the original digital pathology slice image to obtain an optimized image block; performing cell-tissue collaborative segmentation on the optimized image block to obtain a cell entity set and a tissue entity set; mapping the cell entity set and the tissue entity set into a heterogeneous graph structure, the heterogeneous graph structure taking cell graph nodes and tissue graph nodes as vertices and taking spatial adjacency relationships as edges; and performing a dry syndrome state judgment based on the heterogeneous graph structure.

[0058] At present, the analysis of dry syndrome based on pathological images mainly relies on deep learning methods based on convolutional neural networks (CNN). Image classification networks represented by ResNet-50 are usually used for the process of patching, feature extraction, and re-aggregated classification of WSI (Whole Slide Image, whole slide digital pathology slice image) images. However, the convolution operation of CNN has a limited receptive field, which means that it mainly focuses on local features and is difficult to capture the global relationship between cells. For pathological image analysis, it is crucial to understand the interaction between cells and their distribution in the entire tissue.

[0059] In addition, existing methods usually ignore the real spatial topological relationship between cells. For example, in pathological images, different types of cells are mixed together to form specific functional structures or clusters, such as immune cell clusters. Traditional CNN models have difficulty accurately capturing these complex structural information because they mainly rely on fixed-size convolution kernels to extract features. This leads to incomplete understanding of disease characteristics and increased risk of misdiagnosis.

[0060] In summary, the existing task of analyzing dry syndrome based on pathological images faces many challenges, which limit the effectiveness of existing methods in actual clinical applications. It is necessary to overcome the above shortcomings and improve the diagnosis efficiency and accuracy.

[0061] Therefore, the present application provides a more efficient, accurate, and highly interpretable automatic digital pathology image diagnosis solution. By constructing a heterogeneous graph structure, cells and tissues are abstracted as nodes and spatial adjacency relationship edges are established, modeling the global relationship between cells and the cross-scale relationship between cells and tissues, effectively capturing the complex interaction between cells, and solving the problem of CNN's local field of view limitation. Therefore, the present application can overcome the limitation of CNN's local receptive field, establish global relationships between cells, achieve global topological perception, improve diagnosis accuracy, and is particularly suitable for early detection and diagnosis of complex diseases such as dry syndrome, forming an end-to-end intelligent analysis process from pathological images to dry syndrome diagnosis.

[0062] It should be noted that the execution subject of the embodiment can be a pathological image dry syndrome automatic diagnosis system, or a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, or a pathological image dry syndrome automatic diagnosis device capable of realizing the above functions, and the like, and the embodiment does not make specific limitations. The following takes a pathological image dry syndrome automatic diagnosis device (referred to as a diagnosis device) as an execution subject, and the embodiments described below are described.

[0063] Based on this, the pathological image dry syndrome automatic diagnosis method provided in the embodiment of the present application is provided, which is described with reference to Figure 1 , Figure 1 The flowchart of the first embodiment of the pathological image dry syndrome automatic diagnosis method of the present application is shown.

[0064] In the embodiment, the pathological image dry syndrome automatic diagnosis method includes steps S10-S40:

[0065] Step S10, image preprocessing is performed on the original digital pathology section image to obtain an optimized image block;

[0066] It should be understood that the original digital pathology section image Iorig∈R H × W ×C (wherein H, W represent the height and width of the image respectively, and C represents the number of channels (such as RGB or H&E staining)) can be a WSI image of oral labial gland tissue, which is used for dry syndrome (SS) diagnosis and analysis, and is a high-resolution digital pathology image that can fully display the details of the entire oral labial gland tissue section, and the resolution can usually reach 100,000 x 100,000 pixel level. Therefore, the diagnosis device needs to perform image preprocessing on the original digital pathology section image to reduce the computational burden, remove invalid information and improve the accuracy of subsequent processing, so as to obtain an image segment with uniform size and focused on the diagnosis related area, i.e. the above-mentioned optimized image block, which can adopt a fixed size of 512 x 512 pixels in the embodiment, which is convenient for subsequent cell-tissue collaborative segmentation operation.

[0067] It can be understood that in the embodiment, the image preprocessing can include steps such as multi-scale tissue region extraction, region division and data enhancement processing. In a feasible implementation manner, in the embodiment, step S10 can include steps A1-A3:

[0068] Step A1, multi-scale tissue region extraction is performed on the original digital pathology section image to obtain a target tissue mask;

[0069] It is easy to understand that the embodiment can filter out the effective area containing tissues from Iorig through the multi-scale processing of "down-sampling-rough positioning-coordinate mapping", exclude the interference of the background (such as the blank area of the slide), and the core is to obtain a binary matrix of all tissue region pixel positions in the marked original WSI image (the pixel value of 1 represents the tissue region and 0 represents the background), that is, the above-mentioned target tissue mask. It is the basis for subsequent image block division and ensures that only the effective tissue region is processed.

[0070] Therefore, in a feasible implementation, step A1 can include steps A11-A13:

[0071] Step A11, performing down-sampling on the original digital pathology section image to obtain a low-resolution optimized digital pathology section image;

[0072] Step A12, performing rough positioning of the tissue region on the optimized digital pathology section image to obtain an initial tissue mask;

[0073] Step A13, mapping the initial tissue mask back to the coordinate system of the original digital pathology section image to obtain a target tissue mask.

[0074] It is understood that the embodiment can perform resolution reduction processing on the original digital pathology section image Iorig, that is, the above-mentioned down-sampling, to reduce the number of pixels by reducing the image size, and the core is to reduce the calculation burden while retaining the tissue contour information. In the embodiment, the bilinear interpolation method can be preferably used to realize down-sampling to better retain the tissue edge contour and avoid jagged distortion.

[0075] For the low-resolution optimized digital pathology section image after down-sampling, it can be represented as Ilowres, and the resolution thereof is s times that of the original digital pathology section image, s ∈ (0, 1), the macroscopic contour of the tissue is retained, and it can be used for efficient rough positioning of the tissue region to avoid the calculation pressure of directly processing the original high-resolution image. The scaling factor s can be adaptively set according to the size of the input original digital pathology section image Iorig, and the specific value is not limited in the embodiment.

[0076] Therefore, the embodiment can identify and mark the tissue region of Ilowres on Ilowres through the pre-trained SAM2 (Segment Anything Model 2) model, that is, perform the above-mentioned rough positioning of the tissue region, and output the binary mask after preliminary segmentation, that is, the above-mentioned initial tissue mask, which is the "intermediate mask" mapped to the original image coordinate system, so as to quickly filter out the effective tissue region, mark the tissue and background area in the low-resolution image, and provide a basis for the subsequent accurate mask generation of the original image. The formula for rough positioning of the tissue region can be as follows:

[0077] T_init = SAM2(Ilowres) (1)

[0078] where H' and W' are the height and width of the low resolution image, and T_init is the binary mask matrix, i.e. the initial tissue mask mentioned above.

[0079] It should be noted that, according to the pre-established correspondence between the low resolution image and the pixel coordinates of the original WSI image (x_orig = x_low / s, y_orig = y_low / s, x_orig and y_orig are the pixel coordinates of Iorig, and x_low / s and y_low / s are the pixel coordinates of Ilowres), the pixel position of the initial tissue mask can be converted into the pixel position of the original WSI image Iorig, to obtain a binary matrix mapped to the original WSI coordinate system, i.e. the target tissue mask T_target (H x W) mentioned above, so as to accurately mark the pixel position of all tissue regions in the original image and ensure the accuracy of tissue region positioning. In this process, the boundary pixels in the target tissue mask can be further refined by querying whether they belong to specific staining pixels (such as blue nuclei and pink cytoplasm in H&E staining), to improve the pixel-level accuracy of the target tissue mask.

[0080] In this embodiment, the resolution of the original WSI is reduced to within 10,000 pixels by downsampling, and the tissue positioning time of the SAM2 model is significantly shortened compared with existing solutions, the calculation efficiency is greatly improved, and the macroscopic outline of the tissue is retained; at the same time, the accurate target tissue mask can improve the effective tissue proportion of the initial image block, reduce the interference of invalid image blocks on cell segmentation and graph modeling, and indirectly improve the final diagnosis accuracy.

[0081] Step A2, regionally dividing the original digital pathology slice image based on the target tissue mask to obtain an initial image block;

[0082] It is easy to understand that the diagnostic device can further divide the original WSI into initial image blocks of a fixed size based on the target tissue mask, using a sliding window or density clustering method, i.e. performing the above-mentioned regional division, to solve the problem of high computational burden caused by the high resolution of the original WSI. In this embodiment, the fixed-size image segment containing only the tissue region obtained after regional division, i.e. the initial image block mentioned above, can be 512 x 512 pixels in size.

[0083] Exemplarily, in the embodiment, the division parameter can be set first, the initial image block size s_p (in the embodiment, s_p can be set as 512*512 pixels) is determined, and the sliding window step s_step (to avoid the loss of tissue edge information, s_step can be set as 1 / 2 of s_p, that is, 256 pixels) is set; then, according to the tissue distribution, the target tissue mask T_target is taken as a reference, and the s_p*s_p window is slid with s_step as the step from the upper left corner (0, 0) of the original WSI image I_orig. At the same time, for each window, the proportion of the tissue pixels with a pixel value of 1 in T_target in the window is judged: if the proportion is greater than or equal to 50% (the window with a high background proportion is excluded), the original image region corresponding to the window is cropped as the initial image block P_init; finally, for the edge region of the original WSI, if the remaining region is less than s_p*s_p pixels, zero padding is used to supplement to s_p*s_p pixels, so as to ensure that the sizes of all initial image blocks are uniform; finally, N initial image blocks {P_init_1, P_init_2, …, P_init_N} are obtained.

[0084] Step A3, performing data enhancement processing on the initial image block to obtain an optimized image block.

[0085] It can be understood that the diagnostic device can perform data enhancement processing through color normalization, geometric transformation, noise injection and the like, so as to expand data diversity, eliminate the influence of staining difference and image acquisition noise, improve the generalization ability of the subsequent model, and obtain an optimized image block Pi_final e Rsp×sp×C (sp is the image block size) with color standardization, posture diversity and anti-noise ability.

[0086] In the embodiment, by performing stage-by-stage processing on the high-resolution image, the computational complexity is significantly reduced, the response speed of the overall system is improved, only the tissue region is retained for subsequent processing, the interference of invalid background information is avoided, and the concentration and accuracy of the model are improved. In the process, the tissue positioning calculation amount can be reduced through multi-scale downsampling (such as 10 times downsampling), the tissue details of the original image are retained through coordinate mapping, only the tissue region is cut and processed subsequently, the overall calculation efficiency is improved, and no key pathological details are lost, so as to reduce the computational complexity, balance the efficiency and details;

[0087] And through the target tissue mask, only the area with tissue proportion ≥ 50% is cut, the invalid background block proportion is significantly reduced, the waste of computing resources is reduced, and the background interference in cell segmentation is avoided, thereby filtering invalid background and focusing on the diagnosis area; and the uniform size of the optimized image block can provide standardized input for subsequent Stardist cell segmentation and SAM2 tissue segmentation, avoid model adaptation problems caused by inconsistent image sizes, and improve the stability of the entire diagnostic process. At the same time, the general model SAM2 is used for tissue segmentation, without the need for retraining for specific staining modes, and has good compatibility and expandability.

[0088] Step S20, performing cell-tissue collaborative segmentation on the optimized image block to obtain a cell entity set and a tissue entity set;

[0089] Step S30, mapping the cell entity set and the tissue entity set into a heterogeneous graph structure, the heterogeneous graph structure taking cell graph nodes and tissue graph nodes as vertices and spatial adjacency relationships as edges;

[0090] Step S40, determining the dry syndrome state based on the heterogeneous graph structure.

[0091] It should be understood that the present embodiment can simultaneously segment cells and tissues in the optimized image block, i.e., perform the above-mentioned cell-tissue collaborative segmentation, which can realize cell instance segmentation and tissue region segmentation through different segmentation models in the process, and finally obtain a cell entity set and a tissue entity set. The diagnostic device can further convert it into a graph structure containing different types of nodes (cell graph nodes and tissue graph nodes) and edges (edges based on spatial adjacency relationships), i.e., the above-mentioned heterogeneous graph structure, thereby simultaneously modeling the spatial adjacency relationships between cells and between cells and tissues, and realizing the unified expression of microscopic cell individuals and macroscopic tissue backgrounds.

[0092] Finally, the diagnostic device can determine whether the original digital pathology slide image corresponds to a dry syndrome positive pathological sample based on the heterogeneous graph structure through steps such as graph neural network optimization features, classification of immune cells, and clustering analysis, i.e., determine the dry syndrome state.

[0093] Therefore, for the existing pathological image analysis method based on a convolutional neural network (CNN), there is a problem of local field of view limitation (limited CNN receptive field, difficult to capture the global relationship between cells), which leads to low diagnostic accuracy of Sjogren's syndrome. The embodiment abstracts cells and tissues into nodes and establishes spatial adjacency relationship edges to realize modeling of the global relationship between cells and the cross-scale relationship between cells and tissues, effectively capture the complex interaction between cells, and solve the problem of CNN local field of view limitation. Therefore, the embodiment can overcome the CNN local receptive field limitation, realize global topology perception by establishing the global relationship between cells, and improve the diagnostic accuracy.

[0094] Meanwhile, the embodiment proposes a tissue segmentation and cell recognition technology for pathological pictures with adaptive resolution, which adjusts the resolution adaptively and divides the subgraph to process larger pathological pictures into a size convenient for model processing without losing details, and then uses tissue segmentation based on a segmentation model and cell extraction based on polygon recognition to recognize cells and tissues, thereby reducing the computational complexity and balancing efficiency and details.

[0095] The embodiment provides a pathological image Sjogren's syndrome automatic diagnosis method, which comprises: performing down-sampling on an original digital pathology section image to obtain an optimized digital pathology section image with low resolution; performing tissue region coarse positioning on the optimized digital pathology section image to obtain an initial tissue mask; mapping the initial tissue mask back to the coordinate system of the original digital pathology section image to obtain a target tissue mask; dividing the original digital pathology section image based on the target tissue mask to obtain an initial image block; performing data enhancement processing on the initial image block to obtain an optimized image block; performing cell-tissue collaborative segmentation on the optimized image block to obtain a cell entity set and a tissue entity set; mapping the cell entity set and the tissue entity set into a heterogeneous graph structure, the heterogeneous graph structure taking a cell graph node and a tissue graph node as a vertex and taking a spatial adjacency relationship as an edge; and determining the Sjogren's syndrome state based on the heterogeneous graph structure. Therefore, for the existing pathological image analysis method based on a convolutional neural network (CNN), there is a problem of local field of view limitation (limited CNN receptive field, difficult to capture the global relationship between cells), which leads to low diagnostic accuracy of Sjogren's syndrome. The embodiment abstracts cells and tissues into nodes and establishes spatial adjacency relationship edges to realize modeling of the global relationship between cells and the cross-scale relationship between cells and tissues, effectively capture the complex interaction between cells, and solve the problem of CNN local field of view limitation. Therefore, the embodiment can overcome the CNN local receptive field limitation, realize global topology perception by establishing the global relationship between cells, and improve the diagnostic accuracy.

[0096] Meanwhile, the embodiment proposes a tissue segmentation and cell recognition technology for pathological picture adaptive resolution, through adaptive adjustment of resolution and subgraph division, a larger pathological picture is processed to a size convenient for model processing without losing details, on this basis, the cell and tissue are recognized by using tissue segmentation based on segmentation model and cell extraction based on polygon recognition, so as to reduce the calculation complexity and balance the efficiency and details.

[0097] Based on the first embodiment of the present application, the same or similar contents as the above embodiment one can refer to the above introduction, and will not be described in detail hereinafter.

[0098] The pathological image has the characteristics of high resolution, multi-scale and complex structure, and the traditional image analysis method is difficult to effectively model the spatial and functional relationship between cells. In order to realize efficient modeling of cell-level graph structure, how to construct cell-cell graph (CCG, Cell-Cell Graph) and cell-tissue graph (CTG, Cell-Tissue Graph) based on the original pathological image and extract cell features with semantic expression ability is a big difficulty.

[0099] To solve the above problems, on the basis of the first embodiment, the embodiment can perform cell instance segmentation and tissue region recognition on each image patch, and extract a multi-modal feature vector with semantic expression ability for each cell and tissue unit, providing structured input for subsequent graph structure modeling.

[0100] Therefore, in the embodiment, the cell entity set includes single cell instances and cell multi-modal feature vectors corresponding to the single cell instances, and the tissue entity set includes tissue region instances and tissue multi-modal feature vectors corresponding to the tissue region instances; with reference to Figure 2 , Figure 2 The flowchart of the second embodiment of the automatic diagnosis method for the pathological image dry syndrome of the present application is shown in FIG. 2, and step S20 includes steps B1-B4.

[0101] Step B1, the cell contour of the optimized image block is predicted by the preset instance segmentation model, the cell instance mask is obtained, and the single cell instance is extracted from the cell instance mask;

[0102] Step B2, the cell geometric features, color texture features and spatial position features extracted for each single cell instance are fused to obtain the cell multi-modal feature vector;

[0103] Step B3, the tissue region of the optimized image block is predicted by the general segmentation model, the tissue region mask is obtained, and the tissue region instance corresponding to the tissue region mask is obtained;

[0104] Step B4, feature fusion is performed on the tissue geometric features, staining texture features and position context features extracted for each of the tissue region instances to obtain the tissue multi-modal feature vector.

[0105] It is easily understood that in the embodiment, the cell entity set can be a set containing a plurality of single cell instances and a cell multi-modal feature vector corresponding to each single cell instance, wherein the single cell instance is a single cell individual with independent boundaries and unique identification extracted from the cell instance mask, is the basic component unit of the cell entity set, and can reflect the individual morphological features of the cell.

[0106] It needs to be understood that although the traditional convolutional neural network (CNN) can complete image classification or coarse-grained target detection tasks, it is limited in processing high-density arrangement, complex morphology and fuzzy boundary of cell structure. In the embodiment, the above-mentioned preset instance segmentation model can be a Stardist model based on star contour regression, which can be used to solve the problem of cell segmentation with high-density arrangement, complex morphology and fuzzy boundary in pathological images, and can accurately predict the cell contour, perform cell instance segmentation on each optimized image block Pi_final, and output the cell instance mask.

[0107] Therefore, in the embodiment, the Stardist model can be used to predict the radius length in multiple directions from the center of each cell, and the cell geometric shape is reconstructed to determine the cell boundary, that is, the above-mentioned cell contour prediction is performed. The integer matrix output by the Stardist model, in which each unique integer corresponds to an independent cell instance, marks the pixel position range of each cell in the image block, and can be denoted as Mi_cell∈Nsp×sp (sp is the size of the image block), that is, the above-mentioned cell instance mask. Then, each unique value cj in Mi_cell can be extracted to obtain the corresponding cell region: cj∈Mi_cell, j=1, 2,..., Ni_cell, wherein Ni_cell represents the total number of cells detected in the i-th patch.

[0108] Illustratively, the diagnostic device can input any optimized image block Pi_final into the preset Stardist instance segmentation model. The Stardist model can first extract features from the image block to identify potential cell regions; then for each potential cell region, the centroid position is predicted, and the radius length in multiple directions (such as 32 directions) is predicted from the centroid; finally, the geometric contour of the cell is reconstructed according to the centroid and the radius in each direction, and the cell instance mask Mi_cell is generated. The mask is an integer matrix of sp×sp (in the embodiment, sp can be set to 512), and each unique integer in the matrix represents an independent cell instance, for example, integer "1" corresponds to the first cell instance, integer "2" corresponds to the second cell instance, and so on.

[0109] Then, traversing each pixel in the cell instance mask Mj cell, according to the uniqueness of the pixel value, the pixel set corresponding to each pixel value is extracted, and each pixel set is a pixel region of a single cell instance cj, where j = 1, 2,..., Nj cell, Nj cell represents the total number of cells detected in the optimized image block. For example, if the pixel value in the mask contains unique integers from 1 to 100, 100 single cell instances c1-c100 can be extracted.

[0110] Compared with mainstream methods such as U-Net, Mask R-CNN, Stardist has higher recall rate and boundary accuracy in dense cell scenarios, and can achieve good generalization performance without relying on a large amount of labeled data, and is suitable for large-scale pathological image analysis tasks. The specific segmentation effect can be referred to Figure 3 , Figure 3 is a schematic diagram of cell segmentation of the second embodiment of the automatic diagnosis method for pathological images of the present application. Figure 3 An irregular colored shape in the image output after Stardist processing in the embodiment can correspond to an identified single cell instance.

[0111] In order to convert the cell from a pixel-level object to a node entity that can be used for graph learning, the embodiment can extract a set of multi-modal feature vectors for each cell instance, i.e. the above-mentioned cell multi-modal feature vector, which can be a vector formed by fusing the geometric feature fj geo, color texture feature fj color and spatial position feature fj pos of the single cell instance, which can fully represent the attribute information of the cell, and provide data basis for subsequent graph structure modeling and immune cell classification, denoted as fj cell = [fj geo, fj color, fj pos] ∈ R dcell (dcell is the vector dimension, covering multiple dimensions such as geometric properties, color texture features, spatial position information, etc.).

[0112] Among them, the cell geometric feature can be a feature describing the shape and size of the cell, including the area, perimeter, aspect ratio (the ratio of the length of the long axis to the length of the short axis) and circularity (4π x area / perimeter2, the closer the value is to 1, the closer the cell is to a circular shape) of a single cell instance, which can reflect the morphological characteristics of the cell. The cell color texture feature can be a feature describing the cell color distribution and surface texture of a single cell instance, including the mean value of the RGB three channels (reflecting the overall color depth of the cell), the standard deviation (reflecting the uniformity of the cell color distribution), the histogram of oriented gradients (HOG, reflecting the edge and texture direction information of the cell) and the Haralick texture feature (reflecting the spatial correlation of the cell gray scale distribution), which can distinguish the color and texture differences of different types of cells. And the cell spatial position feature can be a feature describing the position of a single cell instance in the image block, specifically the centroid coordinates (xj, yj) of the cell, which are obtained by calculating the mean value of the pixel coordinates of the cell, and can reflect the spatial relative position relationship between cells. The combination of the above features not only contains the morphological information of the cell itself, but also integrates the spatial context relationship in the image, which can make the cell graph nodes in the subsequent graph construction have stronger semantic expression ability. These features will be used as the basis for node embedding in the subsequent graph construction process, supporting similarity calculation between cells, adjacency relationship modeling and information propagation of graph neural networks.

[0113] It can be understood that cells do not exist in isolation, but are embedded in a specific tissue environment. Therefore, in order to establish the context relationship between cells and tissues, the embodiment can further perform tissue region segmentation on each optimized image block Pi_final, and the above tissue entity set can be a set containing multiple tissue region instances and the tissue multi-modal feature vector corresponding to each tissue region instance. The tissue region instance can be an independent tissue region with specific function and structure obtained from the tissue region mask, which is the basic constituent unit of the tissue entity set and can reflect the macroscopic structural characteristics of the tissue.

[0114] The general segmentation model can also be a SAM2 model, which does not need to be retrained for specific staining patterns or tissue types to achieve accurate segmentation of tissue regions in pathological images. For example, the diagnostic device can identify and divide the tissue regions in the image block through the SAM2 model, determine the boundaries and ranges of different tissue regions, and output a tissue region mask, i.e., perform the above-mentioned tissue region prediction. The integer matrix output by the SAM2 model, in which each unique integer corresponds to an independent tissue region instance, marks the pixel position range of each tissue region in the image block, and can be denoted as Mi_tissue∈Nsp×sp, which is the above-mentioned tissue region mask. Accordingly, each unique value in Mi_tissue represents an independent tissue region, and the above-mentioned tissue instance can be represented as: tk∈Mi_tissue,k=1,2,...,Ni_tissue, where Ni_tissue represents the total number of tissue regions detected in the i-th optimized image block.

[0115] For example, the diagnostic device can input the optimized image block Pi_final into the general segmentation model SAM2. The SAM2 model performs multi-scale feature extraction on the image block through a pre-trained image feature extraction network (such as ViT-backbone), combines a learnable mask decoder, and predicts the tissue regions; by setting a suitable segmentation threshold (such as 0.5), a tissue region mask Mi_tissue is generated, which is an sp×sp integer matrix, each unique integer corresponds to an independent tissue region instance, for example, integer "1" corresponds to the first tissue region instance, and integer "2" corresponds to the second tissue region instance.

[0116] Then, each pixel in the tissue region mask Mi_tissue is traversed, and according to the uniqueness of the pixel value, a pixel set corresponding to each pixel value is extracted, and each pixel set is a pixel region of a tissue region instance t k , where k=1,2,...,Ni_tissue, and Ni_tissue represents the total number of tissue regions detected in the optimized image block. The segmentation diagram of the tissue instance can be referred to as Figure 4 , Figure 4 which is the tissue segmentation diagram of the second embodiment of the automatic diagnosis method for pathological images of the present application. Figure 4 Any irregular colored shape (different from the background color) in the image output after processing by the SAM2 model in

[0117] It is easy to understand that the above-mentioned tissue multi-modal feature vector can be a vector formed by fusing the tissue geometric feature fk_geo, the staining texture feature fk_color and the position context feature fk_context of the tissue region instance, which can comprehensively represent the attribute information of the tissue and provide a data basis for cell-tissue graph construction and cross-scale information fusion, denoted as fk_tissue=[fk_geo, fk_color, fk_context]∈R_dtissue(dtissue is the vector dimension).

[0118] Among them, the tissue geometric feature can be a feature describing the shape and size of the tissue region, including the area, perimeter and shape factor (4π×area / perimeter², reflecting the irregularity of the shape of the tissue region) of the tissue region, which can reflect the macro morphological characteristics of the tissue. The tissue staining texture feature can be an index describing the staining condition and texture feature of the tissue region, including the mean value (reflecting the staining depth of the tissue) and texture score (obtained by calculating the eigenvalue of the gray level co-occurrence matrix of the tissue region, reflecting the thickness and uniformity of the texture of the tissue) of the H (hematoxylin), E (eosin) and D (xylene) three channels in the H&E staining image, which can distinguish the staining and texture differences of different types of tissues.

[0119] And the tissue position context feature can be a feature describing the position and surrounding environment of the tissue region, including the number of surrounding cells (surrounding_cells, counting the total number of cells within a certain range outside the boundary of the tissue region) and the centroid coordinates (centroid_x, centroid_y) of the tissue region, which can reflect the spatial correlation between the tissue and the surrounding cells. These tissue features not only help to understand the macro environment in which the cells are located, but also can participate in information aggregation and propagation as context nodes when constructing the cell-tissue graph (Cell-Tissue Graph) in the subsequent, improving the understanding ability of the model to local-global relationship.

[0120] Finally, the diagnostic device can splice and combine different types of features (such as the geometric, color texture and spatial position features of the cells) of the same entity (single cell instance or tissue region instance) in a certain order to form a unified multi-modal feature vector, and in the embodiment, a direct splicing method can be used to make the feature vector contain comprehensive information of the entity.

[0121] In this embodiment, the Stardist model can be used for cell instance segmentation, and the cell boundary can be accurately reconstructed by star contour regression. Compared with traditional models such as U-Net and MaskR-CNN, the recall rate is increased by 15%-20% and the boundary accuracy is increased by 10%-15% in the high-density cell scene, effectively solving the segmentation problem caused by cell density and blurred boundary in pathological images. The general segmentation model SAM2 is used for tissue segmentation, which has zero sample migration capability and can adapt to different staining methods (such as H&E and IHC) and image differences in different laboratories. It does not need to be retrained for specific data, and the model compatibility is increased by 40%-50%, reducing the adaptation cost of cross-center application.

[0122] At the same time, the embodiment fuses the geometric, color texture, spatial position features of cells and the geometric, staining texture, position context features of tissues to construct a multi-modal feature vector. Compared with single pixel features, the feature information is increased by 3-5 times, which can more comprehensively represent the cell and tissue attributes, providing high-quality data support for subsequent graph modeling and immune cell classification, and improving the immune cell recognition accuracy by more than 20%. Moreover, the features extracted in this embodiment have clear physical meaning (such as cell area and tissue staining mean), and the correlation between features and diagnosis results can be traced back in the subsequent diagnosis process. For example, by analyzing the geometric features (such as roundness anomaly) of lymphocytes or the tissue staining texture features (such as deepening of inflammation area staining), the doctor can understand the diagnosis basis, and further improve the model interpretability.

[0123] It is easy to understand that the prior art in pathological image analysis only focuses on local pixel features or single scale (cell level or tissue level) information, and cannot establish global spatial relationships between cells and cross-scale relationships between cells and tissues, which makes it difficult for the model to understand the functional state and interaction of cells in the tissue environment. For example, it is difficult to effectively associate the positional relationship between lymphocytes and inflammatory tissues, which further affects the recognition accuracy of key lesions (such as lymphocyte aggregation focus) in Sjogren's syndrome diagnosis. At the same time, traditional graph structures are mostly single node types (such as only cell nodes), which cannot fuse multi-scale information, limiting the modeling ability of the model for complex structures of pathological images. Therefore, in a feasible implementation, the step S30 in this embodiment can include steps C1-C3:

[0124] Step C1, each single cell instance is taken as a cell graph node, and a cell-cell edge set is established between the cell graph nodes that meet the preset interval requirements;

[0125] Step C2, each tissue region instance is taken as a tissue graph node, and a cell-tissue edge set is established between the cell graph nodes and the tissue graph nodes that meet the preset coverage requirements;

[0126] Step C3, integrating the cell-cell edge set and the cell-tissue edge set to generate a heterogeneous graph structure, the nodes of the heterogeneous graph structure carrying corresponding cell multi-modal feature vectors or tissue multi-modal feature vectors.

[0127] It is easy to understand that the diagnostic device can abstract each single cell instance in the cell entity set as a node in the graph structure to obtain the above-mentioned cell graph node, which is a basic unit for constructing the cell-cell edge set, and each cell graph node carries a corresponding cell multi-modal feature vector, which can be denoted as vj (corresponding to a single cell instance cj). Meanwhile, a cell graph node set V_cell={vj_1, vj_2,..., vj_Ncell} (Ncell is the total number of cells) is formed, and the corresponding cell multi-modal feature vector fj_cell is bound to each cell graph node vj.

[0128] The above-mentioned preset interval requirement can be a spatial distance standard for the connection between the two cell graph nodes. In the embodiment, the distance between the centroids of the single cell instances corresponding to the two cell graph nodes can be less than or equal to a preset adjacency radius r, and r is 1.5-2 times the average diameter of the cells (the average value can be calculated by statistically optimizing the diameters of all cells in the image block, for example, when the average diameter is 25 pixels, r is set to 37.5-50 pixels). At this time, the diagnostic device can construct a set of undirected edges between the cell graph nodes that meet the preset interval requirement, i.e., the above-mentioned cell-cell edge set, which is used to model the spatial adjacency relationship between cells, denoted as E_cell, and the existence of the edge represents the possibility of direct spatial interaction between the corresponding cells.

[0129] In addition, the diagnostic device can abstract each tissue region instance in the tissue entity set as another kind of node in the graph structure, i.e., the above-mentioned tissue graph node, which is a basic unit for constructing the cell-tissue edge set, and each tissue graph node carries a corresponding tissue multi-modal feature vector, denoted as vk (corresponding to a tissue region instance tk). A tissue graph node set V_tissue={vk_1, vk_2,..., vk_Ntissue} (Ntissue is the total number of tissue regions) is formed, and the corresponding tissue multi-modal feature vector fk_tissue is bound to each tissue graph node vk.

[0130] Correspondingly, the above-mentioned preset coverage requirement can be a spatial position standard for the connection between the cell graph node and the tissue graph node. In the embodiment, the following two conditions can be set: first, the centroid of the single cell instance corresponding to the cell graph node is located within the tissue region instance corresponding to the tissue graph node; second, the distance between the boundary of the cell instance and the boundary of the tissue region instance is less than a preset threshold d (d can be set to 10-15 pixels, which is adjusted adaptively according to the resolution of the pathological image and the size of the cell, and the threshold can be appropriately reduced as the resolution is higher).

[0131] Therefore, the diagnostic device can establish undirected edges between the cell graph node-tissue graph node pairs satisfying the preset coverage requirement, and form an edge set, i.e., the above-mentioned cell-tissue edge set E_ct, which is used to model the cross-scale spatial correlation between cells and tissues.

[0132] Finally, the diagnostic device can combine the cell-cell edge set E_cell and the cell-tissue edge set E_ct while retaining the cell graph node set V_cell and the tissue graph node set V_tissue, form a heterogeneous graph structure containing two node types and two edge types, and ensure that the graph structure completely covers the relationships between cells and between cells and tissues. And because the cell graph nodes in the heterogeneous graph structure carry corresponding cell multi-modal feature vectors f j_cell, and the tissue graph nodes carry corresponding tissue multi-modal feature vectors f k_tissue, these feature vectors can be used as input data for subsequent graph neural network models, providing a basis for node feature optimization and relationship reasoning.

[0133] In this embodiment, the global spatial adjacency relationship between cells is modeled through the cell-cell edge set, breaking through the limitation of the local receptive field of CNN and capturing potential interactions between distant cells; the cell-tissue edge set is used to associate cells and tissues, combining microscopic cell information with macroscopic tissue background, so that the model can understand the tissue environment in which the cells are located (such as whether the lymphocytes are located in the inflammatory tissue), providing context support for subsequent immune cell classification and cluster recognition, and improving the accuracy of lesion recognition. By establishing edges through spatial adjacency relationships and coverage relationships, image noise (such as background impurities) can be effectively filtered out of node associations, for example, only connections between real cells are established, avoiding the influence of impurity pixels on the graph structure, and improving the diagnostic robustness of the model on low-quality pathology images (such as uneven staining and the presence of impurities).

[0134] At the same time, the present embodiment constructs a heterogeneous graph structure containing cell nodes and tissue nodes, which has richer information dimensions than a graph structure with a single node type, and can represent the features and relationships of microscopic individuals (cells) and macroscopic environments (tissues) at the same time. The modeling capability of the model for complex structures of pathological images is improved, and the biological structure of the pathological tissue can be simulated more realistically. Because the edges in the heterogeneous graph structure explicitly indicate the association between nodes, the subsequent graph attention network can dynamically adjust the attention weights of different nodes (such as adjacent cells and the tissue they are in) based on the connection relationship of the edges, so that the model focuses more on key nodes (such as lymphocytes near inflammatory tissue), further improving the feature optimization accuracy.

[0135] It needs to be understood that the existing CNN model needs to rely on a large amount of labeled data (medical annotation is time-consuming and expensive), and the feature extraction lacks global structure perception, resulting in low accuracy of immune cell classification; at the same time, the traditional clustering method (such as K-Means) is only based on pixel features, cannot combine the spatial relationship between cells, and is difficult to accurately identify lymphocyte clusters (a key marker for diagnosis of Sjogren's syndrome), and the model has poor interpretability, and doctors cannot trace the basis for diagnosis. Therefore, in order to realize accurate identification of cells and tissues without relying on manual annotation, and to construct an expression vector for each cell graph node. In a feasible implementation, step S40 can include steps D1-D3:

[0136] Step D1, the cell multi-modal feature vector in the heterogeneous graph structure is optimized by a self-supervised graph attention network to obtain an optimized cell feature vector; the self-supervised graph attention network is obtained by self-supervised pre-training through node masking reconstruction and contrast learning by applying structural disturbance;

[0137] It should be noted that the above self-supervised graph attention network can be a model pre-trained by combining a self-supervised learning mechanism and a graph attention network (GAT, Graph Attention Network) iteratively. In the present embodiment, the network parameters of the self-supervised graph attention network can be optimized through self-supervised pre-training tasks (node masking reconstruction, contrast learning), and then the weights of the node neighbors can be dynamically adjusted through the attention mechanism in the inference stage, so as to realize accurate optimization of the node features. Therefore, the present embodiment can be used to optimize the cell multi-modal feature vector in the heterogeneous graph structure.

[0138] In the self-supervised pre-training process of the self-supervised graph attention network, unannotated oral labial gland WSI images can be collected first, and the corresponding heterogeneous graph structure set {G1, G2,..., Gn} (n is the amount of unannotated data) can be generated according to the steps mentioned above. Each heterogeneous graph Gn=(Vn, En) contains cell nodes, tissue nodes and corresponding multi-modal feature vectors.

[0139] Then in the node masking reconstruction process, the present embodiment can randomly mask part of the dimension of the node feature vector in each heterogeneous graph structure Gn (such as masking 30% to 50% of the feature dimension), and train the model to predict the value of the masked dimension according to the unmasked feature dimension (for example, input the masked feature vector into the graph attention network (GAT), and the GAT transforms f_masked through a linear transformation matrix W, and then aggregates the neighbor node features through the attention mechanism to output the predicted masked dimension value f_pred), and adopts mean square error loss (MSE Loss) to optimize the graph attention network parameters, so that the model learns the internal correlation rules of the node features.

[0140] Regarding the contrastive learning of applying structural disturbance, the embodiment can construct different views of each heterogeneous graph Gn in two ways: one is structural disturbance (i.e., randomly deleting 10%-20% of the edges of Gn, destroying part of the node association relationship); the other is node feature disturbance (i.e., adding Gaussian noise N(0, σ²) to all node feature vectors in Gn, σ² = 0.01, or locally masking 5%-10% of the feature dimensions); then for each node, the node embedding of the original view and the node embedding of the disturbed view are taken as positive sample pairs, and the embedding of other nodes is taken as negative sample pairs; finally, the graph attention network is optimized through the contrastive loss (Contrastive Loss), so that the model learns discriminative node embedding representation.

[0141] Finally, the embodiment can jointly train the model parameters of the graph attention network by combining the node masking reconstruction loss and the contrastive loss, until the preset training round ends, the self-supervised pre-training is completed, and the self-supervised graph attention network is obtained.

[0142] Therefore, the embodiment can train the parameters (including linear transformation matrix W, attention vector a, etc.) of the self-supervised graph attention network on unannotated pathological image data through the node masking reconstruction and the contrastive learning task of applying structural disturbance, so that the model can learn the feature association rules of cells and tissues in pathological images without relying on manual annotation, and reduce the dependence on annotated data.

[0143] It is easy to understand that the self-supervised graph attention network pre-trained through self-supervised pre-training can dynamically assign attention weights to the neighbor nodes of the cell graph nodes in the heterogeneous graph structure, so as to focus on the nodes in the key area, such as lymphocytes at the lesion boundary, cells frequently interacting with tissue nodes, etc. Therefore, the output feature vector of the cell multi-modal feature vector updated by the self-supervised graph attention network is the optimized cell feature vector, which integrates the node's own features, neighbor node (other cells and the tissue where they are located) features and attention weight information, has stronger semantic expression ability and discriminativeness, and can be denoted as (corresponding to the cell graph node v i ).

[0144] Exemplarily, in the embodiment, the constructed heterogeneous graph structure G to be diagnosed can be input into the pre-trained self-supervised graph attention network. Each cell node vj (carrying a cell multi-modal feature vector f j cell ) in G is enhanced for structure perception, at this time, the self-supervised graph attention network can perform the following feature optimization process:

[0145] 1) Linear transformation: map f j cell to a high-dimensional space through a learnable linear transformation matrix W to obtain a new feature vector: f itrans = Wxfj_cell (In this embodiment, the W dimension is 256x128, and the 128-dimensional feature is mapped to 256 dimensions).

[0146] 2) Attention weight calculation: For all adjacent nodes vi (including adjacent cell nodes ci and tissue nodes ti, the adjacent node set N(i) is determined by E_cell and E_ct) of the cell graph node vj, the attention coefficient is calculated ), the calculation method is as follows:

[0147] ; (2)

[0148] Where a is a learnable attention vector; "||" represents the vector concatenation operation; N(i) represents the adjacent node set of node vj; σ(.) is a nonlinear activation function; LeakyReLU is an activation function; W is a learnable linear transformation matrix.

[0149] Then, the node is updated in combination with the attention weight:

[0150] ; (3)

[0151] Where, represents the cell or tissue vector representation fj_cell or fk_tissue obtained previously, represents the new feature vector, and note that only the cell feature vector is calculated and saved. For cell node i, its refined feature representation is denoted as .

[0152] 3) Multi-head attention aggregation: To further enhance the expression ability of the model, a multi-head attention mechanism (Multi-head Attention) is used for feature enhancement based on formula (3). For example, 8 attention heads (i.e., the number of attention heads H=8) can be used, and each attention head (through independent weight matrix and attention vector ) independently calculates the attention weight and aggregates the adjacent node features. The output of the 8 attention heads is concatenated and then linearly transformed to obtain an optimized cell feature vector, which is represented as follows:

[0153] ; (4)

[0154] Where h=1 to 8, is the weight of the hth attention head. Similarly, only the cell feature vector is calculated and saved. Specifically, for cell node i, its refined feature representation is denoted as , which serves as a high-quality input feature for subsequent immune cell classification and structure recognition tasks.

[0155] In summary, the module assigns attention weights to neighbor nodes dynamically in the inference process through a self-supervised pre-trained graph attention network (GAT), enabling the model to focus on nodes in key areas, such as lymphocytes at the lesion boundary, cells that frequently interact with tissue graph nodes, etc. By aggregating the neighborhood information through a multi-head attention mechanism, the final refined cell feature representation with structural perception ability is obtained .

[0156] Step D2, performing immune cell classification according to the optimized cell feature vector, obtaining a plurality of immune cell graph nodes;

[0157] It can be understood that in the diagnosis of SS and other immune-related diseases, the focus formed by the aggregation of immune cells such as lymphocytes is a key pathological scoring indicator. Therefore, how to identify these cell clusters with clinical significance from the graph structure and associate them with the final diagnostic decision is a key challenge of this research. Traditional methods rely on simple cell counting or region aggregation, making it difficult to capture complex patterns at the structural level.

[0158] Compared with the prior art, the embodiment can distinguish whether a cell is an immune cell (in this embodiment, specifically lymphocytes, which are a key indicator for Sjogren's syndrome diagnosis) based on the optimized cell feature vector through a binary classification model, and output the probability and classification result of the cell belonging to an immune cell. After immune cell classification, the cell graph nodes determined to be immune cells are the immune cell graph nodes described above. These nodes carry the optimized cell feature vector and are the basis for subsequent clustering analysis, denoted as vi_immune (corresponding to cell graph nodes with a classification result of immune cells).

[0159] Illustratively, the embodiment can pre-construct a binary classification neural network containing 2 fully connected layers (Fully Connected Layers), with the input layer dimension being the dimension of the optimized cell feature vector (in this embodiment, 256 dimensions), the output dimension of the first fully connected layer being 128, and the activation function being LeakyReLU (negative slope set to 0.2) to increase the non-linear expression capability; the output dimension of the second fully connected layer being 1, and the activation function being Sigmoid, outputting the probability yi ∈ [0, 1] of the cell belonging to a lymphocyte. Then a small amount of labeled data (in this embodiment, 100 WSI images of cell labeling, of which 50 are lymphocytes and 50 are non-lymphocytes) is used to train the binary classification network, and the cross-entropy loss function is used for iterative training.

[0160] In actual use, the embodiment can input the optimized cell feature vectors of all cell nodes in the heterogeneous graph to be diagnosed into the binary classification neural network to obtain the probability yi ∈ [0, 1] of each cell node belonging to an immune cell (lymphocyte in this embodiment). Input the trained binary classification network to output the probability yi of each cell; and set the classification confidence threshold θ = 0.7, if yi > θ, then determine that the cell is an immune cell (lymphocyte), and the corresponding cell graph node can be marked as an immune cell graph node vi_immune, and then collect all immune cell graph nodes to form a set V_immune = {vi_immune | yi > θ}.

[0161] Step D3, performing a preset clustering analysis based on the plurality of immune cell graph nodes to output a Sjogren's syndrome state determination result.

[0162] It should be understood that in the prior art, the traditional clustering method (such as K-Means) only performs clustering based on cell pixel features, ignores the spatial adjacency relationship between cells, and cannot accurately identify the key "lymphocyte aggregation focus" (cell mass requiring close spatial connection) in Sjogren's syndrome diagnosis, resulting in missed or false detection of lesions. Therefore, the embodiment can use a graph community detection algorithm (Louvain algorithm in the embodiment) to cluster immune cell graph nodes and identify closely connected immune cell clusters, i.e., perform the above-mentioned preset clustering analysis. The clustering is based on the spatial adjacency relationship and feature similarity between nodes, aiming to mine the aggregation pattern of immune cells (a key pathological feature of Sjogren's syndrome diagnosis).

[0163] The above-mentioned Sjogren's syndrome state determination result can be the number and density of immune cell clusters obtained from the clustering analysis result, combined with the Sjogren's syndrome diagnosis standard (the number of immune cells in each 4mm² tissue is more than 50), to output the "positive" or "negative" diagnosis conclusion, which is the final output of the entire diagnosis process.

[0164] In a feasible implementation, in the embodiment, step D3 can include steps D31-D33:

[0165] Step D31, constructing an immune cell graph node subgraph based on the plurality of immune cell graph nodes;

[0166] Step D32, running a preset community detection algorithm on the immune cell graph node subgraph to perform clustering and obtain immune cell clusters;

[0167] It should be noted that the embodiment can first extract the edges between immune cell graph nodes in the heterogeneous graph G based on the immune cell graph node set V_immune, i.e., E_immune = E_cell ∩ (V_immune × V_immune), and construct a subgraph structure containing only immune cell graph nodes and the original connection relationship between these nodes, i.e., the above-mentioned immune cell subgraph G_immune = (V_immune, E_immune).

[0168] Then, the diagnostic device can run a preset community detection algorithm on G_immune, and preferably a Louvain community detection algorithm, to iteratively optimize community division and identify closely connected substructures (communities) in the graph, thereby accurately identifying multiple highly connected lymphocyte clusters, obtaining clusters of immune cells aggregated to form the immune cell clusters described above, each of which represents a potential lesion area (lymphocyte aggregation is a key lesion marker in Sjogren's syndrome), and can be denoted as C_q (q = 1, 2,..., p, p is the number of clusters).

[0169] Illustratively, in the above clustering process, each immune cell graph node can be first initialized as an independent community and assigned a unique initial community identifier c_i; then each node v_i_immune is traversed and attempted to be assigned to a neighboring community, and the change in modularity ΔQ before and after assignment is calculated, where the modularity formula is Q = (1 / (2m)) x Σ[A_ij-(k_i*k_j) / (2m)] x δ(c_i,c_j), A_ij is an adjacency matrix element, k_i is the node degree, m is the total number of edges, and δ(c_i,c_j) is a community membership indicator function); where if ΔQ > 0, the node is assigned to the community, otherwise the original community is maintained; the above steps are repeated until the modularity no longer improves, and multiple immune cell clusters C1, C2,..., C_p (p is the number of clusters) are obtained.

[0170] Step D33, outputting a Sjogren's syndrome state determination result according to the spatial distribution information of the immune cell clusters and the number of immune cells in the immune cell clusters.

[0171] It can be understood that the spatial distribution information can be information such as the position, coverage range, and relative relationship with the surrounding tissue of the immune cell clusters in the original digital pathology slide image, including the centroid coordinates, circumscribed rectangular range, and image block number of the clusters, which can be used to locate the lesion area and assist the doctor in verification. The total number of immune cell graph nodes included in each immune cell cluster, i.e., the number of immune cells described above, is the core data for calculating the immune cell density and determining whether it meets the Sjogren's syndrome diagnostic criteria, and can be denoted as N_q (corresponding to cluster C_q).

[0172] Finally, the diagnostic device can output a "positive" or "negative" conclusion based on the density of the immune cell clusters (the number of immune cells per 4 mm² (square millimeter)) and the clinical diagnostic criteria, and form a final judgment on the overall pathological state of the WSI sample regarding Sjogren's syndrome.

[0173] Illustratively, referring to Figure 5 It can be seen that Figure 5This is a schematic diagram of the Sjögren's syndrome status determination process in the second embodiment of the automatic diagnosis method for pathological images of Sjögren's syndrome in this application. After performing adaptive resolution patch (image block) division and extraction, identifying lymphocytes and lesions, the diagnostic device counts the number of immune cells N_q in each cluster C_q (q=1 to p) in each image block.

[0174] Simultaneously, based on the centroid coordinates (x_i, y_i) of all nodes within the cluster, the centroid coordinates (xq_centroid, yq_centroid) = (Σx_i / N_q, Σy_i / N_q) of the cluster can be calculated to determine the original image patch number where the cluster is located. Then, combined with the area S_patch of the corresponding image patch (in this embodiment, 512×512 pixels correspond to an actual area of ​​0.1mm²), the number of immune cells within each 4mm² is calculated: N_4mm² = N_q × (4 / S_patch).

[0175] Wherein, if an immune cell cluster C_q is detected in any image patch satisfying N_4mm²>50 (i.e. Figure 5 If the number of lymphocytes shown is >50 / 4mm², then the WSI sample is considered positive for Sjögren's syndrome (SS positive); if all image patches contain clusters that satisfy N_4mm²≤50 (i.e., ... Figure 5 If the number of lymphocytes shown is <= 50 / 4mm², it is determined to be negative for Sjögren's syndrome (SS negative), and the final diagnostic result is output. At this time, the final Sjögren's syndrome status determination result may include the following: basic sample information (such as sample number, WSI scan parameters); statistical results of immune cell clusters (number of clusters, location of each cluster, N_q, N_{4mm^2}_q), in this embodiment, it may be the number of lymphocytes in the lesion; visualization image of key clusters (marking the location and range of the clusters in the original image block); final diagnostic conclusion (positive / negative) and determination criteria, for example, outputting "the number of lymphocytes in cluster C3 within 4mm² is 62, which meets the positive criterion".

[0176] In this embodiment, the traditional clustering methods (such as K-Means) in the prior art only cluster based on cell pixel features, ignoring the spatial adjacency relationship between cells. This makes it impossible to accurately identify the key "lymphocyte aggregation focus" (cell clusters that need to be closely connected in space) in the diagnosis of Sjögren's syndrome, leading to missed or false detections of lesions. The Louvain community detection algorithm can be used to cluster based on the spatial adjacency relationship (edge ​​set) and feature similarity between cells. Compared with the existing K-Means algorithm, this embodiment improves the accuracy of lymphocyte aggregation focus identification and reduces the false detection rate, which can effectively capture the key pathological markers of Sjögren's syndrome.

[0177] And for the traditional diagnosis process relies on manual statistics lymphocyte quantity, low efficiency and easy to be influenced by subjective factors (such as the judgment standard of the doctor to 'aggregation' is inconsistent), and cannot associate the spatial position of the cluster with the tissue background, it is difficult to provide clear diagnostic basis, the embodiment calculates the association between the physical area of the image block and the number of cluster cells, standardizes the number of immune cells to the 'per 4mm²' index, directly matches the clinical diagnosis standard, avoids the subjectivity of manual statistics, and improves the clinical recognition of the diagnosis result.

[0178] Finally, due to the lack of standardized calculation of cluster density in existing methods, it cannot directly match the clinical diagnosis standard (such as the cell number threshold per 4mm²), resulting in a diagnosis result that is disconnected from clinical needs. The embodiment can associate the tissue environment where the cluster is located (such as whether it is located around the salivary duct) through the mapping of the cluster position and the image block and the tissue area, providing more pathological context information for doctors to assist in judging the clinical significance of the lesion. And the final output of the diagnosis report includes the position, number, density and visualization result of the cluster, doctors can intuitively trace the diagnosis basis (such as 'the cluster density at a certain position is over-standard'), solving the 'black box' problem of traditional models; At the same time, the automatic statistical and judgment process shortens the single sample diagnosis time from 30-60 minutes to 5-10 minutes, greatly improving the diagnosis efficiency;

[0179] In summary, the embodiment proposes a self-supervised graph learning representation mechanism for cells and tissues, which reduces the dependence on a large amount of labeled data and improves the generalization ability of the model, which helps the model to learn effective feature representation in the absence of labels, thereby enhancing its performance stability when facing new data. At the same time, the self-supervised graph attention network combines the node's own features and neighbor (cell+tissue) features, focusing on key associated nodes (such as cells near inflammatory tissue) with attention weights, and the discriminability of the optimized feature vector is significantly improved. In addition, the embodiment proposes a community detection-based cell cluster structure recognition, which extracts subgraphs from nodes predicted as immune cells and detects tightly connected clusters for density, shape and other analysis, which are used as auxiliary diagnostic features in the final classification decision process, not only improving the diagnostic accuracy, but also providing structural-level explanation, enhancing the model's explainability.

[0180] The embodiment discloses that the cell contour of an optimized image block is predicted by a preset instance segmentation model, a cell instance mask is obtained, and a single cell instance is extracted from the cell instance mask; cell geometric features, color texture features and spatial position features extracted for each single cell instance are fused to obtain a cell multi-modal feature vector; an organization region mask is obtained by predicting the organization region of the optimized image block through a general segmentation model, and an organization region instance corresponding to the organization region mask is obtained; organization geometric features, staining texture features and position context features extracted for each organization region instance are fused to obtain an organization multi-modal feature vector.

[0181] The embodiment also discloses that each single cell instance is taken as a cell graph node, and a cell-cell edge set is established between cell graph nodes that meet preset interval requirements; each organization region instance is taken as an organization graph node, and a cell-organization edge set is established between cell graph nodes and organization graph nodes that meet preset coverage requirements; the cell-cell edge set and the cell-organization edge set are integrated to generate a heterogeneous graph structure, and the nodes of the heterogeneous graph structure carry corresponding cell multi-modal feature vectors or organization multi-modal feature vectors.

[0182] Meanwhile, the embodiment optimizes the cell multi-modal feature vector in the heterogeneous graph structure through a self-supervised graph attention network to obtain an optimized cell feature vector; the self-supervised graph attention network is obtained through contrastive learning of node masking reconstruction and structure disturbance after self-supervised pre-training; immune cell classification is performed according to the optimized cell feature vector to obtain a plurality of immune cell graph nodes; an immune cell graph node subgraph is constructed based on the plurality of immune cell graph nodes; a preset community detection algorithm is run on the immune cell graph node subgraph for clustering to obtain immune cell clusters; and a dry syndrome state judgment result is output according to the spatial distribution information of the immune cell clusters and the number of immune cells in the immune cell clusters.

[0183] To sum up, the embodiment utilizes the spatial relationship expression capability of the graph structure, combines the community detection algorithm to identify lymphocyte aggregation clusters, accurately captures the key spatial topological features (lymphocyte aggregation focus) in SS diagnosis, and improves the diagnosis accuracy. The self-supervised pre-training mechanism is introduced, the graph attention network is trained on unannotated data through the tasks of node masking reconstruction and contrastive learning, the model can learn effective feature representation, the demand for large-scale artificial annotation data is significantly reduced, and the cost is reduced. In the whole diagnosis process, from cell-tissue segmentation to graph structure construction and immune cell cluster identification, the feature extraction and decision basis (such as the number of lymphocytes and the aggregation density) of each step are clear and traceable, and a diagnosis report containing specific image block analysis (position, cell count, density) can be generated, which is convenient for doctors to understand and verify, and solves the CNN “black box” problem.

[0184] Exemplarily, in order to facilitate understanding of the technical concept or technical principle of the pathological image Sjogren's syndrome automatic diagnosis method combined with the above embodiment one and embodiment two, please refer to Figure 6 , Figure 6 The figure is a process schematic diagram of the pathological image Sjogren's syndrome automatic diagnosis method of the present application, and the specific process is as follows:

[0185] 1. Image preprocessing:

[0186] Firstly, multi-scale tissue region extraction is performed on the original digital pathology section image (for example, H&E image). The original digital pathology section image is down-sampled according to a scaling factor s e (0, 1) to obtain a low-resolution image. The scaling factor s can be adaptively set according to the size of the original digital pathology section image. For example, when the original image is 100,000 x 100,000 pixels, s can be set to 0.1 to obtain a low-resolution image of 10,000 x 10,000 pixels.

[0187] Then, the Segment Anything Model 2 (SAM2) model is used to coarsely locate the tissue region of the low-resolution image to obtain an initial tissue mask. The mask is a binary matrix that marks the pixel positions in the low-resolution image that belong to the tissue region.

[0188] Next, the initial tissue mask T is mapped back to the coordinate system of the original WSI image to obtain a target tissue mask. Based on the target tissue mask, the original WSI image is divided into regions using a sliding window or a density clustering-based method, and a plurality of 512 x 512 pixel initial image blocks are cropped. Finally, data augmentation processing is performed on the initial image blocks, including color normalization, geometric transformation, and noise injection, to obtain optimized image blocks.

[0189] 2. Cell-tissue collaborative segmentation:

[0190] The Stardist model based on star contour regression is used to segment the optimized image blocks to obtain cell detection results (Cell Detection Result). As shown in Figure 6 , the cell analysis result can include single cell instances, i.e. Figure 6 single cell features (Feature of the single cell) and cell multi-modal feature vectors, i.e. Figure 6Feature of surrounding cells. Specifically, the model reconstructs the cell geometry by predicting the radius length in multiple directions from each cell centroid, obtains the cell instance mask (an integer mask matrix, each unique value represents an independent cell instance), and extracts single cell instances from the mask. A multi-modal feature vector is extracted for each single cell instance, including geometric features, color texture features, and spatial position features. These features are fused to obtain a cell multi-modal feature vector, thereby forming a cell entity set.

[0191] Then, the SAM2 model is used to obtain the tissue detection result (Tissue Detection Result), which contains Figure 6 Feature of surrounding tissue, i.e., the above-mentioned tissue multi-modal feature vector. The optimized image block is used to predict the tissue region, obtain a tissue region mask (an integer mask matrix, each unique value represents an independent tissue region), and extract tissue region instances from it. A multi-modal feature vector is extracted for each tissue region instance, including geometric features, staining texture features, and position context features. These features are fused to obtain a tissue multi-modal feature vector, thereby forming a tissue entity set.

[0192] 3. Heterogeneous graph structure construction:

[0193] Each single cell instance is taken as a cell graph node, and the cell centroid distance corresponding to any two cell graph nodes is calculated. If the distance is less than or equal to the preset adjacency radius (usually 1.5-2 times the average diameter of the cell, for example, when the average diameter of the cell is 20 pixels, the preset adjacency radius can be set to 30-40 pixels), a undirected edge is established between the two cell graph nodes, forming a cell-cell edge set.

[0194] Each tissue region instance is taken as a tissue graph node, and it is determined whether the centroid of the cell corresponding to the cell graph node is located within the tissue corresponding to the tissue graph node, or the distance between the boundary of the cell and the boundary of the tissue is less than a preset threshold. If any of the conditions is met, a cross-scale undirected edge is established between the cell graph node and the tissue graph node, forming a cell-tissue edge set.

[0195] The cell-cell edge set and the cell-tissue edge set are integrated to obtain a heterogeneous graph structure, and each node carries a corresponding multi-modal feature vector (the cell graph node carries a cell multi-modal feature vector, and the tissue graph node carries a tissue multi-modal feature vector).

[0196] 4. Sjogren's syndrome state determination:

[0197] The self-supervised graph attention network (GAT) is used to optimize the cell multi-modal feature vector in the heterogeneous graph structure. The GAT is pre-trained through self-supervision, and the pre-training tasks include node masking reconstruction (randomly masking part of the dimensions of the node features, training the model to predict the masked features, and using the mean square error loss to optimize the parameters) and contrastive learning (constructing different views of the graph through graph structure disturbance and node feature disturbance, using cosine similarity as the metric, and optimizing the objective function to improve the discriminability of node embedding). After training, the node features in the heterogeneous graph structure are input into the GAT, which dynamically adjusts the neighbor node weight through the attention mechanism, updates and optimizes the cell feature vector, and obtains the optimized cell feature vector.

[0198] Then the optimized cell feature vector is classified by the pre-constructed binary classification neural network (Classification MLP), which includes a fully connected layer (mapping the input feature vector to a higher dimensional space), a LeakyReLU activation function (increasing the non-linear expression ability), and an output layer (containing a neuron, using the Sigmoid activation function to output the probability yi∈[0,1] that the cell belongs to lymphocytes), and the network is optimized with the cross-entropy loss function. Cells with a prediction probability greater than the classification confidence threshold θ (θ can be set to 0.7) are labeled as immune cells, and multiple immune cell graph nodes are obtained, i.e. Figure 6 the classification result (Classification Result) shown in FIG.

[0199] Based on the multiple immune cell graph nodes, an immune cell graph node subgraph is constructed, and the Louvain community detection algorithm is run on the immune cell graph node subgraph for clustering. The algorithm identifies multiple immune cell clusters (i.e. Figure 6 the Group Result shown in FIG.

[0200] In summary, existing pathological image analysis methods based on convolutional neural networks (CNNs) suffer from several problems, including limited local field of view (CNNs have limited receptive fields, making it difficult to capture global relationships between cells), neglect of spatial topological relationships (they cannot effectively capture specific spatial arrangement patterns of cells, such as the focal points of lymphocyte aggregation in the diagnosis of Sjögren's syndrome), heavy reliance on data annotation (requiring a large amount of professionally annotated data, which is time-consuming and expensive), and poor interpretability (the classification results are difficult to trace back to specific diagnostic criteria). These issues result in low diagnostic accuracy and limited application of Sjögren's syndrome.

[0201] This application constructs a heterogeneous graph structure, abstracting cells and tissues into nodes and establishing spatial adjacency edges. This enables the modeling of global relationships between cells and cross-scale relationships between cells and tissues, effectively capturing complex interactions between cells and solving the problem of limited local field of view in CNNs. Furthermore, by utilizing the inherent spatial relationship expression capabilities of graph structures and combining them with community detection algorithms to identify lymphocyte clusters, it accurately captures key spatial topological features (lymphocyte aggregation focal points) in SS diagnosis, thereby improving diagnostic accuracy.

[0202] Simultaneously, a self-supervised pre-training mechanism is introduced. Through node masking reconstruction and contrastive learning tasks, the graph attention network is trained on unlabeled data, enabling the model to learn effective feature representations. This significantly reduces the need for large-scale manually labeled data, lowering costs. Throughout the diagnostic process, from cell-tissue segmentation and graph structure construction to immune cell cluster identification, the feature extraction and decision-making basis (such as lymphocyte count and aggregation density) at each step are clearly traceable. Furthermore, it can generate diagnostic reports containing specific image patch analyses (location, cell count, density), facilitating physician understanding and verification, and addressing the "black box" problem of CNNs. By automating image preprocessing, segmentation, graph learning, and judgment processes, it replaces traditional manual histopathological assessments, significantly improving diagnostic efficiency. Simultaneously, the application of multimodal feature fusion, attention mechanism optimization, and community detection clustering technologies further enhances diagnostic accuracy, aiding in the early and accurate diagnosis of Sjögren's syndrome, improving patient prognosis, and forming an end-to-end intelligent analysis process from image to diagnosis.

[0203] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the automatic diagnosis method for pathological images of Sjögren's syndrome in this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0204] This application also provides an automatic diagnostic device for Sjögren's syndrome in pathological images; please refer to [reference needed]. Figure 7 , Figure 7 This is a schematic diagram of the module structure of the automatic diagnostic device for Sjögren's syndrome in pathological images according to an embodiment of this application. In this embodiment, the device includes:

[0205] The image preprocessing module 701 is configured to perform image preprocessing on the original digital pathology slice image to obtain an optimized image block.

[0206] The image segmentation module 702 is configured to perform cell-tissue collaborative segmentation on the optimized image block to obtain a cell entity set and a tissue entity set.

[0207] The graph learning module 703 is configured to map the cell entity set and the tissue entity set into a heterogeneous graph structure, wherein the heterogeneous graph structure has cell graph nodes and tissue graph nodes as vertices and has spatial adjacency relationships as edges.

[0208] The identification module 704 is configured to determine the Sjogren's syndrome state based on the heterogeneous graph structure.

[0209] As an implementable manner, in the embodiment, the image preprocessing module 701 is further configured to perform multi-scale tissue region extraction on the original digital pathology slice image to obtain a target tissue mask; perform region division on the original digital pathology slice image based on the target tissue mask to obtain an initial image block; and perform data enhancement processing on the initial image block to obtain the optimized image block.

[0210] As an implementable manner, in the embodiment, the image segmentation module 702 is further configured to perform cell contour prediction on the optimized image block by using a preset instance segmentation model to obtain a cell instance mask and extract single cell instances from the cell instance mask; perform feature fusion on cell geometric features, color texture features and spatial position features extracted from each single cell instance to obtain a cell multi-modal feature vector; perform tissue region prediction on the optimized image block by using a general segmentation model to obtain a tissue region mask and obtain tissue region instances corresponding to the tissue region mask; and perform feature fusion on tissue geometric features, staining texture features and position context features extracted from each tissue region instance to obtain a tissue multi-modal feature vector.

[0211] As an implementable manner, in the embodiment, the graph learning module 703 is further configured to take each single cell instance as a cell graph node and establish a cell-cell edge set between cell graph nodes that meet preset interval requirements; take each tissue region instance as a tissue graph node and establish a cell-tissue edge set between cell graph nodes and tissue graph nodes that meet preset coverage requirements; and integrate the cell-cell edge set and the cell-tissue edge set to generate a heterogeneous graph structure, wherein the nodes of the heterogeneous graph structure carry corresponding cell multi-modal feature vectors or tissue multi-modal feature vectors.

[0212] As an implementable manner, in this embodiment, the recognition module 704 is further configured to optimize the cell multi-modal feature vector in the heterogeneous graph structure through a self-supervised graph attention network to obtain an optimized cell feature vector; the self-supervised graph attention network is obtained through self-supervised pre-training by node masking reconstruction and contrastive learning with structural disturbance; immune cell classification is performed according to the optimized cell feature vector to obtain a plurality of immune cell graph nodes; and a preset clustering analysis is performed based on the plurality of immune cell graph nodes to output a dry syndrome state judgment result.

[0213] As an implementable manner, in this embodiment, the recognition module 704 is further configured to construct an immune cell graph node subgraph based on the plurality of immune cell graph nodes; a preset community detection algorithm is run on the immune cell graph node subgraph to perform clustering to obtain an immune cell cluster; and a dry syndrome state judgment result is output according to the spatial distribution information of the immune cell cluster and the number of immune cells in the immune cell cluster.

[0214] As an implementable manner, in this embodiment, the image preprocessing module 701 is further configured to perform down-sampling on the original digital pathology slice image to obtain a low-resolution optimized digital pathology slice image; perform coarse positioning of the tissue region on the optimized digital pathology slice image to obtain an initial tissue mask; and map the initial tissue mask back to the coordinate system of the original digital pathology slice image to obtain a target tissue mask.

[0215] The pathological image dry syndrome automatic diagnosis device provided in the present application adopts the pathological image dry syndrome automatic diagnosis method in the above embodiments, and can solve the technical problem of low analysis accuracy caused by the difficulty in capturing the global relationship and spatial topological relationship between cells in the prior art. Compared with the prior art, the device models the spatial association of cells and tissues through a heterogeneous graph structure, enhances the feature expression capability in combination with a self-supervised graph attention network, and realizes accurate identification of lymphocyte aggregation areas based on community detection, thereby significantly improving the accuracy and interpretability of dry syndrome diagnosis. The other technical features of the device are the same as those disclosed in the above method embodiments, and will not be described here.

[0216] The present application provides a pathological image dry syndrome automatic diagnosis device, which comprises at least one processor and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the pathological image dry syndrome automatic diagnosis method in the above embodiment one.

[0217] Reference will be made to the following Figure 8The diagram illustrates a structural schematic suitable for implementing an automated diagnostic device for Sjögren's syndrome in pathological images according to embodiments of this application. The automated diagnostic device for Sjögren's syndrome in pathological images according to embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 8 The illustrated automated diagnostic device for Sjögren's syndrome in pathological images is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0218] like Figure 8 As shown, the automated diagnostic device for Sjögren's syndrome in pathological images may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1002 or a program loaded from storage device 1003 into random access memory (RAM) 1004. The random access memory 1004 also stores various programs and data required for the operation of the automated diagnostic device for Sjögren's syndrome in pathological images. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the automated diagnostic device for Sjögren's syndrome in pathological images to communicate wirelessly or wiredly with other devices to exchange data. Although the automated diagnostic device for Sjögren's syndrome in pathological images with various systems is shown in the figure, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems may be implemented alternatively.

[0219] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a pathological image Sjogren's syndrome automatic diagnosis program product, which includes a pathological image Sjogren's syndrome automatic diagnosis program carried on a computer readable medium, and the pathological image Sjogren's syndrome automatic diagnosis program contains program codes for executing the method shown in the flowchart. In such embodiments, the pathological image Sjogren's syndrome automatic diagnosis program can be downloaded and installed from the network through a communication device, or installed from the storage device 1003, or installed from the read-only memory 1002. When the pathological image Sjogren's syndrome automatic diagnosis program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiments disclosed in the present application are executed.

[0220] The pathological image Sjogren's syndrome automatic diagnosis device provided by the present application adopts the pathological image Sjogren's syndrome automatic diagnosis method in the above-mentioned embodiments, and can solve the technical problem of low analysis accuracy caused by the difficulty in capturing the global relationship and spatial topological relationship between cells in the prior art. Compared with the prior art, the pathological image Sjogren's syndrome automatic diagnosis device provided by the present application has the same beneficial effects as the pathological image Sjogren's syndrome automatic diagnosis method provided by the above-mentioned embodiments, and other technical features in the pathological image Sjogren's syndrome automatic diagnosis device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0221] It should be understood that various parts of the present application can be realized by hardware, software, firmware or a combination thereof. In the description of the above-mentioned embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0222] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0223] The present application provides a storage medium having computer readable program instructions (i.e. pathological image Sjogren's syndrome automatic diagnosis program) stored thereon, the computer readable program instructions being used to execute the pathological image Sjogren's syndrome automatic diagnosis method in the above-mentioned embodiments.

[0224] The storage medium provided in the present application may, for example, be a U disk, but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination of the above. More specific examples of the storage medium may, for example, include, but are not limited to, an electrical connection having one or more conductive wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present embodiment, the storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the storage medium can be transmitted in any suitable medium, including, but not limited to, an electrical wire, an optical cable, an RF (Radio Frequency) cable, or any suitable combination of the above.

[0225] The storage medium described above can be included in the pathological image dry syndrome automatic diagnosis device; or can exist separately and not be assembled into the pathological image dry syndrome automatic diagnosis device.

[0226] The storage medium described above carries one or more programs, which, when executed by the pathological image dry syndrome automatic diagnosis device, cause the pathological image dry syndrome automatic diagnosis device to: automatically diagnose the pathological image dry syndrome.

[0227] The pathological image dry syndrome automatic diagnosis program code for performing the operations of the present application can be written in one or more programming languages or combinations of the same, including an object-oriented programming language such as Java, Smalltalk, C++, or a conventional procedural programming language such as the "C" language or a similar programming language. The program code can be executed entirely on the user computer, partially on the user computer, as a separate software package, partially on the user computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user computer through any kind of network, including a LAN (Local Area Network) or a WAN (Wide Area Network), or can be connected to an external computer (for example, through the Internet using an Internet service provider).

[0228] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and pathological image Sjogren's syndrome automatic diagnosis program products according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks noted in succession can in fact be executed substantially concurrently or can sometimes be executed in reverse order, depending on the functionality involved. It will also be noted that each block of the block diagrams and / or flowcharts, and combinations thereof, can be implemented by dedicated hardware-based systems which perform the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0229] The modules involved in the embodiments of the present application can be implemented in the form of software or in the form of hardware. Among them, the name of the module does not constitute a limitation to the unit itself in some cases.

[0230] The readable storage medium provided by the present application is a storage medium, and the storage medium stores computer readable program instructions (i.e., pathological image Sjogren's syndrome automatic diagnosis program) for executing the pathological image Sjogren's syndrome automatic diagnosis method described above, which can solve the technical problem of low analysis accuracy caused by the difficulty in capturing the global relationship and spatial topological relationship between cells in the prior art. Compared with the prior art, the beneficial effects of the storage medium provided by the present application are the same as those of the pathological image Sjogren's syndrome automatic diagnosis method provided by the above-mentioned embodiments, and are not described here.

[0231] The above is only some embodiments of the present application, and does not limit the scope of the solutions of the present application, and any equivalent structural transformation made by using the contents of the specification and drawings of the present application, or direct / indirect application in other related technical fields is included in the protection scope of the present application.

Claims

1. A method for automatically diagnosing pathological image dry eye syndrome, characterized by, The method comprises: performing image preprocessing on the original digital pathology slice image to obtain an optimized image block; performing cell-tissue collaborative segmentation on the optimized image block to obtain a cell entity set and a tissue entity set; mapping the cell entity set and the tissue entity set into a heterogeneous graph structure, wherein the heterogeneous graph structure has cell graph nodes and tissue graph nodes as vertices and spatial adjacency relationships as edges; determining a dry syndrome state based on the heterogeneous graph structure; the step of mapping the cell entity set and the tissue entity set into a heterogeneous graph structure comprises: taking each single cell instance as a cell graph node and establishing a set of cell-cell edges between the cell graph nodes that meet the preset interval requirement; taking each tissue region instance as a tissue graph node and establishing a set of cell-tissue edges between the cell graph nodes and the tissue graph nodes that meet the preset coverage requirement; integrating the set of cell-cell edges and the set of cell-tissue edges to generate a heterogeneous graph structure, wherein the nodes of the heterogeneous graph structure carry corresponding cell multi-modal feature vectors or tissue multi-modal feature vectors; the step of determining a dry syndrome state based on the heterogeneous graph structure comprises: optimizing the cell multi-modal feature vectors in the heterogeneous graph structure through a self-supervised graph attention network to obtain optimized cell feature vectors; the self-supervised graph attention network is obtained through self-supervised pre-training by node masking reconstruction and contrast learning with structure disturbance; performing immune cell classification according to the optimized cell feature vectors to obtain a plurality of immune cell graph nodes; performing preset clustering analysis based on the plurality of immune cell graph nodes to output a dry syndrome state determination result; the step of performing preset clustering analysis based on the plurality of immune cell graph nodes to output a dry syndrome state determination result comprises: constructing an immune cell graph node subgraph based on the plurality of immune cell graph nodes; running a preset community detection algorithm on the immune cell graph node subgraph to obtain immune cell clusters; outputting a dry syndrome state determination result according to the spatial distribution information of the immune cell clusters and the number of immune cells in the immune cell clusters.

2. The method of claim 1, wherein the pathological image of Sjogren's syndrome is automatically diagnosed, and the method is characterized by, The cell entity set comprises single cell instances and cell multi-modal feature vectors corresponding to the single cell instances, and the tissue entity set comprises tissue region instances and tissue multi-modal feature vectors corresponding to the tissue region instances; the step of performing cell-tissue collaborative segmentation on the optimized image block to obtain a cell entity set and a tissue entity set comprises: performing cell contour prediction on the optimized image block through a preset instance segmentation model to obtain a cell instance mask, and extracting single cell instances from the cell instance mask; performing feature fusion on the cell geometric features, color texture features, and spatial position features extracted for each single cell instance to obtain the cell multi-modal feature vectors; performing tissue region prediction on the optimized image block through a general segmentation model to obtain a tissue region mask and obtain tissue region instances corresponding to the tissue region mask; The tissue geometric features, staining texture features and position context features extracted for each tissue region instance are fused to obtain the tissue multi-modal feature vector.

3. The automatic diagnostic method for Sjögren's syndrome in pathological images as described in claim 1, characterized in that, The step of performing image preprocessing on the original digital pathology slice image to obtain an optimized image block comprises: Performing multi-scale tissue region extraction on the original digital pathology slice image to obtain a target tissue mask; Based on the target tissue mask, the original digital pathology slice image is divided into regions to obtain an initial image block; The initial image block is subjected to data enhancement processing to obtain an optimized image block.

4. The method of claim 3, wherein the pathological image is a fundus image. The step of performing multi-scale tissue region extraction on the original digital pathology slice image to obtain a target tissue mask comprises: Performing down-sampling on the original digital pathology slice image to obtain a low-resolution optimized digital pathology slice image; Performing tissue region coarse positioning on the optimized digital pathology slice image to obtain an initial tissue mask; The initial tissue mask is mapped back to the coordinate system of the original digital pathology slice image to obtain a target tissue mask.

5. An automatic diagnostic device for Sjögren's syndrome in pathological images, characterized in that, The pathological image Sjogren's syndrome automatic diagnosis device comprises: An image preprocessing module is configured to perform image preprocessing on an original digital pathology slice image to obtain an optimized image block; An image segmentation module is configured to perform cell-tissue collaborative segmentation on the optimized image block to obtain a cell entity set and a tissue entity set; A graph learning module is configured to map the cell entity set and the tissue entity set into a heterogeneous graph structure, wherein the heterogeneous graph structure has cell graph nodes and tissue graph nodes as vertices and spatial adjacency relationships as edges; An identification module is configured to determine the Sjogren's syndrome state based on the heterogeneous graph structure; The graph learning module is further configured to regard each single cell instance as a cell graph node and establish a cell-cell edge set between the cell graph nodes that meet a preset interval requirement; regard each tissue region instance as a tissue graph node and establish a cell-tissue edge set between the cell graph nodes and the tissue graph nodes that meet a preset coverage requirement; integrate the cell-cell edge set and the cell-tissue edge set to generate a heterogeneous graph structure, wherein the nodes of the heterogeneous graph structure carry corresponding cell multi-modal feature vectors or tissue multi-modal feature vectors; The identification module is further configured to optimize the cell multi-modal feature vectors in the heterogeneous graph structure through a self-supervised graph attention network to obtain optimized cell feature vectors; the self-supervised graph attention network is obtained through self-supervised pre-training by node masking reconstruction and contrast learning with structure disturbance; perform immune cell classification according to the optimized cell feature vectors to obtain a plurality of immune cell graph nodes; and perform preset clustering analysis based on the plurality of immune cell graph nodes to output a Sjogren's syndrome state determination result; The identification module is further configured to construct an immune cell graph node subgraph based on the plurality of immune cell graph nodes, run a preset community detection algorithm on the immune cell graph node subgraph to obtain immune cell clusters, and output a Sjogren's syndrome state determination result according to spatial distribution information of the immune cell clusters and the number of immune cells in the immune cell clusters.

6. An apparatus for automatically diagnosing a pathological image of Sjogren's syndrome, characterized by, The device comprises a memory, a processor, and a pathological image Sjogren's syndrome automatic diagnosis program stored on the memory and executable on the processor, and the pathological image Sjogren's syndrome automatic diagnosis program is configured to implement the steps of the pathological image Sjogren's syndrome automatic diagnosis method according to any one of claims 1 to 4.

7. A storage medium, characterized by The storage medium stores a pathological image Sjogren's syndrome automatic diagnosis program, and the pathological image Sjogren's syndrome automatic diagnosis program implements the steps of the pathological image Sjogren's syndrome automatic diagnosis method according to any one of claims 1 to 4 when executed by the processor.

Images