Iif image processing method, system, electronic device, and storage medium

Abstract

This invention relates to an IIF image processing method, system, electronic device, and storage medium, belonging to the field of image processing technology. The method includes: performing target detection on a target image to obtain regions of interest (ROIs), each ROI being treated as a node; acquiring descriptive data and its embedded representation for each node, the descriptive data including at least a fluorescence pattern prediction category; constructing node features based on the descriptive data of each node, and constructing edge features between nodes based on the embedded representation of each node; constructing graph features using the node and edge features of the target image; and inputting the graph features of the target image into a trained graph neural network model, which outputs the final fluorescence pattern category of the target image. This invention achieves high accuracy in identifying fluorescence patterns in IIF images, with a traceable and highly interpretable decision-making process, and strong data scalability due to its graph structure.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and particularly to an indirect immunofluorescence (IIF) image processing method, system, electronic device, and storage medium. Background Technology

[0002] Immunofluorescence is a powerful technique built upon immunology, biochemistry, and microscopy. It primarily utilizes the fluorescence pattern resulting from the binding of fluorescently labeled antibodies and antigens for qualitative disease analysis. Due to its high sensitivity, specificity, and speed, it is widely used in scientific research and clinical diagnosis. Indirect immunofluorescence (IIF) is a classic technique in medical testing for autoantibody detection. Its basic principle is as follows: A solid-phase detection matrix containing a biological matrix expressing the target antigen is prepared. The sample to be tested (such as human serum) is incubated with this matrix under suitable conditions, and then a fluorescein-labeled anti-human immunoglobulin antibody (i.e., secondary antibody) is added. If the sample contains specific antibodies against the target antigen (i.e., primary antibodies), they will bind to the antigenic components on the matrix, and subsequently to the fluorescently labeled antibody, forming a specific fluorescent characteristic.

[0003] Currently, the methods for interpreting fluorescence patterns in IIF images include the following:

[0004] 1. Indirect immunofluorescence images are captured by a camera connected to a microscope and then manually interpreted by laboratory personnel.

[0005] Second, the readings are directly observed and interpreted under a microscope by experienced technicians.

[0006] The two methods described above are currently the commonly used interpretation methods for IIF fluorescence patterns. They rely on manual identification of fluorescence and morphological features, resulting in high subjectivity, insufficient standardization, and low interpretation efficiency. Furthermore, because the characteristics of atypical fluorescence patterns are not sufficiently significant, the accuracy of the detection results highly depends on the professional level and interpretation experience of the technicians. This can lead to a high false positive rate and reduce detection efficiency.

[0007] Third, traditional machine learning methods are employed, involving manual feature extraction from IIF images followed by analysis using shallow classification models. For example, an article titled "Expert Consensus on the Clinical Application of Anti-neutrophil Cytoplasmic Antibody Detection" published in the September 2018 issue of the *Chinese Journal of Medical Laboratory Science* (Vol. 41, No. 9, page 644) discloses a scheme for classifying ANCA ethanol-fixed matrix fluorescence images using manually designed features combined with algorithms such as logistic regression and decision trees. Currently, machine learning-based interpretation methods perform well in identifying typical fluorescence patterns, but struggle to accurately identify atypical fluorescence patterns with insufficient fluorescence or morphological characteristics. Summary of the Invention

[0008] In view of the technical problems existing in the prior art, the present invention proposes an IIF image processing method, system, electronic device and storage medium to improve the recognition accuracy of fluorescence patterns in IIF images.

[0009] To address the aforementioned technical problems, according to one aspect of the present invention, an IIF image processing method is provided, comprising:

[0010] Target detection is performed on the target image to obtain the region of interest in the target image, where each region of interest is a node and each node corresponds to a cell region;

[0011] Obtain the description data and its embedded representation for each node, wherein the description data includes at least fluorescence pattern prediction data;

[0012] Node features are constructed based on the description data of each node, and edge features between nodes are constructed based on the embedded representation of each node.

[0013] Construct graph features using the node and edge features of the target image; and

[0014] The graph features of the target image are input into a trained graph neural network model, and the graph neural network model outputs the final fluorescence pattern category of the target image.

[0015] The graph features include a node feature matrix and an edge feature matrix. Correspondingly, the steps of constructing graph features using the node features and edge features of the target image include:

[0016] Construct a node feature matrix based on the node features and number of nodes in the target image. , Where m is the number of nodes, n is the dimension of node features, and the node feature matrix is... Node feature matrix elements Let k represent the feature of the k-th node, where k = 1, 2, ..., m. ;

[0017] Construct an edge feature matrix based on the number of nodes and the edge features between nodes. Construct the edge feature matrix The steps include:

[0018] The adjacency matrix A of the target image is generated based on the embedded representation distance between every two nodes; where, The row index order of the adjacency matrix A is the same as the column index order, and the same row index and column index represent the same node in the target image;

[0019] A degree matrix D is constructed based on the adjacency matrix, where each diagonal element of the degree matrix D represents the connection between a node in the corresponding row and the remaining nodes in the same row; and

[0020] The edge feature matrix is ​​obtained by summing the adjacency matrix A and the degree matrix D. , .

[0021] Optionally, when performing target detection on the target image, the target image is input to the trained target detection model, and the target detection model outputs the regions of interest detected in the target image and the first fluorescence pattern prediction data of each region of interest; correspondingly, the description data of the node includes the first fluorescence pattern prediction data, which includes one of a variety of preset fluorescence pattern categories and their confidence levels.

[0022] Optionally, the step of obtaining the description data and embedded representation of each node further includes: inputting each region of interest into the trained first model to obtain the second fluorescence mode prediction data and embedded representation of each region of interest, wherein the second fluorescence mode prediction data includes the probabilities of multiple preset fluorescence mode categories; correspondingly, the description data of each node includes the second fluorescence mode prediction data and the first fluorescence mode prediction data.

[0023] Optionally, the step of constructing node features based on the descriptive data of each node includes: fusing the second fluorescence mode prediction data and the first fluorescence mode category data of each region of interest to construct node features.

[0024] Optionally, the matrix element in the i-th row and j-th column of the adjacency matrix A Calculated using the following formula:

[0025] ,

[0026] in, and These represent the embedded representations of the node in the i-th row and the node in the j-th column, respectively.

[0027] diagonal elements Calculated using the following formula:

[0028] ,

[0029] The distance threshold for the embedded representation of a node; t is the indicator function; t is the column number.

[0030] Optionally, the number of nodes m is a preset number of nodes, further including:

[0031] Count the number h of regions of interest detected from the target image;

[0032] Determine whether the number of regions of interest (h) counted is less than the preset number of nodes (m);

[0033] In response to the fact that the number of regions of interest h is less than the preset number of nodes m, an additional dimension is added to the node features as a padding code. The padding code in the node features corresponding to each region of interest is set to the first value; the padding code in the node features of the remaining (mh)th node is set to the second value.

[0034] In response to the fact that the number of regions of interest (h) detected is greater than or equal to the preset number of nodes (m), the preset number of regions of interest (m) are randomly selected from the detected regions of interest or in descending order of confidence level as nodes.

[0035] Optionally, the graph neural network model includes at least a multi-level graph structure convolutional layer and a classification layer connected in sequence. The multiple graph structure convolutional layers sequentially perform convolution calculations on the input graph features, and the classification layer aggregates and transforms the convolution calculation results passed from the graph structure convolutional layers to obtain the final fluorescence mode category of the target image.

[0036] According to another aspect of the present invention, the present invention also provides an IIF image processing system, comprising:

[0037] The target detection module is configured to perform target detection on the target image to obtain the region of interest in the target image, wherein each region of interest is a node, and each node corresponds to a cell region;

[0038] The node data acquisition module is configured to acquire description data and its embedded representation for each node, wherein the description data includes at least the fluorescence pattern prediction category;

[0039] The graph feature construction module is configured to construct node features based on the description data of each node, and to construct edge features between nodes based on the embedded representation of each node; it then constructs graph features using the node features and edge features of the target image; wherein, the graph features include a node feature matrix and an edge feature matrix, and the graph feature construction module constructs the node feature matrix based on the node features of the target image and the number of nodes. , Where m is the number of nodes, n is the dimension of node features, and the node feature matrix is... Node feature matrix elements Let k represent the feature of the k-th node, where k = 1, 2, ..., m. ;

[0040] The graph feature construction module constructs an edge feature matrix based on the number of nodes and the edge features between nodes. Construct the edge feature matrix The steps include:

[0041] The adjacency matrix A of the target image is generated based on the embedded representation distance between every two nodes; where, The row index order of the adjacency matrix A is the same as the column index order, and the same row index and column index represent the same node in the target image;

[0042] A degree matrix D is constructed based on the adjacency matrix, where each diagonal element of the degree matrix D represents the connection between a node in the corresponding row and the remaining nodes in the same row; and

[0043] The edge feature matrix is ​​obtained by summing the adjacency matrix A and the degree matrix D. , ;as well as

[0044] The prediction module is configured to input the graph features of the target image into a trained graph neural network model, and the graph neural network model outputs the final fluorescence pattern category of the target image.

[0045] According to another aspect of the present invention, an electronic device is also provided, comprising a processor and a memory, wherein a set of computer program instructions is stored in the memory, and the aforementioned IIF image processing method or the aforementioned IIF image processing system is executed when the processor executes the set of computer program instructions in the memory.

[0046] According to another aspect of the present invention, the present invention also provides a computer-readable storage medium storing a computer program instruction set thereon, which, when executed by a processor, performs the aforementioned IIF image processing method or implements the aforementioned IIF image processing system.

[0047] According to another aspect of the present invention, the present invention also provides a computer program product comprising a computer program instruction set, which, when executed by a processor, performs the aforementioned IIF image processing method or implements the aforementioned IIF image processing system.

[0048] This invention abstracts cellular regions in IIF images into graph nodes, morphological and fluorescence characteristics of cells into node features, and similarity relationships between cells into edges. Through message passing and aggregation in a graph neural network, nodes with similar features (cell populations belonging to the same fluorescence pattern) will mutually reinforce each other, effectively suppressing interference from sparse and scattered abnormally stained cells. Therefore, it can identify patterns represented by cell populations with more consistent morphology and more distinct features, with high recognition accuracy. The decision-making process is traceable and highly interpretable, and the graph structure has strong data scalability. Attached Figure Description

[0049] The preferred embodiments of the present invention will now be described in further detail with reference to the accompanying drawings, wherein:

[0050] Figure 1 This is a flowchart of an IIF image processing method according to an embodiment of the present invention;

[0051] Figure 2 This is a schematic diagram of the IIF image of VCA-EBV;

[0052] Figure 3 This is a schematic diagram of IIF images of anti-neutrophil cytoplasmic antibodies (ANCA) against one or more antigens based on an ethanol-fixed matrix.

[0053] Figure 4 This is a schematic diagram of the structure of a first model according to an embodiment of the present invention;

[0054] Figure 5 This is a flowchart of a method for constructing graph features according to an embodiment of the present invention;

[0055] Figure 6 This is a schematic diagram of the structure of a graph neural network model according to an embodiment of the present invention;

[0056] Figure 7 This is a block diagram of an IIF image processing system according to an embodiment of the present invention;

[0057] Figure 8 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] In the following detailed description, reference can be made to the accompanying drawings, which form part of this application and illustrate specific embodiments of the present application. In the drawings, similar reference numerals describe substantially similar components in different figures. Specific embodiments of the present application are described in sufficient detail below to enable those skilled in the art to implement the technical solutions of the present application. It should be understood that other embodiments may also be utilized, or structural, logical, or electrical changes may be made to the embodiments of the present application.

[0060] Figure 1 This is a flowchart of an IIF image processing method according to an embodiment of the present invention, the method comprising the following steps:

[0061] Step S1: Perform target detection on the target image to obtain the region of interest in the target image, wherein each region of interest is a node, and each node corresponds to a cell region.

[0062] Step S2: Obtain the description data and embedded representation of each node, wherein the description data includes at least fluorescence pattern prediction data.

[0063] Step S3: Construct node features based on the description data of each node, and construct edge features between nodes based on the embedded representation of each node.

[0064] Step S4: Construct graph features using the node features and edge features of the target image.

[0065] Step S5: Input the graph features of the target image into the trained graph neural network model, and output the final fluorescence pattern category of the target image through the graph neural network model.

[0066] The target image in step S1 is an indirect immunofluorescence image (IIF image) of an antibody detection sample against certain antigens, such as an Epstein-Barr virus (EBV) antibody using the viral capsid antigen (VCA) as the antigen. See [link to relevant documentation] for the corresponding IIF image. Figure 2 , Figure 2This is a schematic diagram of the IIF image for VCA-EBV. Another example is anti-neutrophil cytoplasmic antibodies (ANCA) against one or more antigens. See the corresponding IIF image for details. Figure 3 , Figure 3 This is a schematic diagram of an IIF image of anti-neutrophil cytoplasmic antibodies (ANCA) against one or more antigens based on an ethanol-fixed matrix. An IIF image includes multiple cellular regions that exhibit specific fluorescence and morphological characteristics. These specific fluorescence and morphological characteristics constitute the fluorescence pattern, and different antigens and antibodies possess specific fluorescence patterns.

[0067] Before target detection in the target image, the original IIF image is preprocessed to improve the stability of image quality. The preprocessing includes, for example, cropping the image to a specified size, normalizing the pixel values ​​of the cropped image, and then storing it as the image to be processed. The target image in step S1 is the preprocessed image to be processed. The normalization process in this invention can employ any algorithm, such as Min-Max normalization, z-score normalization, logarithmic function normalization, arctangent function normalization, etc. Those skilled in the art can choose any algorithm based on application habits, and will not be elaborated further here.

[0068] In one embodiment, when performing target detection on the target image in step S1, the target image is input to the trained target detection model, and the target detection model outputs the regions of interest detected in the target image and the first fluorescence pattern prediction data of each region of interest; correspondingly, the description data of the regions of interest includes the first fluorescence pattern prediction data.

[0069] Object detection models can be based on deep learning neural network models. These neural network models can employ one-stage or two-stage detection networks. Algorithms for one-stage detection networks include YOLO (You Only LookOnce) and SSD (Single Shot MultiBox Detector), while algorithms for two-stage detection networks include Faster R-CNN (Faster Region-based Convolutional Neural Network), Mask R-CNN, and Cascade R-CNN. During the training of the target detection model, cellular regions in the sample set are manually outlined with rectangular boxes, and the fluorescence pattern category of each region is labeled according to its fluorescence characteristics. Antibodies targeting different antigens have corresponding possible fluorescence pattern categories. For example, for ANCA detection samples, the fluorescence pattern categories in their IIF images are typical cytoplasmic ANCA (tC), atypical cytoplasmic ANCA (aC), typical perinuclear ANCA (tP), atypical perinuclear ANCA (aP), and nonspecific ANCA (NS). Therefore, after the target image is input into the target detection model, the model processes the image using appropriate algorithms to detect the rectangular regions containing cellular areas, i.e., regions of interest (ROIs), and outputs corresponding descriptive data. The descriptive data includes, for example, region coordinates, fluorescence pattern category, and its confidence level. The region coordinate value consists of four values: the coordinates of the top-left corner of the region, the region width, and the height. In this embodiment, since it is necessary to distinguish between five fluorescence pattern categories, the fluorescence pattern category output by the target detection model is set as a data structure consisting of 5 binary digits. Each data bit represents a fluorescence pattern category, and the binary digits 0 and 1 represent the model's judgment result, respectively. 0 represents that it is not a fluorescence pattern category, and 1 represents that it is a fluorescence pattern category. The fluorescence pattern category confidence level is, for example, a number between 0 and 1. The larger the value, the greater the probability that the ROI belongs to that category, and the higher the confidence level. Therefore, the fluorescence pattern category data output by the target detection model includes one of the five fluorescence pattern categories and its confidence level.

[0070] Furthermore, to obtain the embedded representation of the ROI, the aforementioned object detection model can also be configured to output a high-dimensional feature representation for each ROI. For example, by adding an embedding branch to the output layer of the object detection model, the corresponding embedded representation can be obtained simultaneously with the ROI and its description data. Therefore, the ROI description data in this embodiment includes the first fluorescence mode prediction data.

[0071] In another embodiment, after obtaining the ROI and its descriptive data through the target detection model, each ROI is input into a trained first model to obtain second fluorescence mode prediction data and an embedded representation for each ROI. The second fluorescence mode prediction data includes the probabilities of multiple preset fluorescence mode categories. Then, the second fluorescence mode prediction data for each ROI is merged into the descriptive data. That is, in this embodiment, the descriptive data for each ROI includes both second fluorescence mode prediction data and first fluorescence mode prediction data.

[0072] The first model in this embodiment is, for example, a classic convolutional neural network (CNN) model, such as ResNet, ResNeXt, EfficientNet, DenseNet, MobileNet / ShuffleNet, etc., or it can be a Transformer series model, such as ViT, Swing Transformer, DeiT, PVT, etc. See also Figure 4 , Figure 4 This is a schematic diagram of the structure of a first model according to an embodiment of the present invention. The first model in this embodiment is a structural principle block diagram, comprising a backbone feature extraction network 11, a feature processing network 12, and an output network 13. The backbone feature extraction network 11 is configured to extract features from the input target image through parallel feature extraction branches at multiple scales (hereinafter referred to as scale branches) to obtain first information D1. Furthermore, when extracting features from the input target image, the backbone feature extraction network 11 performs multiple iterative feature extraction operations, gradually constructing high-level abstract semantic features from low-level local features, thereby improving the ability of the output embedded representation to express the details of the target image.

[0073] The feature processing network 12 is configured to dynamically weight the feature information at each scale and the correlation information between scales in the first information through an attention mechanism to obtain the second information D2. The feature processing network 12 can adaptively determine the attention weights of the feature information at each scale based on the calculation results through attention calculation.

[0074] Output network 13 is configured to perform prediction output operations on the second information D2 to obtain category data of the target image. Embedded means embedded.

[0075] In step S3, when constructing node features based on the description data of each node, the category data and confidence scores in the description data are fused into a vector structure of a certain length as the node features. , , where n is the dimension of the node features. For example, when the description data only includes the first fluorescence pattern prediction data, the first fluorescence pattern prediction data includes the first category One-hot encoding. and category confidence At that time, among them, , Where f is the total number of categories, the node features can be represented as , , n=f+1. When the description data includes the first category One-hot encoding. Category confidence The probability distribution of the second class in the time and second fluorescence mode prediction data hour, Node features can be represented as , , n=2f+1.

[0076] In step S3, when constructing the edge features between nodes based on the embedded representation of each node, the embedded representation distance between two regions of interest is calculated as the edge feature between the nodes. In one embodiment, the embedded representation distance between two nodes... Calculated using the following formula (1-1).

[0077] (1-1)

[0078] Among them, among them, and Let represent the embedded representation of the node in the i-th row and the embedded representation of the node in the j-th column, respectively.

[0079] The graph features in step S4 include a node feature matrix. Sum of edge feature matrices For details, see Figure 5 , Figure 5 This is a flowchart of a method for constructing graph features according to an embodiment of the present invention. The method includes:

[0080] Step S41, based on the node features of the target image And construct the node feature matrix with the number of nodes m. . Where m is the number of nodes, n is the dimension of node features, and the node feature matrix is... Node feature matrix elements Let k represent the feature of the k-th node, where k = 1, 2, ..., m. .

[0081] Step S42: Generate the adjacency matrix A of the target image based on the pairwise embedded representation distances of all nodes in the target image. Wherein, The row index order of the adjacency matrix A is the same as the column index order, and the same row index and column index represent the same node in the target image.

[0082] Step S43: Construct a degree matrix D based on the adjacency matrix. The degree matrix D reflects the distribution pattern between nodes. Each diagonal element in the degree matrix D... This represents the connection between the nodes in the corresponding row and other nodes. The specific calculation formula is shown in equation (1-2) below.

[0083] (1-2)

[0084] Where i and j are the row and column indices of the adjacency matrix A, respectively, and in the degree matrix D, the diagonal elements... The row index and column index are the same, i.e., i=j; Ⅱ(∙) is an indicator function, which is 1 when the condition is met and 0 otherwise; Let be the matrix element in the i-th row and t-th column of the adjacency matrix A, and τ be the node activation threshold. According to formula (1-2), when the cosine similarity between a node (ROI, a cell region) and another node exceeds τ, the node connection between them will be activated. After accumulation, the connection information between the corresponding node and all other nodes will be obtained.

[0085] Step S44: Calculate the sum of the adjacency matrix A and the degree matrix D to obtain the edge feature matrix. Specifically, the edge feature matrix is ​​calculated based on the following formula (1-3). .

[0086] (1-3)

[0087] Edge feature matrix Its matrix elements include the matrix elements of adjacency matrix A. Each diagonal element in the sum-degree matrix D Matrix elements Let represent the distance between the embedded representation of the node in row i and the embedded representation of the node in column j, i=1,2,……,m, j=1,2,……,m, where the row number and column number of the same node in the edge feature matrix are the same.

[0088] As shown in formula (1-1), the diagonal of the adjacency matrix A is all 0, which introduces some invalid learning information. To obtain richer edge relationships, this invention utilizes the Laplacian matrix transformation principle to calculate the degree matrix D. The value of each element in the degree matrix D reflects whether the fluorescence pattern of the cell corresponding to the current node is the dominant pattern of the entire image. Therefore, the edge features in this invention can represent the correlation between cells. Based on the correlation between all cells in an image and combined with the fluorescence pattern category corresponding to the node, the coexistence relationship of different types of cells in the image can be depicted.

[0089] In step S4, a graph feature consisting of node features and edge features is constructed. , graph features It can be represented as .

[0090] In this invention, the number of nodes *m* is a preset number of nodes. After detecting the region of interest (ROI) from the target image in step S1, the number of ROIs *h* detected from the target image is further counted. It is then determined whether the counted number of ROIs *h* is less than the preset number of nodes *m*. If the counted number of ROIs *h* is less than the preset number of nodes *m*, then when constructing node features, a dimension is added to the node features as a padding code. The padding code in the node features corresponding to each ROI is set to a first value, such as the binary number 1 or 0. The padding code in the node features of the remaining (mh)th node is set to a second value, such as the binary number 0 or 1, which is the opposite of the first value. The node features at this time can be represented as follows: . For the complement bit, .

[0091] For example, when the preset number of nodes m=50, and the number of regions of interest detected from the target image h=45, the feature of the first node is represented as:

[0092] .

[0093] Among them, the top 5 [ [The second class probability distribution is the output of the first model] This indicates that there are 5 categories, and the numbers represent the probability of the corresponding category. The middle 5 digits [...]. [The first category One-hot encoding of the first fluorescence pattern prediction data output by the target detection model] This indicates that there are 5 categories, and a binary value of 1 represents the category predicted by the object detection model. [Class confidence] That is, the confidence level of the predicted category. The last character [ ]for .

[0094] Starting from node 46, node features are represented by the following formula:

[0095] .

[0096] The -1 represents the confidence level. The last character [ ]for The other bits are 0.

[0097] Node features of each node in this embodiment The m node features of the target image constitute a node feature matrix. , .

[0098] If the number of regions of interest (h) detected is greater than or equal to the preset number of nodes (m), then m regions of interest are randomly selected as nodes from the detected regions of interest, either randomly or in descending order of confidence level.

[0099] See Figure 6 , Figure 6 This is a schematic diagram of a graph neural network model according to an embodiment of the present invention. The graph neural network model in this embodiment includes two sequentially connected graph structure convolutional layers and a classification layer. The plurality of graph structure convolutional layers sequentially process the input graph features. The classification layer performs convolution calculations and aggregates and transforms the convolution calculation results passed from the graph structure convolution layer to obtain the final fluorescence mode category of the target image.

[0100] When the map features of a target image When input is given to a graph neural network model, the computation process of the graph neural network model is as follows:

[0101] First, the first-level graph structure convolution calculation is performed according to the following formula (2-1) to obtain the first-level graph structure convolution calculation result. .

[0102] (2-1)

[0103] in, Here are the learnable parameters in this layer of the network, where This indicates the number of channels in the hidden layer. , is the node feature matrix. Let be the edge feature matrix.

[0104] The second-level graph structure convolution calculation result is obtained by performing the second-level graph structure convolution calculation as shown in equation (2-2). .

[0105] (2-2)

[0106] in, The parameters are learnable in the network. The second-level graph structure convolutional layer uses the same propagation mechanism as the first-level graph structure convolutional layer, but no longer uses an activation function to preserve the linear separability of the original feature space.

[0107] Next, in the classification layer, the node features are aggregated into a graph-level representation using formula (2-3). .

[0108] (2-3)

[0109] These are the learnable parameters in the network. The index represents the subgraph, and f is the number of fluorescence pattern categories.

[0110] Finally, the graph-level representation is transformed using the Softmax operation shown in formula (2-4). The fluorescence pattern is converted into a probability distribution, and the category corresponding to the position index of the maximum probability value is determined as the final fluorescence pattern category.

[0111] (2-4)

[0112] The following uses an ANCA-IIF image of an ethanol matrix as an example to illustrate the training process of the graph neural network model in this embodiment.

[0113] First, a training sample set is constructed. For example, an initial sample set is obtained, which is a set of ANCA-IIF images on an ethanol matrix. Then, the dominant fluorescence mode of each ANCA-IIF image is labeled and used as the category label for the ANCA-IIF image.

[0114] Each ANCA-IIF image is then input into the target detection model to obtain cell regions within the ANCA-IIF images, and to obtain fluorescence pattern category data and confidence scores for these cell regions. The sample set includes multiple subsets of cell region images, each subset consisting of a cell region image from one ANCA-IIF image. The fluorescence pattern category data and confidence scores for each cell region image are used as initial annotation information (label). For example, the initial annotation information data structure is set to 6 fields. The first 5 fields correspond to 5 fluorescence pattern categories, the field corresponding to the category determined by the target detection model is set to 1, the remaining fields are set to 0, and the last field is the confidence score when determining the category. For example, the initial annotation information label for a cell region image... i =[1,0,0,0,0,0.87].

[0115] Then, the image of each cell region in the sample set is output to the neural network model to obtain the corresponding second fluorescence mode prediction data and embedded representation. The second fluorescence mode prediction data includes the probability of each preset fluorescence mode category. The data structure of the second fluorescence mode prediction data is set to 5 fields, which correspond to the 5 fluorescence mode categories respectively. The data of each field is the probability of the corresponding category. For example, a second fluorescence mode prediction data is represented as [0.83, 0.05, 0.07, 0.05, 0].

[0116] Then, the second fluorescence pattern prediction data of the same cell region image were merged into the initial annotation information. label In the middle, add a two's complement bit to obtain the first annotation information. label1 For example, the first annotation information of the image of the i-th cell region. label1 i =[0.83,0.05,0.07,0.05,0,1,0,0,0,0,0.87,0].

[0117] The embedded representation of the cell region image is used as the second annotation information. Label2 i .

[0118] It is understandable that the first annotation information in the training samples label1 The structure is related to the application model. For example, when only one object detection model is used to obtain descriptive data including category data and embedded representations of ROIs, the first annotation information... label1 It only includes the corresponding category data and confidence level.

[0119] The aforementioned processing yielded the first annotation information for each cell region image. label1 Second annotation information label2.

[0120] First annotation information for each cell region image in a subset of cell region images from an ANCA-IIF image. label1 and as the second annotation information label2 Construct the node feature matrix V and the edge feature matrix Thus, the graphical features of an ANCA-IIF image were obtained. The sample uses the category labels of the ANCA-IIF image as the graph features. The sample's category label. This involves modeling each cell region image as a node, using the first annotation information (label1) of the cell region image as the node feature. All cell region image node features Construct a node feature matrix V based on the second annotation information of all cell region images. label2Construct the edge feature matrix according to the aforementioned formulas (1-1) to (1-3).

[0121] The graph neural network model is then trained and evaluated until it meets the requirements.

[0122] This invention abstracts cellular regions in an image as graph nodes, cell morphology and fluorescence characteristics as node features, and the similarity relationships between cells as edges. Through learning from sample data and using message passing and aggregation via a graph neural network, nodes with similar features (cell populations belonging to the same fluorescence pattern) reinforce each other. The model can automatically identify and amplify signals from the largest and most coherent cell populations based on graph topology (such as node degree and subgraph density) or attention weights, effectively suppressing interference from sparse and scattered abnormally stained cells. Through learning from samples, this embodiment can identify patterns represented by cell populations with more consistent morphology and more distinct features. For example, for ANCA-IIF images on an ethanol matrix, although there are only a few typical cytoplasmic fluorescence patterns (tC) in the IIF image, and a large number of atypical cytoplasmic fluorescence patterns (aC) also exist, this invention, through the transmission of edge relationships via a graph neural network model, can still infer that the former is more consistent and representative of the pattern, thus correctly outputting the typical cytoplasmic fluorescence pattern (tC) as the dominant category.

[0123] Because the graph features of this invention include the features of cellular regions in each IIF image and the relationships between these regions, the model's decision-making process is traceable. The graph features in this invention visually demonstrate the contribution of nodes (cells) and node relationships to the final classification, i.e., node importance and edge importance. This provides doctors with a basis for classification and offers high interpretability.

[0124] Furthermore, the aforementioned expressions of node features and edge features are merely examples. As needed, prior information other than images, such as cell size, fluorescence intensity statistics, and patient clinical indicators (at the system level), can be fused into node features or edge features to form multimodal inputs, thereby further improving the comprehensiveness and accuracy of decision-making.

[0125] In another aspect, the present invention also provides an IIF image processing system, see [link to relevant documentation]. Figure 7 , Figure 7This is a block diagram of an IIF image processing system according to an embodiment of the present invention. The IIF image processing system includes a target detection module 100, a node data acquisition module 200, a graph feature construction module 300, and a prediction module 400. The target detection module 100 is configured to perform target detection on a target image to obtain regions of interest (ROIs) in the target image, where each ROI is a node, and each node corresponds to a cell region. The node data acquisition module 200 acquires descriptive data and its embedded representation for each node, the descriptive data including at least a fluorescence pattern prediction category. The graph feature construction module 300 constructs node features based on the descriptive data of each node, and constructs edge features between nodes based on the embedded representation of each node; it then constructs graph features using the node features and edge features of the target image. The prediction module 400 inputs the graph features of the target image into a trained graph neural network model, and the graph neural network model outputs the final fluorescence pattern category of the target image.

[0126] In one embodiment, when the target detection module 100 performs target detection on a target image, it inputs the target image into a trained target detection model, which outputs regions of interest (ROIs) detected in the target image and first fluorescence pattern prediction data for each ROI. The node data acquisition module 200 inputs each ROI into a trained first model to obtain second fluorescence pattern prediction data and an embedded representation for each ROI. The second fluorescence pattern prediction data includes the probability of each preset fluorescence pattern category. The description data for each node includes both the second fluorescence pattern prediction data and the first fluorescence pattern prediction data.

[0127] The graph feature construction module 300 fuses the second fluorescence mode prediction data and the first fluorescence mode category data of each region of interest to construct node features. The graph feature construction module 300 calculates the embedded representation distance between two regions of interest as the edge feature between nodes. The graph features include a node feature matrix and an edge feature matrix; the specific construction process is described in the aforementioned method section and will not be repeated here. The graph neural network model and the final fluorescence mode category of the output target image are described in the aforementioned method section and will not be repeated here.

[0128] Figure 8 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. The electronic device can be implemented as a server or other various terminal devices, such as desktop personal computers, tablet computers, laptop computers, etc., including a processor 601 and a memory 602. The memory 602 stores a program instruction set, and when the processor 601 executes the program instruction set in the memory 602, it implements any of the aforementioned IIF image processing methods or systems.

[0129] Specifically, the processor 601 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of the present invention.

[0130] Memory 602 may include mass storage for data or instructions. For example, and not limitingly, memory 602 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 602 may include removable or non-removable (or fixed) media. Where appropriate, memory 602 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 602 is non-volatile solid-state memory.

[0131] The memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the IIF image processing method or system provided by this invention.

[0132] In one example, the electronic device may also include a communication interface 603 and a bus 610. The processor 601, memory 602, and communication interface 603 are connected via the bus 610 and communicate with each other.

[0133] The communication interface 603 is mainly used to realize communication between various modules, systems, units and / or devices in the embodiments of the present invention.

[0134] Bus 610 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 610 may include one or more buses. While specific buses are described and illustrated in embodiments of the invention, the invention contemplates any suitable bus or interconnect.

[0135] The present invention also provides a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement any of the IIF image processing methods or systems described in the foregoing embodiments. The computer-readable storage medium can be any medium that can tangibly contain or store computer-executable instructions for use by or in conjunction with an instruction execution system, apparatus, or device. The storage medium can be a transient computer-readable storage medium or a non-transitory computer-readable storage medium. Non-transitory computer-readable storage media may include, but are not limited to, magnetic storage devices, optical storage devices, and / or semiconductor storage devices. Examples of such storage devices include, for example, magnetic disks, optical discs based on CD, DVD, or Blu-ray technology, and persistent solid-state storage such as flash memory and solid-state drives.

[0136] The present invention also provides a computer program product, comprising a set of computer program instructions, which, when executed by a processor, implement any of the IIF image processing methods or systems described in the foregoing embodiments. The computer program product includes, but is not limited to, application installation packages published on websites or in app stores, application plugins, and mini-programs that can run within certain applications.

[0137] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0138] The above embodiments are for illustrative purposes only and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the scope of the invention. Therefore, all equivalent technical solutions should also fall within the scope of the invention.

Claims

1. A method for processing IIF images, characterized in that, include: Target detection is performed on the target image to obtain the region of interest in the target image, where each region of interest is a node and each node corresponds to a cell region; Obtain the description data and its embedded representation for each node, wherein the description data includes at least fluorescence pattern prediction data; Node features are constructed based on the description data of each node, and edge features between nodes are constructed based on the embedded representation of each node. Construct graph features using the node and edge features of the target image; and The graph features of the target image are input into a trained graph neural network model, and the graph neural network model outputs the final fluorescence pattern category of the target image. The graph features include a node feature matrix and an edge feature matrix. Correspondingly, the steps of constructing graph features using the node features and edge features of the target image include: Construct a node feature matrix based on the node features and number of nodes in the target image. , Where m is the number of nodes, n is the dimension of node features, and the node feature matrix is... Node feature matrix elements Let k represent the feature of the k-th node, where k = 1, 2, ..., m. ; Construct an edge feature matrix based on the number of nodes and the edge features between nodes. Construct the edge feature matrix The steps include: The adjacency matrix A of the target image is generated based on the embedded representation distance between every two nodes; where, The row index order of the adjacency matrix A is the same as the column index order, and the same row index and column index represent the same node in the target image; A degree matrix D is constructed based on the adjacency matrix, where each diagonal element of the degree matrix D represents the connection between a node in the corresponding row and the remaining nodes in the same row; and The edge feature matrix is ​​obtained by summing the adjacency matrix A and the degree matrix D. , .

2. The IIF image processing method according to claim 1, characterized in that, When performing target detection on a target image, the target image is input to a trained target detection model, and the target detection model outputs the regions of interest detected in the target image and the first fluorescence pattern prediction data for each region of interest; correspondingly, the description data of the node includes the first fluorescence pattern prediction data, which includes one of a variety of preset fluorescence pattern categories and its confidence level.

3. The IIF image processing method according to claim 2, characterized in that, The step of obtaining the description data and embedded representation of each node further includes: inputting each region of interest into the trained first model to obtain the second fluorescence mode prediction data and embedded representation of each region of interest, wherein the second fluorescence mode prediction data includes the probabilities of multiple preset fluorescence mode categories; correspondingly, the description data of each node includes the second fluorescence mode prediction data and the first fluorescence mode prediction data.

4. The IIF image processing method according to claim 3, characterized in that, The steps for constructing node features based on the descriptive data of each node include: fusing the second fluorescence mode prediction data and the first fluorescence mode category data of each region of interest to construct node features.

5. The IIF image processing method according to claim 1, characterized in that, The matrix element in the i-th row and j-th column of the adjacency matrix A Calculated using the following formula: ， in, and These represent the embedded representations of the node in the i-th row and the node in the j-th column, respectively. diagonal elements Calculated using the following formula: ， The distance threshold for the embedded representation of a node; t is the indicator function; t is the column number.

6. The IIF image processing method according to claim 1, characterized in that, The number of nodes m is a preset number of nodes, further including: Count the number h of regions of interest detected from the target image; Determine whether the number of regions of interest (h) counted is less than the preset number of nodes (m); In response to the fact that the number of regions of interest h is less than the preset number of nodes m, an additional dimension is added to the node features as a padding code. The padding code in the node features corresponding to each region of interest is set to the first value; the padding code in the node features of the remaining (mh)th node is set to the second value. In response to the fact that the number of regions of interest (h) detected is greater than or equal to the preset number of nodes (m), the preset number of regions of interest (m) are randomly selected from the detected regions of interest or in descending order of confidence level as nodes.

7. The IIF image processing method according to claim 1, characterized in that, The graph neural network model includes at least a multi-level graph structure convolutional layer and a classification layer connected in sequence. The multiple graph structure convolutional layers sequentially perform convolution calculations on the input graph features. The classification layer aggregates and transforms the convolution calculation results passed from the graph structure convolutional layers to obtain the final fluorescence mode category of the target image.

8. An IIF image processing system, characterized in that, include: The target detection module is configured to perform target detection on the target image to obtain the region of interest in the target image, wherein each region of interest is a node, and each node corresponds to a cell region; The node data acquisition module is configured to acquire description data and its embedded representation for each node, wherein the description data includes at least the fluorescence pattern prediction category; The graph feature construction module is configured to construct node features based on the description data of each node, and to construct edge features between nodes based on the embedded representation of each node. It then constructs graph features using the node and edge features of the target image. These graph features include a node feature matrix and an edge feature matrix. The graph feature construction module constructs the node feature matrix based on the node features of the target image and the number of nodes. , Where m is the number of nodes, n is the dimension of node features, and the node feature matrix is... Node feature matrix elements Let k represent the feature of the k-th node, where k = 1, 2, ..., m. ; The graph feature construction module constructs an edge feature matrix based on the number of nodes and the edge features between nodes. Construct the edge feature matrix The steps include: The adjacency matrix A of the target image is generated based on the embedded representation distance between every two nodes; where, The row index order of the adjacency matrix A is the same as the column index order, and the same row index and column index represent the same node in the target image; A degree matrix D is constructed based on the adjacency matrix, where each diagonal element of the degree matrix D represents the connection between a node in the corresponding row and the remaining nodes in the same row; and The edge feature matrix is ​​obtained by summing the adjacency matrix A and the degree matrix D. , ;as well as The prediction module is configured to input the graph features of the target image into a trained graph neural network model, and the graph neural network model outputs the final fluorescence pattern category of the target image.

9. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a set of computer program instructions, which, when executed by the processor, implement the IIF image processing method of any one of claims 1-7 or the IIF image processing system of claim 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a set of computer program instructions, which, when executed by a processor, implement the IIF image processing method of any one of claims 1-7 or the IIF image processing system of claim 8.

11. A computer program product, characterized in that, It includes a computer program instruction set, which, when executed by a processor, implements the IIF image processing method of any one of claims 1-7 or the IIF image processing system of claim 8.

Images