Immunofluorescence image classification method and device, equipment, storage medium

Abstract

The application belongs to the technical field of medical image processing, and discloses an immunofluorescence image classification method and device, equipment and a storage medium. Through application of a hierarchical self-attention mechanism, the attention feature map is repeatedly subjected to downsampling and feature extraction operations. With each downsampling, the width and height of the feature map are continuously reduced, the number of channels of the feature map is increased, and the receptive field of the network is continuously expanded. The network gradually transitions from focusing on local regions of the image to focusing on global regions of the image. Finally, the feature map after the last downsampling and feature extraction is used as the target classification image for classification input, and a classification result is obtained through category judgment. Therefore, multi-scale feature information of the immunofluorescence image can be obtained, so that the multi-resolution problem of the glomerular immunofluorescence image can be adapted, and the classification accuracy is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing, and in particular relates to an immunofluorescence image classification method and apparatus, equipment, and storage medium. Background Art

[0002] With the continuous advancement of medical imaging and computer technology, medical image analysis has become an indispensable tool and technical means in medical research, clinical disease diagnosis, and treatment. However, the complexity of medical image data poses a huge challenge to traditional machine learning algorithms for medical image analysis. In recent years, deep learning (DL) technology has become a hot application in medical image analysis, capable of automatically identifying hidden disease features in medical images. Over the past few years, the success of deep learning in computer vision tasks has primarily relied on convolutional neural networks (CNNs), largely due to the strong inductive bias of convolutional layers.

[0003] Currently, classification algorithms for renal immunofluorescence images primarily rely on traditional image processing, convolutional neural networks, or a combination of the two. Traditional image processing methods are used to preprocess immunofluorescence images, such as filtering, color channel conversion, and extracting information-dense regions based on grayscale values. Classification is then performed using a convolutional neural network, or the original image is directly fed into the convolutional neural network for classification. Commonly used convolutional neural network models include the ResNet, GoogleNet, and DenseNet models.

[0004] The aforementioned renal immunofluorescence image classification algorithms are limited in their ability to handle fluorescence images in complex backgrounds, such as those with uneven brightness, impurities, and unclear focus. Current research on renal fluorescence images has primarily focused on the detection or segmentation of glomerular regions, or simply on the classification of glomerular deposition sites. Less research has focused on the classification of glomerular immunofluorescence images of fluorescent-positive glomerular disease morphologies.

[0005] This is attributed to the significant differences between glomerular immunofluorescence images and traditional natural images. Renal fluorescence images are generally obtained by professional renal pathologists or technicians using a fluorescence microscope to photograph and scan fluorescently stained renal tissue sections. Different operators and imaging systems will bring certain differences to the images, such as some images being overexposed or too dark, with different clarity, and partially blurred images. In addition, clinically acquired images sometimes cannot guarantee the same magnification, resulting in large differences in image resolution. This makes it very difficult to accurately classify the morphology of fluorescence-positive glomerular diseases in glomerular immunofluorescence images. Summary of the Invention

[0006] The purpose of the present invention is to provide an immunofluorescence image classification method and apparatus, equipment, and storage medium that can adapt to the multi-resolution problem of glomerular immunofluorescence images and thereby improve the classification accuracy.

[0007] A first aspect of an embodiment of the present invention discloses an immunofluorescence image classification method, comprising:

[0008] Acquire immunofluorescence images to be tested;

[0009] Inputting the immunofluorescence image to be tested into a convolutional layer, and performing a convolution operation on the immunofluorescence image to be tested using the convolutional layer to obtain a mapping feature map of a specified dimension;

[0010] Inputting the mapping feature map into a regional self-attention network, and using the regional self-attention network to perform feature extraction on the mapping feature map to obtain an attention feature map;

[0011] Repeat the following steps on the attention feature map until the number of repetitions reaches a specified number: downsampling the attention feature map to obtain a downsampled feature map; inputting the downsampled feature map into the regional self-attention network, and extracting features from the downsampled feature map using the regional self-attention network to obtain a new attention feature map; and incrementing the number of repetitions by one;

[0012] When the number of repetitions reaches a specified number, the finally obtained new attention feature map is used as the target classification image;

[0013] The target classification image is classified to obtain a classification result of the immunofluorescence image to be tested.

[0014] A second aspect of an embodiment of the present invention discloses an immunofluorescence image classification device, comprising:

[0015] an acquisition unit, used for acquiring the immunofluorescence image to be tested;

[0016] A convolution unit, configured to input the immunofluorescence image to be tested into a convolution layer, and perform a convolution operation on the immunofluorescence image to be tested using the convolution layer to obtain a mapping feature map of a specified dimension;

[0017] An attention unit, configured to input the mapping feature map into a regional self-attention network, and perform feature extraction on the mapping feature map using the regional self-attention network to obtain an attention feature map;

[0018] The multi-scale unit is configured to repeatedly perform the following steps on the attention feature map until the number of repetitions reaches a specified number: downsampling the attention feature map to obtain a downsampled feature map; inputting the downsampled feature map into the regional self-attention network, and extracting features from the downsampled feature map using the regional self-attention network to obtain a new attention feature map; and incrementing the number of repetitions by one.

[0019] a determination unit, configured to use the new attention feature map finally obtained by the multi-scale unit as a target classification image when the number of repetitions reaches a specified number;

[0020] The classification unit is used to classify the target classification image to obtain a classification result of the immunofluorescence image to be tested.

[0021] A third aspect of an embodiment of the present invention discloses an electronic device, comprising a memory storing executable program code and a processor coupled to the memory; the processor calls the executable program code stored in the memory to execute the immunofluorescence image classification method disclosed in the first aspect.

[0022] A fourth aspect of an embodiment of the present invention discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program enables a computer to execute the immunofluorescence image classification method disclosed in the first aspect.

[0023] The beneficial effect of the present invention is that the provided immunofluorescence image classification method, apparatus, device, and storage medium repeatedly perform downsampling and feature extraction operations on the attention feature map by applying a hierarchical self-attention mechanism. With each downsampling, the width and height of the feature map are continuously reduced, and the number of channels of the feature map is increased. At the same time, the receptive field of the network is continuously expanded, gradually transitioning from the local area of ​​the attention image to the global area of ​​the attention image. Finally, the feature map after the last downsampling and feature extraction is used as the target classification image for classification input, and category judgment is performed to obtain the classification result. Therefore, multi-scale feature information of the immunofluorescence image can be obtained, thereby being able to adapt to the multi-resolution problem of glomerular immunofluorescence images, thereby improving the classification accuracy.

[0024] In addition, by using a local area-based self-attention network instead of directly using the global self-attention mechanism, it can ensure that the attention calculation is limited to a small area rather than the entire image, thereby reducing the amount of calculation and improving computational efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings herein illustrate specific examples of the technical solutions described in the present invention, and together with the specific implementation methods constitute a part of the specification, and are used to explain the technical solutions, principles and effects of the present invention.

[0026] Unless otherwise specified or defined, the same reference numerals in different drawings represent the same or similar technical features, and the same or similar technical features may also be represented by different reference numerals.

[0027] Figure 1 is a flow chart of an immunofluorescence image classification method of the present invention;

[0028] Figure 2 This is a schematic diagram of the network structure of an immunofluorescence image classification model of the present invention;

[0029] Figure 3 Schematic diagram of the network structure of a regional self-attention module of the present invention;

[0030] Figure 4 This is a schematic diagram of the network structure of a block merging layer of the present invention;

[0031] Figure 5 These are the six classification categories of the glomerular immunofluorescence image disclosed in the embodiment of the present invention;

[0032] Figure 6 It is a structural schematic diagram of an immunofluorescence image classification device of the present invention;

[0033] Figure 7 It is a structural schematic diagram of an electronic device of the present invention.

[0034] Description of reference numerals:

[0035] 601, acquisition unit; 602, convolution unit; 603, attention unit; 604, multi-scale unit; 605, determination unit; 606, classification unit; 701, memory; 702, processor. DETAILED DESCRIPTION

[0036] To facilitate understanding of the present invention, specific embodiments of the present invention will be described in more detail below with reference to the accompanying drawings.

[0037] Unless otherwise specified or defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art. In the context of combining the technical solution of the present invention with realistic scenarios, all technical and scientific terms used herein may also have meanings corresponding to the purpose of implementing the technical solution of the present invention. "First, second..." used herein is merely used to distinguish names and does not represent a specific quantity or order. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0038] Unless otherwise specified or defined, the “said” and “the” used in this document refer to the technical features or technical contents mentioned or described before the corresponding position, and the technical features or technical contents may be the same as or similar to the technical features or technical contents mentioned therein.

[0039] There is no doubt that technical contents or technical features that are contrary to the purpose of the present invention or are obviously contradictory should be excluded.

[0040] It should be noted that the immunofluorescence image classification method disclosed in the present invention is applicable to different types of immunofluorescence images, including but not limited to glomerular immunofluorescence images, as well as immunofluorescence images of other lesions. In the embodiments of the present invention, glomerular immunofluorescence images are used as an example for explanation, which should not be considered a limitation of the present invention. The execution subject of the immunofluorescence image classification method can be a computer device or an immunofluorescence image classification device embedded in a computer device, and the present invention is not limited to this.

[0041] Before executing this immunofluorescence image classification method, the model training and testing stages are also included:

[0042] (1) Training phase

[0043] S01. Acquire a plurality of immunofluorescence sample images, determine a plurality of target sample images from the plurality of immunofluorescence sample images, and pre-process each target sample image to obtain a plurality of pre-processed sample images.

[0044] Due to the particularity of medical problems, the incidence of different diseases varies greatly among the population, resulting in serious data imbalance in the sample image categories. To address this data imbalance, different data augmentation methods are used for each category of data, including color transformation, brightness change, and geometric transformation. Among them, brightness transformation and color transformation eliminate the effects of different exposures during scanning and differences in fluorescent staining during sectioning. Geometric transformation can, to a certain extent, reduce the differences in image resolution caused by changes in image focus scanning magnification and different camera focal lengths during image acquisition.

[0045] Specifically, each immunofluorescence sample image carries category label information annotated by a doctor. All immunofluorescence sample images are classified according to the category label information. For categories with a large number of samples, the image clarity evaluation function is applied to remove some immunofluorescence sample images with poor quality; for categories with a small number of samples, one or more data amplification methods including color transformation, brightness change and geometric transformation are applied to expand the sample size, thereby balancing the data set.

[0046] For example, for categories where the ratio of the number of samples to the total number of samples reaches a specified ratio, the immunofluorescence sample images are screened, and the immunofluorescence sample images in this category whose image clarity is less than a first specified threshold are removed as immunofluorescence images of poor quality; and for categories where the ratio of the number of samples to the total number of samples does not reach the specified ratio, a corresponding number of expanded sample images are added to the category by applying one or more data augmentation methods including color transformation, brightness change, and geometric transformation, where the numerical value of the corresponding number can be determined according to the number of samples in different categories, and the ratio of the number of samples after the corresponding number of expansions to the total number of samples should be able to reach the specified ratio, or the numerical value of the corresponding number can also be pre-set by the developer as a specified value.

[0047] In addition, in order to avoid excessive differences in the numerical distribution range of immunofluorescence sample images, after sample data expansion, the immunofluorescence sample images in all categories are further normalized with fixed mean and variance to obtain multiple target sample images.

[0048] Finally, the remaining target sample images with good quality are preprocessed and all target sample images are scaled to 384*384 to obtain preprocessed sample images. This can preserve rich detail feature information while ensuring that the network does not require complex computations.

[0049] S02. Input each preprocessed sample image into the segmentation model to obtain a mask sample image; and fuse each mask sample image with its corresponding target sample image to obtain a fused sample image of each target sample image, and divide the multiple fused sample images into a training set and a test set.

[0050] Considering that the acquired glomerular immunofluorescence images contain some other contents besides the glomeruli, including impurities and other tissues, this will also cause certain interference in the morphological recognition of the glomeruli. Therefore, a segmentation model is used to obtain a mask sample image for each target sample image, remove impurities and other tissues in the background of the target sample image, and then fuse the segmented mask sample image with the target sample image to form a fused sample image as the training set and test set sample data. This can add prior knowledge to the classification network model, allowing the classification network model to focus on the important glomerular area.

[0051] S03. Construct a classification network model including a convolutional layer and a regional self-attention network, use the fused sample images in the training set and their corresponding category label information to perform supervised deep learning training on the classification network model, and use the final trained target classification model as the immunofluorescence image classification model.

[0052] In addition, due to the privacy of medical images, the data sources of immunofluorescence sample images are generally limited to individual hospitals, and it is difficult to reach the large quantity required in conventional image classification tasks. The amount of data is usually small, especially some categories of glomerular immunofluorescence images are particularly scarce, which makes it difficult to meet the conditions for deep learning models to require large amounts of data training, and it is very easy to cause overfitting in the network.

[0053] To address this issue, the regional self-attention network in this immunofluorescence image classification model is preferably configured as a regional sparse self-attention network. Traditional self-attention mechanisms distribute attention across all content within a range, leading to a lack of focus and increasing network computational complexity. However, compared to traditional self-attention mechanisms, a sparse self-attention mechanism automatically selects the most relevant elements within a range using a top-k algorithm. This helps focus on the key elements used for classification while ignoring irrelevant information. This approach effectively retains important information and removes noise, acting as a regularization method to help the model converge better with small datasets.

[0054] Therefore, the immunofluorescence image classification model proposed in the present invention can efficiently identify samples under a limited training set and has high classification accuracy and robustness.

[0055] (2) Testing phase

[0056] S04. Use each fused sample image in the test set and its corresponding category label information to test the immunofluorescence image classification model.

[0057] The test results indicate that the immunofluorescence image classification model provided by the embodiments of the present invention can accurately classify renal immunofluorescence images. Specifically, the classification accuracy of the immunofluorescence image classification model on the test set is shown in Table 1 below. It can be seen that the classification accuracy of the immunofluorescence image classification model is higher than that of the other two existing classification models. The training and test datasets used in the above embodiments are immunofluorescence images of glomeruli. Furthermore, in other possible embodiments, the scope of application of the present invention can also be expanded to other types of immunofluorescence images.

[0058] Table 1 Comparison of the accuracy of this immunofluorescence image classification model and other existing network models

[0059] Classification network model Test set accuracy VIT 75.38％ MobileNetV2 86.65％ Embodiments of the present invention 93.22％

[0060] like Figure 1 As shown, an embodiment of the present invention discloses an immunofluorescence image classification method, which is mainly used to classify samples to be tested with unknown category labels. The classification method includes the following steps S10 to S80:

[0061] S10. Acquire the immunofluorescence image to be tested.

[0062] In some possible embodiments, the immunofluorescence image to be tested may refer to a preprocessed immunofluorescence image obtained after preprocessing the original immunofluorescence image of the sample to be tested; and in this embodiment, preferably, after preprocessing the original immunofluorescence image of the sample to be tested to obtain the preprocessed immunofluorescence image, the preprocessed immunofluorescence image is further input into the segmentation model to obtain a mask image; and the mask image and the preprocessed immunofluorescence image are fused, and the fusion result is used as the immunofluorescence image to be tested.

[0063] Among them, the preprocessing operation generally includes obtaining the original immunofluorescence image of the glomerulus of the sample to be tested, and then scaling the original immunofluorescence image of the glomerulus to a size of 384*384 to obtain a preprocessed immunofluorescence image. This can not only retain rich detailed feature information, but also ensure that the network does not have complex computational complexity. Moreover, in this embodiment, it is preferred that a segmentation model is simultaneously used to obtain a segmentation mask of the original immunofluorescence image of the glomerulus, remove impurities and other tissues in the background of the original immunofluorescence image of the glomerulus, and then fuse the segmented mask image with the original immunofluorescence image of the glomerulus as the input of the classification network model. Prior knowledge can be added to the classification network model, so that the classification network model focuses on important glomerular areas.

[0064] S20: input the immunofluorescence image to be tested into the convolution layer, and use the convolution layer to perform a convolution operation on the immunofluorescence image to be tested to obtain a mapping feature map of a specified dimension.

[0065] like Figure 2 As shown, the immunofluorescence image classification model trained in an embodiment of the present invention includes a feature mapping network, a regional self-attention network, a plurality of feature extraction networks, and an output network connected in sequence; wherein, the feature mapping network includes a convolutional layer and a linear layer connected in sequence, and each feature extraction network includes a block merging layer and a regional self-attention network connected in sequence.

[0066] Among them, the convolution layer includes 128 convolution kernels of size 4*4, and performs a convolution operation with a stride of 4 on the immunofluorescence image to be tested. The immunofluorescence image to be tested is divided into blocks to obtain a 96*96*128 feature map, and the first two dimensions are expanded through the linear layer to form a 9216*128 mapping feature map.

[0067] S30. Input the mapping feature map into the regional self-attention network, use the regional self-attention network to extract features from the mapping feature map, and obtain an attention feature map.

[0068] The regional self-attention network includes at least two regional self-attention modules connected in sequence, such as Figure 3 As shown, each regional self-attention module includes a first normalization layer, a regional self-attention layer, a first residual connection module, a second normalization layer, a multi-layer perceptron layer, and a second residual connection module, which are connected in sequence. The two input ends of the first residual connection module are respectively connected to the input end of the first normalization layer and the output end of the regional self-attention layer, the output end of the first residual connection module is connected to the input end of the second normalization layer, the two input ends of the second residual connection module are respectively connected to the input end of the second normalization layer and the output end of the multi-layer perceptron layer, and the output end of the second residual connection module is connected to the input end of the next regional self-attention module. A regional division module is also provided between the output end of the first normalization layer and the input end of the regional self-attention layer, and a regional restoration module is also provided between the output end of the regional self-attention layer and the input end of the first residual connection module. Figure 3 The symbol “+” represents the residual connection.

[0069] Based on this, the regional self-attention network is used to extract features from the mapping feature map. The specific implementation method of obtaining the attention feature map includes:

[0070] The mapping feature map is used as the input feature map and input into the first regional self-attention module. In each regional self-attention module, the input feature map is normalized by the first normalization layer (i.e., LayerNorm layer) to obtain a first feature map of the same dimension as the input feature map. On this basis, the first feature map is divided into regions by the regional division module to obtain a second feature map of the specified region size. For example, if the local region size is specified to be 12, the size of the second feature map is 64*144*128. The second feature map is used as the input of the regional self-attention layer based on the local region; the second feature map is input into the regional self-attention layer, and the regional self-attention layer is used to calculate the self-attention within the divided region to obtain a third feature map of the same dimension as the second feature map. The regional restoration module is then used to restore the third feature map to obtain a fourth feature map of 9216*128. The fourth feature map is then residually connected to the input feature map of the first normalization layer by the first residual connection module to obtain a fifth feature map. The fifth feature map is then normalized by the second normalization layer and input into the multi-layer perceptron (Multi-Layer A sixth feature map with the same dimension as the fifth feature map is obtained in the Multi-Perceptron (MLP) layer. The sixth feature map is residually connected to the fifth feature map through the second residual connection module and used as the output of the regional self-attention module. The sixth feature map is then input into the next regional self-attention module as a new input feature map, and the same operation as in the regional self-attention module is performed. Finally, the output of the last regional self-attention module is used as the final attention feature map.

[0071] In this embodiment, the regional self-attention network includes two regional self-attention modules connected in sequence. After the two regional self-attention modules perform the above operations, the final attention feature map size is 64*144*128. Among them, the regional self-attention module is preferably a regional sparse self-attention module, and the regional self-attention layer it includes is a regional sparse self-attention layer.

[0072] S40. Perform a downsampling operation on the attention feature map to obtain a downsampled feature map; and add one to the number of repetitions.

[0073] Among them, the block merging layer included in the feature extraction network is mainly used to perform downsampling operations on the attention feature map, such as Figure 4 As shown in the figure, the block merging layer specifically includes an element selection + deformation layer, a third normalization layer, and a linear layer connected in sequence. When downsampling the attention feature map, the attention feature map is first expanded into a 96*96*128 candidate feature map. The element selection + deformation layer selects elements in the first and second dimensions of the candidate feature map with an interval of 2 to obtain a feature vector. The selected feature vectors are spliced ​​together along the third dimension (also known as the channel dimension) through the third normalization layer to obtain a 48*48*512 spliced ​​vector. Then, the linear layer is used to reduce the third dimension of the spliced ​​vector to half of the original, forming a new downsampled feature map of size 48*48*256.

[0074] S50. Input the downsampled feature map into the regional self-attention network, use the regional self-attention network to extract features of the downsampled feature map, and obtain a new attention feature map.

[0075] Among them, the regional self-attention network included in the feature extraction network is mainly used to extract features from the downsampled feature map to obtain a new attention feature map. Its operation method is similar to the above-mentioned specific implementation method of using the regional self-attention network to extract features from the mapping feature map to obtain the attention feature map, that is:

[0076] The downsampled feature map is used as the input feature map and input into the first regional self-attention module. In each regional self-attention module, the input feature map is normalized by the first normalization layer to obtain the first feature map of the same dimension. Then, the first feature map is divided into regions by the regional division module to obtain the second feature map of the specified region size. The second feature map is input into the regional self-attention layer, and the regional self-attention layer is used to calculate the self-attention within the divided region to obtain the third feature map of the same dimension as the second feature map. The third feature map is then restored by the regional restoration module to obtain the fourth feature map. The difference connection module performs a residual connection on the fourth feature map and the input feature map of the first normalization layer to obtain a fifth feature map; then, the fifth feature map is normalized by the second normalization layer and input into the multi-layer perceptron layer to obtain a sixth feature map of the same dimension as the fifth feature map; the sixth feature map is residually connected to the fifth feature map by the second residual connection module as the output of the regional self-attention module, and is input into the next regional self-attention module as a new input feature map, and the same operation as in the regional self-attention module is performed; the output of the last regional self-attention module is used as the new attention feature map.

[0077] S60: Determine whether the number of repetitions reaches the specified number. If so, execute steps S70 to S80; otherwise, go to step S40.

[0078] In practical applications, the repeated execution of steps S40 to S50 can be performed repeatedly through a feature extraction network. For a clearer explanation, in this embodiment, Figure 2 3 feature extraction networks are shown in FIG. 3 . It can be understood that in this embodiment, the number of times specified is 3.

[0079] S70: Use the new attention feature map finally obtained as the target classification image.

[0080] The specified number of repetitions can be determined based on the original immunofluorescence image. For example, a pre-stored number of repetitions corresponding to the sizes of multiple immunofluorescence images can be used. After obtaining the immunofluorescence image to be tested, the corresponding specified number of repetitions can be further determined based on the size of the immunofluorescence image to be tested. The specified number of repetitions is positively correlated with the size of the immunofluorescence image to be tested. In other words, a larger size increases the number of repetitions that can be performed, and a greater number of feature extraction networks can be configured.

[0081] In addition, the number of sequentially connected regional self-attention modules included in the regional self-attention network is generally set to 2. There is no limit on this number. If the task is simple, the number can be smaller.

[0082] The number of repetitions can specifically be the number of downsampling times. First, the number of repetitions is set to 0. After each downsampling and feature extraction, the number of repetitions is incremented by one. It is then determined whether the number of repetitions is equal to or greater than a specified number (e.g., 3, 4, or 5). If so, the new attention feature map obtained at this time is used as the target classification image. Generally, when the number of repetitions is equal to the specified number, the latest attention feature map is output as the target classification image.

[0083] S80: classify the target classification image to obtain a classification result of the immunofluorescence image to be tested.

[0084] After the feature extraction network outputs the target classification image, the target classification image is input into the last network layer in the immunofluorescence image classification model, that is, the output network. The output network includes a fourth normalization layer (LayerNorm layer), a global average pooling layer, and a linear layer connected in sequence. Specifically, the target classification image is input into the fourth normalization layer for normalization, and then input into the global average pooling layer. The global average pooling layer is used to obtain a global output, and finally input into the linear layer to obtain the final classification result of the immunofluorescence image. Among them, the classification result includes the probability value of the immunofluorescence image to be tested belonging to each category. Therefore, the target category with the largest probability value can be determined as the final classification category of the immunofluorescence image to be tested.

[0085] Taking glomeruli as an example, the immunofluorescence image classification model in the embodiment of the present invention can identify 6 classification categories of glomerular immunofluorescence images, as shown in the attached figure. Figure 5 As shown, including idiopathic membranous nephropathy, membranoproliferative glomerulopathy, IgA nephropathy, anti-glomerular basement membrane disease, post-strand nephritis and other kidney diseases (including diabetic nephropathy and lupus nephritis, etc.), it can assist doctors in clinical diagnosis more effectively and accurately.

[0086] In summary, the implementation of the present invention, for the morphological recognition problem of glomerular immunofluorescence images, can focus on both the global information of the image and the local detail information in the image, and adopt a hierarchical sparse self-attention mechanism based on local regions. The model focuses on the local region of the image at the shallow level and focuses on some detail features. Subsequently, multiple downsampling operations are adopted. As the network layer deepens, the receptive field of the model continues to expand, gradually transitioning from focusing on the local region of the image to the global region. Therefore, it is possible to obtain multi-scale feature information of the image, thereby adapting to the multi-resolution problem of glomerular immunofluorescence images and improving the classification accuracy. Moreover, by using a self-attention network based on local regions instead of directly using a global self-attention mechanism, it is possible to ensure that the attention calculation is limited to a small region rather than the entire image, thereby reducing the amount of calculation and improving the computational efficiency. In the implementation of the present invention, the attention mechanism adopts a sparse self-attention mechanism, which can ensure the effectiveness of a large model on small data. Therefore, it can efficiently recognize samples under a limited training set and has high classification accuracy and robustness.

[0087] like Figure 6 As shown, the embodiment of the present invention discloses an immunofluorescence image classification device, including an acquisition unit 601, a convolution unit 602, an attention unit 603, a multi-scale unit 604, a determination unit 605 and a classification unit 606, wherein:

[0088] An acquisition unit 601 is used to acquire an immunofluorescence image to be tested;

[0089] The convolution unit 602 is used to input the immunofluorescence image to be tested into the convolution layer, and use the convolution layer to perform a convolution operation on the immunofluorescence image to be tested to obtain a mapping feature map of a specified dimension;

[0090] An attention unit 603 is used to input the mapping feature map into the regional self-attention network, and use the regional self-attention network to extract features from the mapping feature map to obtain an attention feature map;

[0091] The multi-scale unit 604 is configured to repeatedly perform the following steps on the attention feature map until the number of repetitions reaches a specified number: downsampling the attention feature map to obtain a downsampled feature map; inputting the downsampled feature map into the regional self-attention network, and extracting features from the downsampled feature map using the regional self-attention network to obtain a new attention feature map; and incrementing the number of repetitions by one.

[0092] A determination unit 605 is configured to use the new attention feature map finally obtained by the multi-scale unit as the target classification image when the number of repetitions reaches a specified number;

[0093] The classification unit 606 is used to classify the target classification image to obtain a classification result of the immunofluorescence image to be tested.

[0094] like Figure 7 As shown, an embodiment of the present invention discloses an electronic device, including a memory 701 storing executable program code and a processor 702 coupled to the memory 701;

[0095] The processor 702 calls the executable program code stored in the memory 701 to execute the immunofluorescence image classification method described in the above embodiments.

[0096] An embodiment of the present invention further discloses a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the immunofluorescence image classification method described in the above embodiments.

[0097] The purpose of the above embodiments is to exemplify and deduce the technical solution of the present invention, and to fully describe the technical solution, purpose and effect of the present invention. Its purpose is to enable the public to have a more thorough and comprehensive understanding of the disclosed content of the present invention, and it does not limit the scope of protection of the present invention.

[0098] The above embodiments are not exhaustive and may include many other embodiments not listed above. Any replacements and improvements made without violating the concept of the present invention are within the scope of protection of the present invention.

Claims

1. An immunofluorescence image classification method, characterized in that: include: Acquire immunofluorescence images to be tested; Inputting the immunofluorescence image to be tested into a convolutional layer, and performing a convolution operation on the immunofluorescence image to be tested using the convolutional layer to obtain a mapping feature map of a specified dimension; Inputting the mapping feature map into a regional self-attention network, and using the regional self-attention network to perform feature extraction on the mapping feature map to obtain an attention feature map; Repeat the following steps on the attention feature map until the number of repetitions reaches a specified number: downsampling the attention feature map to obtain a downsampled feature map; inputting the downsampled feature map into the regional self-attention network, and extracting features from the downsampled feature map using the regional self-attention network to obtain a new attention feature map; and incrementing the number of repetitions by one; When the number of repetitions reaches a specified number, the finally obtained new attention feature map is used as the target classification image; Classifying the target classification image to obtain a classification result of the immunofluorescence image to be tested; The regional self-attention network includes at least two regional self-attention modules connected in sequence, and each of the regional self-attention modules includes a first normalization layer, a first residual connection module, a second normalization layer, a multi-layer perceptron layer, and a second residual connection module connected in sequence, wherein the two input ends of the first residual connection module are respectively connected to the input end of the first normalization layer and the output end of the regional self-attention layer, and the two input ends of the second residual connection module are respectively connected to the input end of the second normalization layer and the output end of the multi-layer perceptron layer.

2. The immunofluorescence image classification method according to claim 1, wherein: The step of obtaining the immunofluorescence image to be tested comprises: preprocessing the original immunofluorescence image to obtain a preprocessed immunofluorescence image; The preprocessed immunofluorescence image is input into the segmentation model to obtain a mask image; The mask image is fused with the pre-processed immunofluorescence image to obtain an immunofluorescence image to be tested.

3. The immunofluorescence image classification method according to claim 2, wherein: After fusing the mask image with the pre-processed immunofluorescence image to obtain the immunofluorescence image to be tested, the method further includes: According to the size of the immunofluorescence image to be tested, a corresponding designated number is determined; wherein the designated number is positively correlated with the size of the immunofluorescence image to be tested.

4. The immunofluorescence image classification method according to claim 1, wherein: A region division module is further provided between the output end of the first normalization layer and the input end of the region self-attention layer, and a region restoration module is further provided between the output end of the region self-attention layer and the input end of the first residual connection module.

5. The immunofluorescence image classification method according to claim 4, wherein: Inputting the downsampled feature map into the regional self-attention network, and using the regional self-attention network to extract features from the downsampled feature map to obtain a new attention feature map, including: The downsampled feature map is used as the input feature map and input into the first region self-attention module; In each of the regional self-attention modules, the input feature map is normalized by the first normalization layer to obtain a first feature map of the same dimension; Performing region division on the first feature map using the region division module to obtain a second feature map of a specified region size; Inputting the second feature map into the regional self-attention layer, and using the regional self-attention layer to calculate self-attention within the divided regions to obtain a third feature map of the same dimension as the second feature map; Performing regional restoration on the third feature map using the regional restoration module to obtain a fourth feature map; Performing a residual connection on the fourth feature map and the input feature map input by the first normalization layer through the first residual connection module to obtain a fifth feature map; Normalizing the fifth feature map through the second normalization layer and then inputting the normalized value into the multi-layer perceptron layer to obtain a sixth feature map having the same dimension as the fifth feature map; Performing a residual connection on the sixth feature map and the fifth feature map through the second residual connection module as the output of the regional self-attention module, and inputting it into the next regional self-attention module as a new input feature map, performing the same operation as in the regional self-attention module; The output of the last region self-attention module is used as the new attention feature map.

6. The immunofluorescence image classification method according to claim 5, wherein: Performing a downsampling operation on the attention feature map to obtain a downsampled feature map, comprising: Expand the attention feature map into The candidate feature map of ; Selecting elements at an interval of 2 on the first dimension and the second dimension of the to-be-selected feature map, respectively, to obtain a feature vector; The feature vectors are spliced ​​together according to the channel dimension to obtain The concatenation vector of Reduce the channel dimension of the concatenated vector to half of its original size, forming a vector of size The downsampled feature map.

7. An immunofluorescence image classification device, characterized in that include: an acquisition unit, used for acquiring the immunofluorescence image to be tested; A convolution unit, configured to input the immunofluorescence image to be tested into a convolution layer, and perform a convolution operation on the immunofluorescence image to be tested using the convolution layer to obtain a mapping feature map of a specified dimension; An attention unit, configured to input the mapping feature map into a regional self-attention network, and perform feature extraction on the mapping feature map using the regional self-attention network to obtain an attention feature map; a multi-scale unit, configured to repeatedly perform the following steps on the attention feature map until a specified number of repetitions are reached: downsampling the attention feature map to obtain a downsampled feature map; inputting the downsampled feature map into the regional self-attention network, and extracting features from the downsampled feature map using the regional self-attention network to obtain a new attention feature map; Increase the number of repetitions by one; a determination unit, configured to use the new attention feature map finally obtained by the multi-scale unit as a target classification image when the number of repetitions reaches a specified number; A classification unit, configured to classify the target classification image to obtain a classification result of the immunofluorescence image to be tested; The regional self-attention network includes at least two regional self-attention modules connected in sequence, and each of the regional self-attention modules includes a first normalization layer, a first residual connection module, a second normalization layer, a multi-layer perceptron layer, and a second residual connection module connected in sequence, wherein the two input ends of the first residual connection module are respectively connected to the input end of the first normalization layer and the output end of the regional self-attention layer, and the two input ends of the second residual connection module are respectively connected to the input end of the second normalization layer and the output end of the multi-layer perceptron layer.

8. An electronic device, characterized in that It comprises a memory storing executable program code and a processor coupled to the memory; the processor calls the executable program code stored in the memory to execute the immunofluorescence image classification method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program enables a computer to execute the immunofluorescence image classification method according to any one of claims 1 to 6.

Images