A method for constructing a glomerular image generation model and its application

Abstract

The present invention discloses a construction method and application of a glomerular image generation model, belonging to the technical field of intelligent assisted image reading. In the method, an encoding module splits an initial feature map into multiple groups of feature maps according to channels, performs pyramid convolution of different scales on each group of feature maps, and then merges them. Based on a residual mechanism, the merged feature map is fused with the initial feature map to obtain a fused feature map. The splitting, merging and fusion operations are repeatedly performed on the fused feature map of the previous level until the number of repetitions reaches a preset number. The overall correlation information in the fused feature map obtained last time is learned based on a self-attention mechanism. Through the above means, it is ensured that the field of view of the convolution can capture the global image and focus on the local image, which is beneficial to the generation of the texture morphological features of the glomerulus itself, and can also ensure the fusion of the newly generated glomerular area and the surrounding background, thereby generating an image that is closer to the real glomerular image, and the generated glomerular image has high quality.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent assisted image reading, and more specifically, relates to a method for constructing a glomerular image generation model and its application. Background Art

[0002] The pathological characteristics of glomeruli are one of the key indicators of renal function. In recent years, with the advancement of computers and digitization, more and more pathologists are electronically scanning stained renal pathology slides into digital slides to facilitate the extraction of glomerular pathological features. Simultaneously, with the increasing application of deep learning technology in image recognition, a growing number of AI-assisted detection methods or systems are emerging for tasks such as detection, identification, and classification of glomerular regions. These methods assist pathologists in completing repetitive, simple tasks, thereby reducing their workload and freeing them to focus on complex tasks such as difficult and complicated cases, thereby improving their work efficiency.

[0003] However, deep learning methods require a large number of well-labeled glomerular images to train the model, and the labeling of medical images is highly professional and costly. In addition, the acquisition of medical data involves privacy issues and is not easy to obtain. The number of glomeruli with specific characteristics is even scarcer. Therefore, the application of this technology is limited.

[0004] To address these issues, existing technologies use traditional data augmentation methods or traditional GAN ​​networks to generate similar glomerular images based on collected real glomerular images to expand the glomerular images. However, traditional data augmentation methods only perform geometric operations or brightness and contrast changes on the image itself. Traditional GAN ​​network-based methods, due to their simple structure, fail to deeply explore the essential features of glomerular images and the semantic relationship between glomeruli and background, and the generated images are also highly random. Consequently, these methods fail to focus on both local and global features of the image, resulting in low-quality glomerular shapes in the generated glomerular images that do not fully conform to those of real samples, resulting in low accuracy for AI-assisted detection. Summary of the Invention

[0005] In response to the above-mentioned defects or improvement needs of the prior art, the present invention provides a method for constructing a glomerular image generation model and its application, so as to solve the technical problem of low quality of glomerular images generated by the prior art.

[0006] To achieve the above objectives, in a first aspect, the present invention provides a method for constructing a glomerular image generation model, comprising:

[0007] The training sample set is input into the MSPSA-CGAN model for training to obtain a glomerular image generation model; wherein the training sample set includes a glomerular image X containing a glomerular region ori and the corresponding glomerular region mask Figure X mask ;

[0008] The MSPSA-CGAN model includes: a preprocessing module, a generator, and a discriminator for determining whether the image generated by the generator is a real glomerulus image;

[0009] The preprocessing module is used to transform the input glomerular image X ori Perform a dot multiplication operation with the corresponding inverse mask map to obtain the vacant map, and then compare the vacant map with the corresponding mask Figure X mask After adding, we get the contour background image; the inverse mask image is the mask Figure X mask The result after negation;

[0010] The generator includes:

[0011] The encoding module is used to extract features from the contour background image to obtain an initial feature map of the contour background image; split the initial feature map into multiple groups of feature maps by channel, perform pyramid convolution on each group of feature maps at different scales, and then merge them. The merged feature map is fused with the initial feature map based on the residual mechanism to obtain a fused feature map; the splitting, merging, and fusion operations are repeated on the fused feature map of the previous level until the number of repetitions reaches a preset number; and the overall correlation information in the fused feature map obtained in the last step is learned based on the self-attention mechanism to obtain a feature encoding map of the contour background image;

[0012] The decoding module is used to perform upsampling operations on the feature coding map to generate the same image as the glomerulus image X ori The image of the same size is obtained as the new glomerulus image X rec .

[0013] Further preferably, in the following method, the training sample set is input into the MSPSA-CGAN model for training, and the generator parameters and the discriminator parameters in the MSPSA-CGAN model are alternately updated by minimizing the overall image loss to obtain a glomerular image generation model;

[0014] or,

[0015] In the second approach, the training sample set is input into the MSPSA-CGAN model for training. By minimizing the overall image loss and the local image loss at the same time, the generator parameters and the discriminator parameters in the MSPSA-CGAN model are updated alternately to obtain the glomerular image generation model.

[0016] Among them, the overall image loss is X ori With X rec The difference loss between them includes: ori With X rec The overall adversarial loss of distribution difference is used to measure X ori With X rec The overall pixel loss of pixel differences and the metric X ori With X rec Overall feature loss for feature variance;

[0017] The local loss of the image is X ori With X rec The differential loss of the mid-glomerular region fraction, including the R ori With R rec Local structure loss of structural difference, used to measure R ori With R rec The local pixel loss of pixel difference and the metric R ori With R rec Local adversarial loss for distribution difference; R ori For X ori The glomerular region in the kidney, specifically X ori With X mask The result after dot product; R rec For X rec The glomerular region in the kidney, specifically X rec With X mask The result of the dot product.

[0018] Further preferably, in the method 1, the above overall confrontation loss is:

[0019]

[0020] The overall pixel loss is:

[0021]

[0022] The overall feature loss includes: perceptual loss and style loss;

[0023] Where E[·] represents the mathematical expectation; D(X rec ) is the pair X output by the decision maker rec The result of judging whether the different positions in the glomerulus are the real glomerular regions; D(X ori ) is the pair X output by the decision maker ori The result of judging whether the different positions are the real glomerular areas.

[0024] Further preferably, in the second mode, the discriminator includes a first judgment module and a second judgment module;

[0025] The overall adversarial loss is:

[0026]

[0027] The overall pixel loss mentioned above is:

[0028]

[0029] The above overall feature loss includes: perceptual loss and style loss;

[0030] The above local structural loss is:

[0031] L ssim =1-SSIM(R ori ,R rec )

[0032] The above local pixel loss is:

[0033]

[0034] The above local adversarial loss is:

[0035]

[0036] Where E[·] represents the mathematical expectation; D1(X rec ) is the pair X output by the first decision module rec The result of judging whether the different positions in the image are the real glomerular regions; D1(X ori ) is the pair X output by the first decision module ori The result of judging whether different positions are the real glomerular regions; SSIM(·) is the structure measurement index function; D2(R rec ) is the pair R output by the second decision module rec The result of judging whether the different positions in the glomerulus are the real glomerular regions; D2(R ori ) is the pair R output by the second decision module ori The result of judging whether the different positions are the real glomerular areas.

[0037] Further preferably, the convolution module includes multiple cascaded convolution layers.

[0038] In a second aspect, the present invention provides a method for generating a glomerular image, comprising:

[0039] A glomerular image without glomerular regions is used as a reference image, and a mask image with the same size as the reference image is randomly selected; the reference image and the mask image are input into the glomerular image generation model to obtain the corresponding new image containing the glomerular region;

[0040] or,

[0041] A glomerular image without glomerular regions is used as a reference image, and a mask image with the same size as the reference image is randomly selected. A new mask image similar to the reference image mask is generated using a GAN algorithm. The reference image and the new mask image are input into the glomerular image generation model to obtain a corresponding new image containing the glomerular region.

[0042] The glomerular image generation model is constructed using the method for constructing the glomerular image generation model provided in the first aspect of the present invention.

[0043] In a third aspect, the present invention provides a method for constructing a glomerular detection model, characterized by comprising:

[0044] Performing sample enhancement on the collected glomerular images in the training set, generating new glomerular images and adding them to the training set to expand the samples in the training set; inputting the expanded training set into the target detection model for training to obtain a glomerular detection model;

[0045] The training set includes glomerular images with position and type labels; the method for sample enhancement of glomerular images includes the method for generating glomerular images provided in the second aspect of the present invention.

[0046] Correspondingly, the present invention also provides a glomerulus detection method, comprising:

[0047] Input the glomerulus image to be detected into the glomerulus detection model to obtain the area and type of the glomerulus;

[0048] The glomerular detection model is constructed using the above-mentioned method for constructing the glomerular detection model.

[0049] Further preferably, the method for sample enhancement of glomerular images further comprises: rotating and / or flipping the glomerular images in the training set, modifying their position labels accordingly, and then adding them to the training set.

[0050] Further preferably, the above-mentioned target detection model is a diffusion target detection model.

[0051] In a fourth aspect, the present invention provides a glomerular detection system, comprising:

[0052] A model building module is configured to perform sample enhancement on the collected glomerular images in the training set, generate new glomerular images, and then add them to the training set to expand the samples in the training set; input the expanded training set into the target detection model for training to obtain a glomerular detection model; wherein the training set includes glomerular images with location and type labels; and the method for performing sample enhancement on the glomerular images includes the method for generating glomerular images provided in the second aspect of the present invention;

[0053] The glomerulus detection module is used to execute the glomerulus detection method provided in the third aspect of the present invention to obtain the area and type of the glomerulus.

[0054] Further preferably, the above-mentioned glomerulus detection system also includes: a glomerulus evaluation module, which is used to crop the original glomerular WSI image, divide it into small pictures, and after preprocessing, input them into the glomerular detection module for detection, so as to obtain the detection frame and type for identifying the glomerular area in each small picture; restore each small picture according to the original position, correct the overlapping detection frames, count the total number of glomeruli in the WSI image, and calculate the ratio of the number of glomeruli of the sclerosis type to the total number of glomeruli to obtain the sclerosis ratio; wherein the type of glomerulus includes sclerosis type and non-sclerosis type.

[0055] Further preferably, the glomerulus assessment module crops the original glomerulus WSI image using an overlapping cropping method.

[0056] Further preferably, the glomerulus assessment module uses the NMS algorithm to correct the overlapping detection frames.

[0057] Further preferably, the glomerulus evaluation module corrects the overlapping detection frames in the following manner: only the detection frames whose intersection-over-union ratio with the detection frame with the highest confidence score is lower than a first preset threshold and whose center point distance is greater than a second preset threshold are retained.

[0058] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

[0059] 1. The present invention provides a method for constructing a glomerular image generation model, wherein an encoding module splits an initial feature map into multiple groups of feature maps by channel, performs pyramid convolution of different scales on each group of feature maps, and then merges them, and fuses the merged feature map with the initial feature map based on a residual mechanism to obtain a fused feature map; the splitting, merging, and fusion operations are repeatedly performed on the fused feature map of the previous level until the number of repetitions reaches a preset number; and the overall correlation information in the fused feature map obtained last time is learned based on a self-attention mechanism to obtain a feature encoding map of a contour background map; through the above means, it is ensured that the field of view of the convolution can capture the global image and focus on the local image, which is beneficial to the generation of the glomerular texture morphological features itself, and at the same time, it can also ensure the fusion of the newly generated glomerular area and the surrounding background, thereby generating an image that is closer to the real glomerular image, and the generated glomerular image has high quality.

[0060] 2. Furthermore, the method for constructing the glomerular image generation model provided by the present invention alternately updates the generator parameters and the discriminator parameters in the MSPSA-CGAN model by minimizing the overall image loss and the local image loss at the same time; wherein the overall image loss is X ori With X rec The loss between the two, the local loss of the image is X ori With X rec During the training process, we focus on both the overall features of the image and the local features of the glomerular region in the image, which enables the generator to generate images that are closer to the real glomerulus, but at the same time ensures the fusion of the newly generated glomerular region and the surrounding background without any sense of separation.

[0061] 3. The present invention provides a method for generating a glomerular image, which uses a GAN algorithm to generate a new mask image similar to the mask image of a reference image, and inputs the reference image and the new mask image into a glomerular image generation model to obtain a corresponding new image containing a glomerular region; compared with the method of directly inputting the reference image and its mask image into the glomerular image generation model, this method can generate more diverse new mask images with different contour shapes, and can generate glomerular images that are more different from the reference image while retaining the glomerular features of the image.

[0062] 4. The present invention provides a method for constructing a glomerular detection model. After the training set of the glomerular detection model is expanded using the glomerular image generation method provided by the present invention, the glomerular detection model is trained. This can solve the problem of low detection accuracy caused by the small number of glomerular samples or unbalanced categories. The present invention expands the training set by generating high-quality glomerular images, thereby greatly improving the accuracy of the glomerular detection model and achieving high detection accuracy.

[0063] 5. The present invention provides a glomerulus detection system. On the premise of accurately detecting glomeruli based on the glomerular detection method, it can also process large-scale images such as original glomerular WSI images, crop the original glomerular WSI images, divide them into small images, and input them into the glomerular detection module for detection, thereby further improving the detection efficiency.

[0064] 6. Furthermore, the glomerulus detection system provided by the present invention, after completing the detection of each small image of the glomerular WSI image, further restores each small image to its original position, and corrects the overlapping detection frames by retaining the detection frames whose intersection-over-union ratio with the detection frame with the highest confidence score is lower than a first preset threshold and whose center point distance is greater than a second preset threshold, thereby avoiding the problem of overlapping cropping of the same glomerulus and further improving the accuracy of detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 Schematic diagram of samples and mask labels in the training sample set provided by Example 1 of the present invention;

[0066] Figure 2 A schematic diagram of the structure of a generator provided in Example 1 of the present invention;

[0067] Figure 3 A schematic diagram of the structure of a multi-scale pyramid convolution unit provided in Example 1 of the present invention;

[0068] Figure 4 A schematic diagram of the structure of the self-attention module provided in Example 1 of the present invention;

[0069] Figure 5 A schematic diagram of training an MSPSA-CGAN model according to an optional implementation method 2 provided in Example 1 of the present invention;

[0070] Figure 6 Flowchart of the method for generating a glomerular image provided in Example 2 of the present invention;

[0071] Figure 7 A flow chart of a method for generating a larger glomerular image provided in Example 2 of the present invention;

[0072] Figure 8 This is a schematic diagram of the glomerular detection model provided in Example 3 of the present invention;

[0073] Figure 9 This is a schematic diagram of an image of a glomerulus image in a training set after being flipped and rotated, as provided in Example 3 of the present invention;

[0074] Figure 10 This is a target merging framework diagram provided in Example 4 of the present invention;

[0075] Figure 11 This is a schematic diagram of the final detection results provided in Example 4 of the present invention. DETAILED DESCRIPTION

[0076] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0077] Example 1

[0078] A method for constructing a glomerulus image generation model, comprising:

[0079] The training sample set is input into the MSPSA-CGAN model for training to obtain the glomerular image generation model; among them, Figure 1 As shown, the training sample set includes glomerular images X containing glomerular regions. ori and the corresponding glomerular region mask Figure X mask ;

[0080] The MSPSA-CGAN model includes: a preprocessing module, a generator, and a discriminator for determining whether the image generated by the generator is a real glomerulus image;

[0081] The preprocessing module is used to transform the input glomerular image X ori Perform a dot multiplication operation with the corresponding inverse mask map to obtain the vacant map, and then compare the vacant map with the corresponding mask Figure X mask After adding, we get the contour background image; the inverse mask image is the mask Figure X mask The result after negation;

[0082] Through the preprocessing module, the glomerular image X ori Multiply the corresponding inverse mask image pixel by pixel, and compare the multiplied vacancy image with the mask Figure X mask After merging in the same dimension, they are input into the generator. Specifically, the outline background image is: X input =X ori ⊙(1-X mask )+X mask ; Among them, ⊙ represents the dot product operation.

[0083] The generator includes:

[0084] The encoding module is used to extract features from the contour background image to obtain an initial feature map of the contour background image; split the initial feature map into multiple groups of feature maps by channel, perform pyramid convolution of different scales on each group of feature maps, and merge them, and fuse the merged feature map with the initial feature map based on the residual mechanism to obtain a fused feature map; and repeatedly perform the splitting, merging, and fusion operations on the fused feature map of the previous level until the number of repetitions reaches a preset number; and learn the overall correlation information in the fused feature map obtained in the last time based on the self-attention mechanism to obtain a feature encoding map of the contour background image; in this embodiment, the above-mentioned preset number of times is 7;

[0085] Specifically, it includes a convolution module (the convolution module includes multiple cascaded convolution layers), a pyramid convolution module and a self-attention module; the convolution module is used to extract features from the contour background image to obtain the initial feature map of the contour background image; the pyramid convolution module includes multiple cascaded multi-scale pyramid convolution units; the multi-scale pyramid convolution unit is used to split the input feature map P into multiple groups of feature maps according to the channel, perform pyramid convolution of different scales on each group of feature maps, and merge them, and fuse the merged feature map with the feature map P based on the residual mechanism to obtain a fused feature map; wherein, the input of the first-level multi-scale pyramid convolution unit is the initial feature map, and the input of the remaining multi-scale pyramid convolution units is the fused feature map output by the previous multi-scale pyramid convolution unit; the self-attention module is used to obtain the feature coding map of the contour background image based on the overall correlation information in the fused feature map output by the last-level multi-scale pyramid convolution unit;

[0086] The decoding module includes multiple cascaded upsampling convolutional layers for upsampling the feature encoding map to generate the same image as the glomerulus image X ori The image of the same size is obtained as the new glomerulus image X rec .

[0087] Specifically, a specific implementation of the generator is as follows Figure 2 As shown, in this embodiment, the generator includes an encoding module and a decoding module. First, it passes through a three-layer convolutional network (convolutional module). The channels after convolution of the first layer of convolutional network are 64, the second layer is 128, and the third layer is 256. The convolutional module is followed by a pyramid convolution module and a convolutional layer. Among them, the pyramid convolution module includes multiple cascaded multi-scale pyramid convolution units based on the residual mechanism, and there are seven of them. Figure 3As shown in the figure, the multi-scale pyramid convolution unit splits a 256-channel feature map into four 64-channel feature maps. These four feature maps are then subjected to pyramid convolutions of scales 1, 2, 4, and 8, respectively. The four convolved feature maps are then merged and, using a residual connection mechanism, are connected to the input feature map of the unit. The final output channel count remains 256. This mechanism ensures that the convolutional field of view captures both the global and local aspects, facilitating the generation of glomerular texture and morphological features while ensuring integration with the surrounding background. Figure 3 Where x1 represents the input of the multi-scale pyramid convolution module; x2 represents the merged feature map; g is the result of convolution and activation of the input x1 of the multi-scale pyramid convolution module (the result after the input x1 passes through the convolution layer and activation layer in sequence).

[0088] Then connect a self-attention module, such as Figure 4 As shown in the figure, the main body of the self-attention module is to use 1*1 convolution to perform correlation operations to ultimately obtain the self-attention feature matrix. The input of the self-attention module is the output of the convolution module x3, and the output is out. The specific formula is: out = gamma * out + x3. Where gamma is the weight value of the self-attention feature matrix, which is initialized to 0. As training progresses, the weight will increase, and the network can learn the overall correlation information of the image.

[0089] The decoding module is a two-layer (can also be 3 or 4 layers, just for illustration) upsampling convolution layer, and the final output is the glomerular image X ori Generate images of the same scale.

[0090] Regarding the specific training process of the glomerular image generation model, two optional implementation methods are given here:

[0091] In an alternative embodiment:

[0092] The training sample set is input into the MSPSA-CGAN model for training. By minimizing the overall image loss, the generator parameters and discriminator parameters in the MSPSA-CGAN model are alternately updated to obtain a glomerular image generation model.

[0093] Among them, the overall image loss is the overall image loss is X ori With X rec The difference loss between them includes: ori With X rec The overall adversarial loss of distribution difference is used to measure X ori With X rec The overall pixel loss of pixel differences and the metric X ori With X recOverall feature loss for feature variance.

[0094] Preferably, the above overall adversarial loss is:

[0095]

[0096] Where, E[·] represents the mathematical expectation; D(X rec 0 is the output of the decision maker for X rec The result of judging whether the different positions in the glomerulus are the real glomerular regions; D(X ori ) is the pair X output by the decision maker ori The result of judging whether the different positions in the image are the real glomerular regions. Specifically, after the convolution kernel of the discriminator convolves the image, rec or X ori The probability of whether the area corresponding to the convolution kernel is the glomerular area.

[0097] The overall pixel loss, that is, the L1 loss is:

[0098]

[0099] Among them, ||·||1 is the L1 norm.

[0100] The overall feature loss includes: perceptual loss and style loss;

[0101] The perceptual loss is:

[0102] L per =||φ(X ori )-φ(X rec )||1

[0103] Among them, φ(X ori ) is X ori Image features; φ(X rec ) is X rec image features.

[0104] The style loss is:

[0105] L sty =||φ(X ori ) T φ(X ori )-φ(X rec ) T φ(X rec )||1

[0106] In order to more accurately calculate the perceptual loss and style loss, the present invention uses a multi-layer feature extraction module (including multiple cascaded feature extraction units) to extract image features. In this case, the perceptual loss is:

[0107]

[0108] The above style loss is:

[0109]

[0110] Among them, φ i (X ori ) is X ori The i-th layer image feature; φ i (X rec ) is X rec The i-th layer image features; N i is the number of layers of feature extraction units in the multi-layer feature extraction module; E[·] represents the mathematical expectation. The multi-layer feature extraction module can be a CNN model, including: AlexNet model, ResNet model, GoogLeNet model, VGG model, such as VGG16 (N i 16), VGG19(N i 19) etc.

[0111] Furthermore, the overall image loss is the weighted sum of the overall adversarial loss, overall pixel loss, perceptual loss, and style loss, specifically:

[0112]

[0113] Among them, λ adv is the weight value of the loss against the loss, which is 0.4 in this embodiment; λ1 is the weight value of the overall pixel loss, which is 50 in this embodiment; per is the weight value of the perceptual loss, which is 0.1 in this implementation; sty is the weight value of style loss, which is 125 in this embodiment.

[0114] In an optional implementation mode 2:

[0115] like Figure 5 As shown, the training sample set is input into the MSPSA-CGAN model for training. By minimizing the overall image loss and the local image loss at the same time, the generator parameters and the discriminator parameters in the MSPSA-CGAN model are updated alternately to obtain the glomerular image generation model.

[0116] Among them, the overall image loss is the overall image loss is X ori With X rec The difference loss between them includes: ori With X rec The overall adversarial loss of distribution difference is used to measure X ori With Xrec The overall pixel loss of pixel differences and the metric X ori With X rec Overall feature loss for feature variance;

[0117] The local loss of the image is X ori With X rec The differential loss of the mid-glomerular region fraction, including the R ori With R rec Local structure loss of structural difference, used to measure R ori With R rec The local pixel loss of pixel difference and the metric R ori With R rec Local adversarial loss for distribution difference; R ori For X ori The glomerular region in the kidney, specifically X ori With X mask The result after dot product; R rec For X rec The glomerular region in the kidney, specifically X rec With X mask The result of the dot product.

[0118] Specifically, the above discriminator includes a first decision module (whose input is X rec and X ori ) and the second decision module (whose input is R rec and R ori );

[0119] The overall adversarial loss is:

[0120]

[0121] Where E[·] represents the mathematical expectation; R ori For X ori The glomerular region in the kidney, specifically X ori With X mask The result after dot product; R rec For X rec The glomerular region in the kidney, specifically X rec With X mask The result after dot product; D1(X rec ) is the pair X output by the first decision module rec The result of judging whether the different positions in the image are the real glomerular regions; D1(X ori ) is the pair X output by the first decision module ori The result of judging whether the different positions in the image are the real glomerular regions. Specifically, after the convolution kernel of the first discriminant module convolves the image, it recor X ori The probability of whether the area corresponding to the convolution kernel is the glomerular area.

[0122] The overall pixel loss mentioned above is:

[0123]

[0124] The above-mentioned overall feature loss includes: perceptual loss and style loss; the specific implementation method is the same as the above-mentioned optional implementation method 1, which will not be repeated here.

[0125] The above local structural loss is:

[0126] L ssim =1-SSIM(R ori ,R rec )

[0127] Among them, SSIM(·) is the structural measurement index function.

[0128] The above local pixel loss is:

[0129]

[0130] The above local adversarial loss is:

[0131]

[0132] Among them, D2(R rec ) is the pair R output by the second decision module rec The result of judging whether the different positions in the glomerulus are the real glomerular regions; D2(R ori ) is the pair R output by the second decision module ori The result of judging whether the different positions in the image are the real glomerular regions. Specifically, after the convolution kernel of the second discrimination module convolves the image, the R rec or R ori The probability of whether the area corresponding to the convolution kernel is the glomerular area.

[0133] Furthermore, the local image loss is the sum of local structure loss, local pixel loss, and local adversarial loss, specifically:

[0134]

[0135] It should be noted that this implementation introduces the overall image loss and the local image loss, and combines the two to obtain the total loss. For each part, there are two parts: reconstruction loss and adversarial loss. Specifically, the total loss is:

[0136] L TOTAL =L image +λLglomerular

[0137] Here, λ is a preset hyperparameter, which is set to 10 in this embodiment.

[0138] By minimizing both the overall image loss and the local image loss, the generator can generate glomeruli that are closer to reality, while also ensuring better integration with the surrounding background without any sense of fragmentation, resulting in higher-quality generated glomerular images.

[0139] The present invention uses a mask to remove the content at the corresponding position in the image, and uses the proposed MSPSA-CGAN to generate and repair the image with the removed content. The removed part is the generated glomerulus, and the glomerular outline is well integrated with the background.

[0140] Example 2

[0141] A method for generating a glomerular image, comprising:

[0142] Method 1:

[0143] A glomerular image without glomerular regions is used as a reference image, and a mask image with the same size as the reference image is randomly selected; the reference image and the mask image are input into the glomerular image generation model to obtain the corresponding new image containing the glomerular region;

[0144] Method 2:

[0145] like Figure 6 As shown in the figure, a glomerular image without glomerular regions is used as a reference image, and a mask image with the same size as the reference image is randomly selected; a new mask image similar to the mask image of the reference image is generated using the GAN algorithm; the reference image and the new mask image are input into the glomerular image generation model to obtain the corresponding new image containing the glomerular region;

[0146] The randomly selected mask image contains areas with pixel values ​​of 1 and areas with pixel values ​​of 0; the areas with pixel values ​​of 1 correspond to the glomerular regions to be filled. The glomerular image generation model is constructed using the method for constructing a glomerular image generation model provided in Example 1 of the present invention. The related technical solutions are the same as those in Example 1 and are not further described here.

[0147] Specifically, in method 2, based on the existing annotated masks, the GAN algorithm is used to generate some new masks with different contour shapes; in addition, the contours in the mask can be scaled so that the generated glomeruli can be of different sizes. The mask and the cropped image are multiplied to obtain an image with vacancies. The image with vacancies is input into the glomerular image generation model to obtain an image with the vacancies repaired and perfected. The repaired part is the generated glomerulus, which is well integrated with the surrounding background.

[0148] It should be noted that if Figure 7 As shown, for larger glomerular images, to improve computational efficiency, the image block I containing the glomerular region in the glomerular image can be first cut out as a reference image, and a glomerular region mask of the reference image can be obtained; then, the above-mentioned glomerular image generation method is used to generate a new image containing the glomerular region, and the newly generated image containing the glomerular region is placed back at the location of I in the original glomerular image, thereby obtaining a new glomerular image. Specifically, in an optional embodiment, the glomerular region in the larger glomerular image can be marked to obtain a marked frame; all images containing only sclerotic glomeruli are cropped from the rectangular frame. The specific cropping method in this embodiment is to crop a 256*256 square image based on the geometric center of the rectangular frame; and the cropped glomerular contours are annotated at the pixel level using Photoshop software, with values ​​within the contour set to 0 and values ​​outside the contour set to 1, to produce a binary mask.

[0149] Example 3

[0150] A method for constructing a glomerular detection model, such as Figure 8 Shown, including:

[0151] Performing sample enhancement on the collected glomerular images in the training set, generating new glomerular images and adding them to the training set to expand the samples in the training set; inputting the expanded training set into the target detection model for training to obtain a glomerular detection model;

[0152] The training set includes glomerular images with position and type labels. In this embodiment, when labeling the position labels, labelimg annotation software is used to select all glomerular regions with rectangular boxes, annotate the categories, and save the data to generate an XML file. The method for sample enhancement of glomerular images includes the glomerular image generation method provided in Example 2 of the present invention. The glomerular image is used as a reference image in the glomerular image generation method provided in Example 2 of the present invention, and a glomerular region mask map is obtained as input to the glomerular image generation method provided in Example 2.

[0153] The relevant technical solution is the same as that in Example 2 and will not be described in detail here.

[0154] Specifically, in an optional implementation, the target detection model needs to find the location of the target and classify the target, so the model is divided into a location loss function and a category loss.

[0155] Position loss function:

[0156] z p ={t x ,t y ,t w ,t h}, represents the center position coordinates of the predicted target rectangle and the length and width of the rectangle, z0 represents the center position coordinates of the target's real rectangle and the length and width of the rectangle, and the loss is the L2 norm value of the two, reflecting the difference between the two positions.

[0157] Classification loss:

[0158] y p Represents the predicted category of the target, y represents the true category of the target, and the cross entropy of the two is used as the category loss, which represents the difference between the predicted analogy and the true category.

[0159] It should be noted that the above-mentioned target detection model can be a Yolo series target detection model, a Faster-RCNN target detection model, a diffusion target detection model, or the DiffusionDET model. The DiffusionDET model is preferably used in this embodiment. Using DiffusionDET, during the training phase, target boxes are diffused from position labels (GT boxes) to a random distribution. The model learns to reverse this noise process, eliminating the need for predefined anchors. During inference, the model progressively refines a set of randomly generated boxes into output results. The key feature of DiffusionDET is that it is trained once for all inferences. Once the model is trained, it can be used to change the number of boxes and sample steps in inference, achieving better accuracy by using more boxes and more refinement steps. This detection model does not require any prior knowledge, simplifies target candidates, and achieves higher detection speed and accuracy. Therefore, the present invention can deploy a single DiffusionDET in multiple scenarios and achieve the desired speed-accuracy trade-off without retraining the network. Parameters can be adjusted as needed to meet the requirements of faster glomerular detection speed or higher accuracy.

[0160] In an optional embodiment, all the expanded images and annotation files are organized into a data set in VOC format to facilitate subsequent training to call a mature data input interface, and the training set and test set are divided into a ratio of 7:3. A total of 1,197 images are annotated, and a total of 2,946 small glomeruli are annotated, and are divided into two categories, such as sclerosis and non-sclerosis. The outlines of all glomeruli are segmented using Photoshop software and converted into binary images as labels. All sclerotic glomeruli are cut out using the coordinates and categories in the annotated xml file. The specific cutting method under this embodiment is to cut out a square image of 256*256 scale based on the geometric center of the rectangular frame. The number of sclerotic glomeruli is 766, which is used as the generated input data set and added to the training set.

[0161] In the process of expanding the training set, in addition to expanding the samples in the training set, the number of samples of different types can also be expanded more evenly based on needs (such as generating samples with less data in a certain category), thereby further improving the accuracy of detection.

[0162] Preferably, the method for sample enhancement of glomerular images further comprises: rotating and / or flipping the glomerular images in the training set, modifying their position labels accordingly, and then adding them to the training set to expand the samples in the training set. Specifically, Figure 9 As shown, in one optional implementation, the glomerular images in the training set are subjected to traditional data augmentation methods such as flipping and rotation. Based on the original position labels, the corresponding position labels are altered according to the corresponding image transformation operations, so that the position labels match the positions of the glomeruli in the enhanced images. Specifically, the training set is rotated by 45° and 75°, respectively; and the training set is flipped horizontally, vertically, and diagonally.

[0163] Correspondingly, this embodiment also provides a glomerulus detection method, comprising:

[0164] The glomerulus image to be detected is input into the above glomerulus detection model to obtain the region and type of the glomerulus.

[0165] Preferably, before using the glomerulus detection method to detect the glomerular image, it may be further determined whether the glomerular image to be detected is an image containing a glomerular region based on a classification model, so as to filter out images that do not contain a glomerular region.

[0166] Specifically, before classification, a classification model can be trained using a collected classification dataset, wherein the classification dataset includes images containing glomerular regions and images not containing glomerular regions. The classification model can be a CNN model, including a DNN model, GoogleNet, AlexNet classification network, etc.

[0167] Furthermore, when the above-mentioned glomerular detection method is used to process the glomerular WSI image, in order to improve the computational efficiency, the original glomerular WSI image can be cropped and divided into small images. After preprocessing (such as color normalization), each image is input into the glomerular detection module for detection to obtain the detection box and type for identifying the glomerular area in each small image.

[0168] Furthermore, the glomeruli in the WSI image can be counted. Specifically, each sub-image is restored to its original position, all detected sub-images are restored to their original positions, and after correcting the overlapping detection frames, the total number of glomeruli in the WSI image is counted, and the ratio of the number of glomeruli of the sclerosis type to the total number of glomeruli is calculated to obtain the sclerosis ratio; wherein the type of glomeruli includes sclerosis type and non-sclerosis type.

[0169] Specifically, in one optional embodiment, the NMS algorithm is used to correct overlapping detection frames. In another optional embodiment, overlapping detection frames are corrected by retaining only those detection frames whose intersection over union (IOU) with the detection frame with the highest confidence score is lower than a first preset threshold and whose center point distance is greater than a second preset threshold. In this embodiment, the first preset threshold is 0.45, and the second preset threshold is 0.65.

[0170] Example 4

[0171] A glomerular detection system, comprising:

[0172] A model building module is used to perform sample enhancement on the collected glomerular images in the training set, generate new glomerular images and add them to the training set to expand the samples in the training set; input the expanded training set into the target detection model for training to obtain a glomerular detection model; wherein the training set includes glomerular images with position and type labels; the method for sample enhancement of glomerular images includes the method for generating glomerular images provided in Example 2 of the present invention; the relevant technical solution is the same as Example 2 and will not be repeated here.

[0173] The glomerulus detection module is used to execute the glomerulus detection method provided in Example 3 of the present invention to obtain the area and type of the glomerulus; the relevant technical solution is the same as that in Example 3 and will not be described in detail here.

[0174] Preferably, the above-mentioned glomerulus detection system also includes: a glomerulus evaluation module, which is used to crop the original glomerular WSI image, divide it into small pictures, and after preprocessing, input them into the glomerular detection module for detection to obtain the detection frame and type for identifying the glomerular area in each small picture; restore each small picture according to the original position, correct the overlapping detection frames, count the total number of glomeruli in the WSI image, and calculate the ratio of the number of glomeruli of the sclerosis type to the total number of glomeruli to obtain the sclerosis ratio; wherein the type of glomerulus includes sclerosis type and non-sclerosis type.

[0175] Preferably, the glomerulus assessment module crops the original glomerulus WSI image using an overlapping cropping method. In an optional embodiment, the cropped image size is 1500*1500, and the cropping step size is 1200.

[0176] Preferably, the preprocessing method includes color normalization. Specifically, in an optional embodiment, a Vahadane staining normalization method, a Macehko staining normalization method, a Reinhard staining normalization method, etc. can be used to select a reference staining so that all images are matched to this standard to achieve the purpose of normalization.

[0177] Specifically, in an optional implementation, the NMS algorithm is used to correct the overlapping detection frames.

[0178] However, when performing NMS operations, such as Figure 10 As shown, there will be a problem of overlapping cropping of the same glomerulus. A glomerulus will appear in at most four cropped small images. Since some glomeruli are cropped into incomplete parts and distributed in different small images, when performing conventional NMS operations, the IOU (intersection-over-union) value between one detection box and another detection box is lower than the threshold, so it is retained. However, it is actually the same glomerulus. In order to avoid this situation, the present invention introduces the concept of frame center distance. Since there will be no overlapping pairs between glomeruli, when the IOU value is lower than the threshold, but the center distance is far, the detection box also needs to be removed. Specifically, under another optional embodiment, the overlapping detection frames are corrected in the following way: only the detection frames whose intersection-over-union (IOU) with the detection frame with the highest confidence score is lower than the first preset threshold and whose center point distance is greater than the second preset threshold are retained. The first preset threshold in this embodiment is 0.45, and the second preset threshold is 0.65. The final detection result is as follows. Figure 11 shown.

[0179] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for constructing a glomerular image generation model, characterized in that: include: The training sample set is input into the MSPSA-CGAN model for training to obtain a glomerular image generation model; wherein the training sample set includes a glomerular image X containing a glomerular region ori And the corresponding glomerular area mask map X mask ; The MSPSA-CGAN model includes: a preprocessing module, a generator, and a discriminator for determining whether the image generated by the generator is a real glomerulus image; The preprocessing module is used to transform the input glomerular image X ori Perform a dot multiplication operation with the corresponding inverse mask map to obtain the vacant map, and then multiply the vacant map with the corresponding mask map X mask After adding, the contour background image is obtained; the anti-mask image is the mask image X mask The result after negation; The generator includes: The encoding module is used to extract features from the contour background image to obtain an initial feature map of the contour background image; split the initial feature map into multiple groups of feature maps by channel, perform pyramid convolution of different scales on each group of feature maps and merge them, and fuse the merged feature map with the initial feature map based on a residual mechanism to obtain a fused feature map; repeatedly perform the splitting, merging and fusing operations on the fused feature map of the previous level until the number of repetitions reaches a preset number; and learn the overall correlation information in the fused feature map obtained in the last step based on the self-attention mechanism to obtain a feature encoding map of the contour background image; A decoding module is used to perform an upsampling operation on the feature coding map to generate a glomerular image X ori New glomerulus image X of the same size rec .

2. The method for constructing a glomerular image generation model according to claim 1, wherein: In a first method, the training sample set is input into the MSPSA-CGAN model for training, and the generator parameters and the discriminator parameters in the MSPSA-CGAN model are alternately updated by minimizing the overall image loss to obtain a glomerular image generation model; or, In the second method, the training sample set is input into the MSPSA-CGAN model for training, and the generator parameters and the discriminator parameters in the MSPSA-CGAN model are alternately updated by minimizing the overall image loss and the local image loss to obtain a glomerular image generation model; Among them, the overall image loss is X ori With X rec The difference loss between them includes: ori With X rec The overall adversarial loss of distribution difference is used to measure X ori With X rec The overall pixel loss of pixel differences and the metric X ori With X rec Overall feature loss for feature variance; The local loss of the image is R ori With R rec The difference loss between them includes: ori With R rec Local structure loss of structural difference, used to measure R ori With R rec The local pixel loss of pixel difference and the metric R ori With R rec Local adversarial loss for distribution difference; R ori For X ori The glomerular region in the kidney, specifically X ori With X mask The result after dot product; R rec For X rec The glomerular region in the kidney, specifically X rec With X mask The result of the dot product.

3. The method for constructing a glomerular image generation model according to claim 2, wherein: In the above method, the overall adversarial loss is: The overall pixel loss is: The overall feature loss includes: perceptual loss and style loss; Where E[·] represents the mathematical expectation; D(X rec ) is the pair X output by the decision maker rec The result of judging whether the different positions in the glomerulus are the real glomerular regions; D(X ori ) is the pair X output by the decision maker ori The result of judging whether the different positions are the real glomerular areas.

4. The method for constructing a glomerular image generation model according to claim 2, wherein: In the second embodiment, the discriminator includes a first judgment module and a second judgment module; The overall adversarial loss is: The overall pixel loss is: The overall feature loss includes: perceptual loss and style loss; The local structural loss is: L ssim =1-SSIM(R ori ,R rec ) The local pixel loss is: The local adversarial loss is: Where E[·] represents the mathematical expectation; D1(X rec ) is the pair X output by the first decision module rec The result of judging whether the different positions in the image are the real glomerular regions; D1(X ori ) is the pair X output by the first decision module ori The result of judging whether different positions are the real glomerular regions; SSIM(·) is the structure measurement index function; D2(R rec ) is the pair R output by the second decision module rec The result of judging whether the different positions in the glomerulus are the real glomerular regions; D2(R ori ) is the pair R output by the second decision module ori The result of judging whether the different positions are the real glomerular areas.

5. A method for generating a glomerular image, characterized in that: include: A glomerular image without a glomerular region is used as a reference image, and a mask image having the same size as the reference image is randomly selected; Inputting the reference image and the mask image into a glomerular image generation model to obtain a corresponding new image containing a glomerular region; or, A glomerular image without glomerular regions is used as a reference image, and a mask image with the same size as the reference image is randomly selected; a new mask image similar to the mask image is generated using a GAN algorithm; the reference image and the new mask image are input into a glomerular image generation model to obtain a corresponding new image containing the glomerular region; Wherein, the glomerular image generation model is constructed using the method for constructing a glomerular image generation model according to any one of claims 1 to 4.

6. A method for constructing a glomerular detection model, characterized in that: include: Performing sample enhancement on the collected glomerular images in the training set, generating new glomerular images and adding them to the training set to expand the samples in the training set; The expanded training set is input into the target detection model for training to obtain the glomerulus detection model; Wherein, the training set includes glomerular images carrying position and type labels; the method for performing sample enhancement on glomerular images includes the method for generating glomerular images according to claim 5.

7. The method for constructing a glomerular detection model according to claim 6, wherein: The method for sample enhancement of glomerular images further includes: rotating and / or flipping the glomerular images in the training set, modifying their position labels accordingly, and then adding them to the training set.

8. A glomerular detection system, characterized in that: include: A model construction module, configured to execute the method for constructing a glomerular detection model according to claim 6 or 7 to obtain a glomerular detection model; The glomerulus detection module is used to input the glomerulus image to be detected into the glomerulus detection model to obtain the area and type of the glomerulus.

9. The glomerular detection system according to claim 8, characterized in that: Also includes: The glomerular assessment module is used to crop the original glomerular WSI image, divide it into small images, perform preprocessing, and input them into the glomerular detection module for detection, thereby obtaining a detection frame and type for identifying the glomerular area in each small image; restore each small image to its original position, correct the overlapping detection frames, count the total number of glomeruli in the WSI image, and calculate the ratio of the number of sclerosis-type glomeruli to the total number of glomeruli to obtain the sclerosis ratio; wherein the type of glomerulus includes sclerosis type and non-sclerosis type.

10. The glomerular detection system according to claim 9, characterized in that: The glomerulus assessment module corrects the overlapping detection frames by retaining only the detection frames whose intersection-over-union ratio with the detection frame with the highest confidence score is lower than a first preset threshold and whose center point distance is greater than a second preset threshold.

Images