A foot process identification method, device and terminal equipment for an electron microscope image of chronic kidney disease

Abstract

The application discloses a foot process identification method and device for electron microscope images of chronic kidney disease and terminal equipment, obtains the electron microscope pathological images of chronic kidney disease with labeled foot process position characteristics, and trains a preset deep network architecture through the electron microscope pathological images of chronic kidney disease to obtain a foot process identification model, so that when the electron microscope images of chronic kidney disease to be detected are obtained, the automatic identification of the foot process area corresponding to the electron microscope images to be detected can be realized through the obtained foot process identification model, and the automatic identification of the foot process area of the electron microscope images of chronic kidney disease is realized. The application accurately and quickly identifies the area corresponding to the foot process through the deep learning mode, improves the foot process identification efficiency of the electron microscope images of chronic kidney disease, is favorable to improving the pathological diagnosis and analysis efficiency of various types of glomerulonephritis for pathologists, and can reduce misdiagnosis and missed diagnosis caused by insufficient experience.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, and in particular to a foot process identification method and device for electron microscope images of chronic kidney disease and a terminal device. BACKGROUND

[0002] Chronic kidney disease is insidious in onset and has no obvious symptoms. Among every 100 patients with chronic kidney disease, one has developed into uremia and needs to receive lifelong kidney replacement therapy. Chronic kidney disease not only affects the quality of life of patients, but also increases the economic burden on families. Kidney tissue pathological examination is the gold standard for diagnosing the type of chronic kidney disease. Electron microscopy has very important value for kidney biopsy diagnosis. Electron microscopy is convenient for observing the site of deposits and has obvious advantages in observing ultrastructure, including crystalline material and foot process fusion. Podocytes are a unique cell with complex finger-like processes (i.e. foot processes) covering the outer surface of the glomerular basement membrane, thereby forming an important structure of the glomerular filtration barrier. Podocyte injury is a marker of many glomerular diseases, including minimal change disease, focal segmental glomerulosclerosis (FSGS), membranous glomerulopathy, diabetic nephropathy, and lupus nephritis. Considering the importance of foot process localization and fusion in kidney disease diagnosis, pathologists need to view different magnification images of the same case when diagnosing kidney pathological tissue electron microscope images to ensure that existing problems can be found. After confirming the region of interest, a diagnosis is given after analyzing the dense material and basement membrane.

[0003] However, during the process of artificial analysis and diagnosis of kidney electron microscope images by pathologists according to pathological morphology, the following problems exist: (1) The workload of identifying target objects by naked eye observation is very large, fatigue is easy to occur, and human subjectivity is relatively strong, which can eventually lead to an increase in misdiagnosis rate and missed diagnosis rate. (2) The diagnosis of chronic kidney disease requires a high level of professionalism from pathologists. Currently, the identification of foot processes and the diagnosis of kidney electron microscope images both require experienced kidney disease pathology specialists. In the current situation of a lack of pathologists, the reading of kidney foot processes, the diagnosis speed, and the quality are all too low.

[0004] Therefore, there is an urgent need for a foot process identification strategy for electron microscope images of chronic kidney disease to solve the problem of low efficiency of foot process identification for electron microscope images of chronic kidney disease. SUMMARY

[0005] Embodiments of the present application provide a foot process identification method, device, and terminal equipment for electron microscope images of chronic kidney disease to improve the efficiency of foot process identification for electron microscope images of chronic kidney disease.

[0006] To solve the above problems, an embodiment of the present application provides a foot process identification method for electron microscope images of chronic kidney disease, comprising:

[0007] Obtain a plurality of electron microscope pathological images of chronic kidney disease; wherein the electron microscope pathological images of chronic kidney disease are images with labeled foot process position features;

[0008] Model training is performed on the preset deep network architecture using a plurality of the electron microscope pathological images of chronic kidney disease to obtain a foot process identification model.

[0009] When obtaining an electron microscope image to be tested of chronic kidney disease, the foot process identification model is used to identify the foot process region of the electron microscope image to be tested.

[0010] As an improvement of the above scheme, the model training on the preset deep network architecture using a plurality of the electron microscope pathological images of chronic kidney disease to obtain a foot process identification model comprises:

[0011] The preset deep network architecture comprises a preset encoder, a preset decoder, and a preset deep network structure.

[0012] A plurality of sample images are extracted from a plurality of the electron microscope pathological images of chronic kidney disease according to a preset sample number.

[0013] Each sample image is input into the preset encoder for down-sampling operation to obtain a plurality of feature maps corresponding to each sample image.

[0014] According to the plurality of feature maps, a decoding result corresponding to each sample image is obtained through a preset scale judgment operation and a preset decoder.

[0015] The decoding results corresponding to the plurality of sample images are input into the preset deep network structure for training to obtain a first foot process identification model; wherein the foot process identification model comprises the first foot process identification model.

[0016] As an improvement of the above scheme, the decoding result corresponding to each sample image is obtained according to the plurality of feature maps through a preset scale judgment operation and a preset decoder, and specifically:

[0017] The plurality of feature maps are respectively input into the preset decoder for pre-decoding to obtain pre-decoding feature scales of the plurality of feature maps.

[0018] The pre-decoding feature scales of the plurality of feature maps are respectively subjected to a preset scale judgment operation to obtain a plurality of first decoding features, a plurality of second decoding features, and a plurality of third decoding features of each sample image.

[0019] Through a channel splicing operation, the plurality of first decoding features, the plurality of second decoding features, and the plurality of third decoding features are spliced into a decoding result of each sample image.

[0020] As an improvement of the above scheme, the preset scale judgment operation comprises:

[0021] When the pre-decoding feature scale is equal to the scale of the feature map, the feature map corresponding to the pre-decoding feature scale is a same-layer feature, the same-layer feature is input into a preset decoder for decoding to obtain a first decoding feature corresponding to the same-layer feature;

[0022] When the pre-decoding feature scale is smaller than the scale of the feature map, the feature map corresponding to the pre-decoding feature scale is a shallow-layer feature, the shallow-layer feature is subjected to a down-sampling operation, and the shallow-layer feature subjected to the down-sampling operation is input into the preset decoder for decoding to obtain a second decoding feature corresponding to the shallow-layer feature;

[0023] When the pre-decoding feature scale is greater than the scale of the feature map, the feature map corresponding to the pre-decoding feature scale is a deep-layer feature, the deep-layer feature is subjected to an up-sampling operation, and the deep-layer feature subjected to the up-sampling operation is input into the preset decoder for decoding to obtain a third decoding feature corresponding to the deep-layer feature.

[0024] As an improvement of the above scheme, the model training of the preset deep network architecture by using the plurality of electron microscopy pathological images of chronic kidney disease to obtain the podocyte recognition model further comprises:

[0025] After the plurality of feature maps corresponding to each of the sample images are obtained, a preset deep feature map is selected for a multi-scale fusion operation to obtain a deep feature map corresponding to each of the sample images;

[0026] The deep feature map is input into a preset attention mechanism, and a preset image processing operation is performed to obtain a prediction map corresponding to each of the sample images;

[0027] The prediction maps corresponding to the plurality of sample images are input into a preset deep network structure for training to obtain a second podocyte recognition model; wherein the podocyte recognition model comprises the second podocyte recognition model.

[0028] As an improvement of the above scheme, after the decoding result corresponding to each of the sample images is obtained, the decoding result is further processed by an argmax function.

[0029] As an improvement of the above scheme, after the podocyte recognition model is obtained, the method further comprises:

[0030] An annotated image of a to-be-tested electron microscopy image of chronic kidney disease is obtained;

[0031] The podocyte prediction image output by the podocyte recognition model and the annotated image are subjected to precision calculation by a Dice coefficient to obtain podocyte recognition precision;

[0032] The foot process recognition model is evaluated according to the foot process recognition accuracy.

[0033] Correspondingly, the embodiment of the application further provides a foot process recognition device for electron microscope images of chronic kidney disease, comprising a data acquisition module, a model training module and a model application module.

[0034] The data acquisition module is configured to acquire a plurality of electron microscope pathological images of chronic kidney disease, wherein the electron microscope pathological images of chronic kidney disease are images with labeled foot process position features.

[0035] The model training module is configured to train a preset deep network architecture using the plurality of electron microscope pathological images of chronic kidney disease to obtain a foot process recognition model.

[0036] The model application module is configured to identify a foot process region of a to-be-tested electron microscope image of chronic kidney disease by using the foot process recognition model when the to-be-tested electron microscope image of chronic kidney disease is acquired.

[0037] As an improvement of the above-mentioned scheme, the model training module comprises a classification unit, an extraction unit, an encoding unit, a decoding unit and a first training unit.

[0038] The classification unit is configured to use the preset deep network architecture, which comprises a preset encoder, a preset decoder and a preset deep network structure.

[0039] The extraction unit is configured to extract a plurality of sample images from the plurality of electron microscope pathological images of chronic kidney disease according to a preset sample number.

[0040] The encoding unit is configured to input each of the sample images into the preset encoder to perform a downsampling operation, thereby obtaining a plurality of feature maps corresponding to each of the sample images.

[0041] The decoding unit is configured to obtain a decoding result corresponding to each of the sample images by using a preset scale judgment operation and the preset decoder according to the plurality of feature maps.

[0042] The first training unit is configured to input the decoding result corresponding to each of the sample images into the preset deep network structure to perform training, thereby obtaining a first foot process recognition model; wherein the foot process recognition model comprises the first foot process recognition model.

[0043] As an improvement of the above-mentioned scheme, the decoding result corresponding to each of the sample images is obtained by using the preset scale judgment operation and the preset decoder according to the plurality of feature maps, and specifically:

[0044] The plurality of feature maps are respectively input into the preset decoder to perform pre-decoding, thereby obtaining pre-decoding feature scales of the plurality of feature maps.

[0045] Respectively, the pre-decoding feature scale of a plurality of feature maps is subjected to a preset scale judgment operation, and a plurality of first decoding features, a plurality of second decoding features and a plurality of third decoding features of each sample image are obtained.

[0046] Through a channel splicing operation, a plurality of first decoding features, a plurality of second decoding features and a plurality of third decoding features are spliced into a decoding result of each sample image.

[0047] As an improvement of the above scheme, the preset scale judgment operation comprises:

[0048] When the pre-decoding feature scale is equal to the scale of the feature map, the feature map corresponding to the pre-decoding feature scale is a same layer feature, and the same layer feature is input into a preset decoder for decoding to obtain a first decoding feature corresponding to the same layer feature;

[0049] When the pre-decoding feature scale is smaller than the scale of the feature map, the feature map corresponding to the pre-decoding feature scale is a shallow layer feature, and a downsampling operation is performed on the shallow layer feature, and the shallow layer feature after the downsampling operation is input into the preset decoder for decoding to obtain a second decoding feature corresponding to the shallow layer feature;

[0050] When the pre-decoding feature scale is greater than the scale of the feature map, the feature map corresponding to the pre-decoding feature scale is a deep layer feature, and an upsampling operation is performed on the deep layer feature, and the deep layer feature after the upsampling operation is input into the preset decoder for decoding to obtain a third decoding feature corresponding to the deep layer feature.

[0051] As an improvement of the above scheme, the model training module further comprises a fusion unit, an image processing unit and a second training unit;

[0052] The fusion unit is configured to, after obtaining a plurality of feature maps corresponding to each sample image, select a preset deep feature map to perform a multi-scale fusion operation to obtain a deep feature map corresponding to each sample image.

[0053] The image processing unit is configured to input the deep feature map into a preset attention mechanism, and perform a preset image processing operation to obtain a prediction map corresponding to each sample image.

[0054] The second training unit is configured to input the prediction maps corresponding to a plurality of sample images into a preset deep network structure for training to obtain a second foot spike recognition model; wherein the foot spike recognition model comprises a second foot spike recognition model.

[0055] As an improvement of the above scheme, after the decoding result corresponding to each of the sample images is obtained, the decoding result is further processed by an argmax function.

[0056] As an improvement of the above scheme, after the podocalyxin recognition model is obtained, the method further comprises:

[0057] An annotated image of a to-be-tested electron microscope image of chronic kidney disease is obtained.

[0058] The podocalyxin prediction image output by the podocalyxin recognition model and the annotated image are subjected to precision calculation by a Dice coefficient to obtain podocalyxin recognition precision.

[0059] The podocalyxin recognition model is evaluated by using the podocalyxin recognition precision.

[0060] Correspondingly, an embodiment of the present application further provides a computer terminal device, which comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and the processor implements the podocalyxin recognition method of the electron microscope image of chronic kidney disease when executing the computer program.

[0061] Correspondingly, an embodiment of the present application further provides a computer-readable storage medium, which comprises a stored computer program, wherein the computer-readable storage medium controls a device where the computer-readable storage medium is located to execute the podocalyxin recognition method of the electron microscope image of chronic kidney disease when the computer program runs.

[0062] As can be seen from the above, the present application has the following beneficial effects:

[0063] The present application provides a podocalyxin recognition method of an electron microscope image of chronic kidney disease, obtains an electron microscope pathological image of chronic kidney disease with annotated podocalyxin position features, and trains a preset deep network architecture by using the electron microscope pathological image of chronic kidney disease to obtain a podocalyxin recognition model, so that when a to-be-tested electron microscope image of chronic kidney disease is obtained, the to-be-tested electron microscope image can be automatically recognized by the obtained podocalyxin recognition model to realize automatic recognition of the podocalyxin region of the electron microscope image of chronic kidney disease. The present application accurately and quickly recognizes the region corresponding to the podocalyxin by means of deep learning, improves the podocalyxin recognition efficiency of the electron microscope image of chronic kidney disease, is conducive to improving the pathological diagnosis and analysis efficiency of various types of glomerulonephritis by a pathologist, and can reduce misdiagnosis and missed diagnosis caused by insufficient experience. BRIEF DESCRIPTION OF DRAWINGS

[0064] Figure 1 is a flowchart of the podocalyxin recognition method of the electron microscope image of chronic kidney disease provided by an embodiment of the present application;

[0065] Figure 2 is a structural schematic diagram of a foot process recognition device for electron microscope images of chronic kidney disease provided by an embodiment of the present application;

[0066] Figure 3 is an artificially annotated image of an electron microscope image of chronic kidney disease provided by an embodiment of the present application;

[0067] Figure 4 is a to-be-detected electron microscope image of chronic kidney disease provided by an embodiment of the present application;

[0068] Figure 5 is a foot process segmentation result of a to-be-detected electron microscope image of chronic kidney disease provided by an embodiment of the present application;

[0069] Figure 6 is a structural schematic diagram of a terminal device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0070] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0071] Embodiment One

[0072] Referring to Figure 1 , Figure 1 is a flowchart of a foot process recognition method for electron microscope images of chronic kidney disease provided by an embodiment of the present application, as shown in the figure, the embodiment includes steps 101 to 103, and each step is specifically as follows: Figure 1

[0073] Step 101: Obtain a plurality of electron microscope pathological images of chronic kidney disease; wherein the electron microscope pathological images of chronic kidney disease are images with annotated foot process position features.

[0074] In the embodiment, the electron microscope pathological images of chronic kidney disease in the historical electron microscope report are screened, the glomerular electron microscope images with clear glomerular structure and magnification of 4000 are selected, and the foot processes of the above glomerular electron microscope images are annotated to form a tissue region division label, so as to obtain a data set S, S = (X, Y), X and Y represent a picture and a label set respectively, X = (X1, X2, Xm, …, Xi), Xi represents the i-th electron microscope picture, there are i pictures in the data set S, Y = (y1, y2, ym, …, yi), yi represents the foot process annotation label corresponding to the i-th electron microscope picture.

[0075] ​As preferred, the picture size can be 2048*2048, and the picture format can be.JEG.

[0076] In a specific embodiment, the embodiment can be applied to the identification of foot processes of seven types of glomerulonephritis, and the seven types of glomerulonephritis are IgA nephropathy, thin basement membrane nephropathy, membranous nephropathy, minimal change nephropathy, mesangial proliferative glomerulonephritis, diabetic nephropathy, and Chinese lupus nephritis.

[0077] During the process of glomerular injury stress, the shape of podocytes often changes, and foot process fusion is the most common change in foot process. Among the seven types of glomerulonephritis, foot process fusion occurs in membranous nephropathy, minimal change nephropathy, diabetic nephropathy, and lupus nephritis. However, the foot processes of IgA nephropathy, thin basement membrane nephropathy, and mesangial proliferative glomerulonephritis are generally normal.

[0078] Step 102: model training is performed on a preset deep network architecture using a plurality of said chronic kidney disease electron microscope pathological images to obtain a foot process identification model.

[0079] In the embodiment, the model training on the preset deep network architecture using a plurality of said chronic kidney disease electron microscope pathological images to obtain a foot process identification model includes:

[0080] The preset deep network architecture includes a preset encoder, a preset decoder, and a preset deep network structure.

[0081] A plurality of sample images are extracted from a plurality of said chronic kidney disease electron microscope pathological images according to a preset sample number.

[0082] Each of the sample images is input into the preset encoder for downsampling operation to obtain a plurality of feature maps corresponding to each of the sample images.

[0083] According to a plurality of said feature maps, a decoding result corresponding to each of the sample images is obtained through a preset scale judgment operation and a preset decoder.

[0084] The decoding results corresponding to a plurality of said sample images are input into the preset deep network structure for training to obtain a first foot process identification model; wherein the foot process identification model includes the first foot process identification model.

[0085] In a specific embodiment, all sample images are input into the preset deep network structure, and after each sample image undergoes a series of operations (functions corresponding to the preset deep network structure) such as dropout, convolution, maxpooling, and sigmoid, an HxWx1 image is generated as a segmentation result, and the value of each pixel is between 0 and 1, indicating the probability that the pixel is a foot process.

[0086] In a specific embodiment, the preset deep network architecture is improved and optimized based on a U-Net3+ deep network model.

[0087] In a specific embodiment, the preset number of sample images are extracted from the plurality of electron microscopy pathological images of chronic kidney disease, specifically: the prepared data set S is randomly divided into two parts, S1 data set is used to train the deep learning model, and S2 data set is used to test the model accuracy.

[0088] In a specific embodiment, the encoder part is responsible for down-sampling the input image, and different backbone networks such as Unet, VGGnet, Resnet, ASPP, and RFB can be used as the encoder (the experimental effect of the encoder VGGnet is the best). The input picture of HxWx3 is down-sampled to obtain 5 scale feature maps X

[0089] In this embodiment, the decoding result corresponding to each of the sample images is obtained according to the plurality of feature maps, through a preset scale judgment operation and a preset decoder, specifically:

[0090] Respectively inputting the plurality of feature maps into the preset decoder for pre-decoding to obtain pre-decoding feature scales of the plurality of feature maps;

[0091] Respectively performing a preset scale judgment operation on the pre-decoding feature scales of the plurality of feature maps to obtain a plurality of first decoding features, a plurality of second decoding features, and a plurality of third decoding features of each of the sample images;

[0092] Through a channel splicing operation, the plurality of first decoding features, the plurality of second decoding features, and the plurality of third decoding features are spliced into the decoding result of each of the sample images.

[0093] In a specific embodiment, the decoder part is used to fuse the feature maps of 5 scales to obtain the decoding result Y i ,i∈{1,2,3,4,5} is operated as follows: for the same layer feature X i , 64 3x3 convolution kernels are used to obtain the decoding feature; for the shallow layer feature X j ,j<i, first down-sampling to the current layer resolution using maximum pooling, and then using the above convolution operation; for the deep layer feature Y jThe 64-channel decoding features obtained from different levels are spliced by channel to obtain a final 320-channel decoding result.

[0094] In the embodiment, the preset scale judgment operation includes:

[0095] When the scale of the pre-decoding feature is equal to the scale of the feature map, the feature map corresponding to the pre-decoding feature is a same-level feature, and the same-level feature is input into the preset decoder to obtain a first decoding feature corresponding to the same-level feature.

[0096] When the scale of the pre-decoding feature is smaller than the scale of the feature map, the feature map corresponding to the pre-decoding feature is a shallow-level feature, the shallow-level feature is down-sampled, and the shallow-level feature after the down-sampling operation is input into the preset decoder to obtain a second decoding feature corresponding to the shallow-level feature.

[0097] When the scale of the pre-decoding feature is greater than the scale of the feature map, the feature map corresponding to the pre-decoding feature is a deep-level feature, the deep-level feature is up-sampled, and the deep-level feature after the up-sampling operation is input into the preset decoder to obtain a third decoding feature corresponding to the deep-level feature.

[0098] In a specific embodiment, the same-level feature refers to an encoding feature map with the same scale as the decoding feature of the layer, the shallow-level feature has a larger scale, and the deep-level feature has a smaller scale. Taking i=3 as an example, the scale of Y3 is H / 4*W / 4, X3 with the same scale is the same-level feature, X1 and X2 with larger scales are the shallow-level features, and X4 and X5 with smaller scales are the deep-level features.

[0099] In the embodiment, the model training on the preset deep network architecture by using the plurality of electron microscopy pathological images of chronic kidney disease obtains a foot process identification model, and further includes:

[0100] After the plurality of feature maps corresponding to each of the sample images are obtained, a preset deep feature map is selected to perform a multi-scale fusion operation to obtain a deep feature map corresponding to each of the sample images.

[0101] The deep feature map is input into a preset attention mechanism to obtain a prediction map corresponding to each of the sample images through a preset image processing operation.

[0102] The prediction maps corresponding to the plurality of sample images are input into a preset deep network structure to obtain a second foot process identification model; wherein the foot process identification model includes the second foot process identification model.

[0103] In one specific embodiment, to reduce computational complexity, the depth feature map X is used in the feature map X obtained at the 5 levels. i Multi-scale fusion is performed on i∈{3,4,5} to obtain feature map U. Then, a Squeeze-and-Excitation (SE) Block is input to implement the attention mechanism of channel fusion, which brings significant performance improvement to the existing deep architecture with minimal additional computational cost. The feature map output by the SE Block is binarized and scaled to generate the prediction map.

[0104] In this embodiment, after obtaining the decoding result corresponding to each sample image, the method further includes: processing the decoding result using the argmax function.

[0105] In one specific implementation, with the help of the argmax function, the two-dimensional tensor is converted into a single output of {0,1}, representing the presence or absence of the target image, thus providing guidance for compensating for oversegmentation defects in the absence of the target image.

[0106] After obtaining the foot process recognition model, the method further includes:

[0107] Obtain labeled images of the electron microscopy images of chronic kidney disease patients;

[0108] The accuracy of foot protrusion recognition is obtained by calculating the accuracy of the predicted and labeled images of the foot protrusion output by the foot protrusion recognition model using the Dice coefficient.

[0109] The foot protrusion recognition model is evaluated using the foot protrusion recognition accuracy.

[0110] In one specific embodiment, in addition to visual effects, this example also objectively evaluates the effectiveness of the invention through quantitative indicators. This example uses the gold standard image manually segmented by pathology experts and employs the Dice coefficient of the image to evaluate the segmentation effect. The Dice coefficient is a geometric similarity function, ranging from [0,1]. A larger Dice coefficient indicates a more accurate segmentation result, and its definition is:

[0111]

[0112] In the formula, A is the predicted image of the foot process output by the model, and B is the manually annotated image.

[0113] In a specific embodiment, the method of the present invention was evaluated based on quantitative indicators. The average Dice coefficient was 0.648, indicating that the segmentation results of the invention are close to the gold standard. Furthermore, the improved Unet3+ can improve the Dice coefficient in the foot segmentation task by about 0.2.

[0114] Step 103: When the to-be-tested electron microscope image of chronic kidney disease is obtained, the foot process region of the to-be-tested electron microscope image is identified through the foot process identification model.

[0115] For better illustration, please refer to Figure 3 、 Figure 4 and Figure 5 , Figure 3 artificially annotated images of electron microscope images of chronic kidney disease; Figure 4 to-be-tested electron microscope images of chronic kidney disease; Figure 5 foot process segmentation results of to-be-tested electron microscope images of chronic kidney disease.

[0116] The embodiment obtains electron microscope pathological images of chronic kidney disease with annotated foot process position features, and trains a preset deep network architecture through the electron microscope pathological images of chronic kidney disease to obtain a foot process identification model. Therefore, when a to-be-tested electron microscope image of chronic kidney disease is obtained, the corresponding foot process region of the to-be-tested electron microscope image can be automatically identified through the obtained foot process identification model, and the automatic identification of the foot process region of the electron microscope image of chronic kidney disease is realized. The training method of the present application can accurately and quickly identify the region corresponding to the foot process in the electron microscope image and separate it from the surrounding tissue, which is convenient for digital reading and can improve the pathological diagnosis and analysis efficiency of pathologists on various types of glomerulonephritis, and can reduce misdiagnosis and missed diagnosis caused by insufficient experience.

[0117] Embodiment two

[0118] For better illustration, please refer to Figure 2 , Figure 2 is a structural schematic diagram of a foot process identification device for electron microscope images of chronic kidney disease provided by an embodiment of the present application, comprising a data acquisition module 201, a model training module 202 and a model application module 203.

[0119] The data acquisition module 201 is configured to acquire a plurality of electron microscope pathological images of chronic kidney disease; wherein the electron microscope pathological images of chronic kidney disease are images with annotated foot process position features.

[0120] The model training module 202 is configured to train a preset deep network architecture using a plurality of electron microscope pathological images of chronic kidney disease to obtain a foot process identification model.

[0121] The model application module 203 is configured to identify the foot process region of the to-be-tested electron microscope image through the foot process identification model when a to-be-tested electron microscope image of chronic kidney disease is obtained.

[0122] As an improvement of the above-mentioned scheme, the model training module 202 comprises a classification unit, an extraction unit, an encoding unit, a decoding unit and a first training unit.

[0123] The classification unit is configured to include a preset encoder, a preset decoder, and a preset deep network structure.

[0124] The extraction unit is configured to extract a plurality of sample images from the plurality of electron microscopy images of chronic kidney disease according to a preset sample number.

[0125] The encoding unit is configured to input each of the sample images into a preset encoder to perform a downsampling operation, and obtain a plurality of feature maps corresponding to each of the sample images.

[0126] The decoding unit is configured to obtain a decoding result corresponding to each of the sample images according to the plurality of feature maps by a preset scale judgment operation and a preset decoder.

[0127] The first training unit is configured to input the decoding results corresponding to the plurality of sample images into a preset deep network structure to perform training, and obtain a first podosome recognition model; wherein the podosome recognition model includes the first podosome recognition model.

[0128] As an improvement of the above scheme, the decoding result corresponding to each of the sample images is obtained according to the plurality of feature maps by a preset scale judgment operation and a preset decoder, specifically:

[0129] The plurality of feature maps are respectively input into a preset decoder to perform pre-decoding, and obtain pre-decoding feature scales of the plurality of feature maps.

[0130] The pre-decoding feature scales of the plurality of feature maps are respectively subjected to a preset scale judgment operation, and a plurality of first decoding features, a plurality of second decoding features, and a plurality of third decoding features of each of the sample images are obtained.

[0131] The plurality of first decoding features, the plurality of second decoding features, and the plurality of third decoding features are spliced into the decoding result of each of the sample images by a channel splicing operation.

[0132] As an improvement of the above scheme, the preset scale judgment operation includes:

[0133] When the pre-decoding feature scale is equal to the scale of the feature map, the feature map corresponding to the pre-decoding feature scale is a same-layer feature, the same-layer feature is input into a preset decoder to perform decoding, and a first decoding feature corresponding to the same-layer feature is obtained.

[0134] When the pre-decoding feature scale is smaller than the scale of the feature map, the feature map corresponding to the pre-decoding feature scale is a shallow-layer feature, the shallow-layer feature is subjected to a downsampling operation, and the shallow-layer feature after the downsampling operation is input into a preset decoder to perform decoding, and a second decoding feature corresponding to the shallow-layer feature is obtained.

[0135] When the pre-decoding feature scale is greater than the scale of the feature map, the feature map corresponding to the pre-decoding feature scale is a deep feature, the deep feature is subjected to an up-sampling operation, and the deep feature after the up-sampling operation is input into a preset decoder for decoding to obtain a third decoding feature corresponding to the deep feature.

[0136] As an improvement of the above scheme, the model training module 202 further comprises a fusion unit, an image processing unit and a second training unit.

[0137] The fusion unit is configured to, after obtaining the plurality of feature maps corresponding to each of the sample images, select a preset deep feature map to perform a multi-scale fusion operation to obtain a deep feature map corresponding to each of the sample images.

[0138] The image processing unit is configured to input the deep feature map into a preset attention mechanism, and perform a preset image processing operation to obtain a predicted image corresponding to each of the sample images.

[0139] The second training unit is configured to input the predicted images corresponding to the plurality of sample images into a preset deep network structure for training to obtain a second foot process identification model; wherein the foot process identification model comprises the second foot process identification model.

[0140] As an improvement of the above scheme, after obtaining the decoding result corresponding to each of the sample images, the method further comprises processing the decoding result by using an argmax function.

[0141] As an improvement of the above scheme, after obtaining the foot process identification model, the method further comprises:

[0142] Obtaining a labeled image of a to-be-detected electron microscope image of chronic kidney disease;

[0143] Calculating the accuracy of the foot process identification model by using a Dice coefficient to calculate the accuracy of the foot process prediction image output by the foot process identification model and the labeled image.

[0144] Evaluating the foot process identification model by using the foot process identification accuracy.

[0145] The embodiment obtains the chronic kidney disease electron microscope image through the data acquisition module, inputs the obtained image into the model training module to train the foot process identification model, and finally identifies the foot process area of the to-be-tested electron microscope image through the foot process identification model when the to-be-tested electron microscope image of the chronic kidney disease is obtained, so that the automatic identification of the foot process area of the chronic kidney disease electron microscope image is realized. The foot process area corresponding to the foot process is accurately and quickly identified through the deep learning mode, the foot process identification efficiency of the chronic kidney disease electron microscope image is improved, the pathological diagnosis and analysis efficiency of the pathologist on various types of glomerulonephritis is improved, and misdiagnosis and missed diagnosis caused by insufficient experience are reduced.

[0146] Embodiment three

[0147] Reference Figure 6 , Figure 6 is a terminal device structure schematic diagram provided by an embodiment of the present application.

[0148] The terminal device of the embodiment includes a processor 601, a memory 602, and a computer program stored in the memory 602 and executable on the processor 601. The processor 601 implements the steps of the above-mentioned various chronic kidney disease electron microscope image foot process identification methods in the embodiment when executing the computer program, for example, all steps of the chronic kidney disease electron microscope image foot process identification method shown in the embodiment. Figure 1 The processor implements the functions of the modules in the above-mentioned various device embodiments when executing the computer program, for example, all modules of the chronic kidney disease electron microscope image foot process identification device shown in the embodiment. Figure 2

[0149] In addition, the embodiment of the present application further provides a computer readable storage medium, which includes a stored computer program, wherein when the computer program runs, the device where the computer readable storage medium is located is controlled to execute the chronic kidney disease electron microscope image foot process identification method according to any one of the above-mentioned embodiments.

[0150] Those skilled in the art can understand that the schematic diagram is only an example of the terminal device and does not constitute a limitation on the terminal device, and can include more or fewer components than the diagram, or combine certain components, or different components, for example, the terminal device can also include an input and output device, a network access device, a bus, etc.

[0151] ​The processor 601 can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The processor 601 is a control center of the terminal device, and is connected with various parts of the terminal device through various interfaces and lines.

[0152] The memory 602 can be used to store computer programs and / or modules, and the processor 601 realizes various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory and calling data stored in the memory 602. The memory 602 can mainly include a program storage area and a data storage area, wherein the program storage area can store operating systems, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), etc.; and the data storage area can store data created according to use of the terminal device (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a nonvolatile memory, for example, a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash storage device, or other volatile solid-state storage device.

[0153] The modules / units integrated in the terminal device, if in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. Based on this understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware. The computer program can be stored in a computer readable storage medium. When the computer program is executed by a processor, the steps of the above-mentioned various method embodiments can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms, etc. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0154] It should be noted that the above-described device embodiments are only schematic, and the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e. they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. In addition, the connection relationship between the modules in the device embodiment provided by the present application indicates that there is a communication connection between them, which can be realized as one or more communication buses or signal lines. Those skilled in the art can understand and implement it without creative labor.

[0155] The above is the preferred embodiment of the present application. It should be noted that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which are also considered within the scope of protection of the present application.

Claims

1. A method for identifying foot processes in electron microscopic images of chronic kidney disease, characterized in that, include: Acquire several electron microscopic pathological images of chronic kidney disease; wherein, the electron microscopic pathological images of chronic kidney disease are images with foot process location features marked; A foot process recognition model is obtained by training a pre-defined deep network architecture using several electron microscopic pathological images of chronic kidney disease. The pre-defined deep network architecture includes a pre-defined encoder, a pre-defined decoder, and a pre-defined deep network structure. Several sample images are extracted from the pre-defined electron microscopic pathological images of chronic kidney disease according to a pre-defined number of samples. Each sample image is input into the pre-defined encoder for downsampling to obtain several feature maps corresponding to each sample image. Based on the feature maps, a pre-defined scale judgment operation and a pre-defined decoder are used to obtain the decoding result corresponding to each sample image. The pre-defined scale judgment operation includes: when the scale of the pre-decoded feature is equal to the scale of the feature map, the feature map corresponding to the pre-decoded feature scale is considered a feature at the same layer, and the feature at the same layer is input into the pre-defined decoder for decoding. The system obtains the first decoded feature corresponding to the same layer feature; when the scale of the pre-decoded feature is smaller than the scale of the feature map, the feature map corresponding to the pre-decoded feature scale is a shallow feature, the shallow feature is downsampled, and the shallow feature after the downsampling operation is input into a preset decoder for decoding to obtain the second decoded feature corresponding to the shallow feature; when the scale of the pre-decoded feature is larger than the scale of the feature map, the feature map corresponding to the pre-decoded feature scale is a deep feature, the deep feature is upsampled, and the deep feature after the upsampling operation is input into a preset decoder for decoding to obtain the third decoded feature corresponding to the deep feature; the decoding results corresponding to several sample images are input into a preset deep network structure for training to obtain a first foot process recognition model; wherein, the foot process recognition model includes: a first foot process recognition model; When acquiring electron microscopic images of chronic kidney disease, the foot process region of the electron microscopic image is identified by the foot process recognition model.

2. The method for identifying foot processes in electron micrographs of chronic kidney disease according to claim 1, characterized in that, The step of obtaining the decoding result corresponding to each sample image based on the plurality of feature maps, through a preset scale judgment operation and a preset decoder, specifically involves: The feature maps are input into a preset decoder for pre-decoding to obtain the pre-decoded feature scale of the feature maps. A preset scale judgment operation is performed on the pre-decoding feature scales of several feature maps respectively to obtain several first decoding features, several second decoding features and several third decoding features for each sample image; Through channel splicing operations, several first decoding features, several second decoding features, and several third decoding features are spliced ​​together to form the decoding result of each sample image.

3. The method for identifying foot processes in electron micrographs of chronic kidney disease according to claim 1, characterized in that, The step of training a pre-defined deep network architecture using several of the aforementioned electron microscopic pathological images of chronic kidney disease to obtain a foot process recognition model further includes: After obtaining several feature maps corresponding to each sample image, a preset depth feature map is selected for multi-scale fusion operation to obtain a depth feature map corresponding to each sample image. The depth feature map is input into a preset attention mechanism, and after preset image processing operations, a prediction map corresponding to each sample image is obtained. The predicted images corresponding to several of the sample images are input into a preset deep network structure for training to obtain a second foot protrusion recognition model; wherein, the foot protrusion recognition model includes: a second foot protrusion recognition model.

4. The method for identifying foot processes in electron micrographs of chronic kidney disease according to claim 1, characterized in that, After obtaining the decoding result corresponding to each sample image, the method further includes processing the decoding result using the argmax function.

5. The method for identifying foot processes in electron micrographs of chronic kidney disease according to claim 1, characterized in that, After obtaining the foot process recognition model, the method further includes: Obtain labeled images of the electron microscopy images of chronic kidney disease patients; The accuracy of foot protrusion recognition is obtained by calculating the accuracy of the predicted and labeled images of the foot protrusion output by the foot protrusion recognition model using the Dice coefficient. The foot protrusion recognition model is evaluated using the foot protrusion recognition accuracy.

6. A foot process recognition device for electron microscopic images of chronic kidney disease, characterized in that, include: Data acquisition module, model training module, and model application module; The data acquisition module is used to acquire several electron microscopic pathological images of chronic kidney disease; wherein, the electron microscopic pathological images of chronic kidney disease are images with foot process location features already marked; The model training module is used to train a preset deep network architecture using several electron microscopic pathological images of chronic kidney disease to obtain a foot process recognition model. The preset deep network architecture includes a preset encoder, a preset decoder, and a preset deep network structure. Several sample images are extracted from the several electron microscopic pathological images of chronic kidney disease according to a preset number of samples. Each sample image is input into the preset encoder for downsampling to obtain several feature maps corresponding to each sample image. Based on the feature maps, a preset scale judgment operation and a preset decoder are used to obtain the decoding result corresponding to each sample image. The preset scale judgment operation includes: when the scale of the pre-decoded feature is equal to the scale of the feature map, the feature map corresponding to the pre-decoded feature scale is considered a feature at the same layer, and the feature at the same layer is input into the preset decoder. The system performs decoding in a pre-decoding device to obtain the first decoded feature corresponding to the same layer feature. When the scale of the pre-decoding feature is smaller than the scale of the feature map, the feature map corresponding to the pre-decoding feature scale is a shallow feature. The shallow feature is downsampled, and the shallow feature after the downsampling operation is input into a preset decoder for decoding to obtain the second decoded feature corresponding to the shallow feature. When the scale of the pre-decoding feature is larger than the scale of the feature map, the feature map corresponding to the pre-decoding feature scale is a deep feature. The deep feature is upsampled, and the deep feature after the upsampling operation is input into a preset decoder for decoding to obtain the third decoded feature corresponding to the deep feature. The decoding results corresponding to several sample images are input into a preset deep network structure for training to obtain a first foot process recognition model. The foot process recognition model includes: a first foot process recognition model. The model application module is used to identify the foot process region in the electron microscope image of chronic kidney disease by means of the foot process recognition model.

7. A computer terminal device, characterized in that, The device includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements a method for identifying foot processes in electron micrographs of chronic kidney disease as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform a method for identifying foot processes in electron micrographs of chronic kidney disease as described in any one of claims 1 to 5.

Images