An open set recognition method for urinary red blood cells, white blood cells and calcium oxalate crystals

Abstract

The application discloses an open set recognition method for urine red blood cells, white blood cells and calcium oxalate crystals. The method comprises the following steps: collecting images of urine red blood cells, white blood cells, calcium oxalate crystals and similar impurities under a microscope, and constituting a data set after professional physician labeling; performing a combined data enhancement operation on the data set, and completing data normalization; secondly, constructing a urine sediment open set recognition network based on a conditional variational autoencoder; setting a multi-constraint optimization objective function and network training parameters; then, obtaining a pre-training model, and further migrating the pre-training model to the urine sediment data set for training; finally, testing the model classification performance, and calculating the recognition accuracy. The application is mainly applied to the field of medical images and urine formed element testing scenes, and has high accuracy and good applicability.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of medical image processing, and particularly relates to an open set recognition method for red blood cells, white blood cells and calcium oxalate crystals in urine. BACKGROUND

[0002] Urine sediment examination is to use a microscope to check the sediment (formed elements) of urine after centrifugation, and to detect and count formed elements such as red blood cells, white blood cells, crystal components, epithelial cells, etc. Urine sediment detection is one of the routine detection items in hospitals, and has an important diagnostic role for kidney diseases, infectious diseases and urinary system diseases. The mainstream urine sediment examination methods at present include manual microscopy and automated examination methods. Manual microscopy mainly includes direct microscopy, centrifugal sedimentation and Sternheimer-Malbin staining method, and automated examination methods mainly include image analysis method, flow analysis method and dry chemical analysis method. Since the traditional manual microscopy method is tedious to operate and requires high professional quality of personnel, the urine sediment examination method based on automation can greatly reduce the workload of the examining physicians, and is more standardized and normalized.

[0003] In the examination of formed elements in urine, the examination of red blood cells, white blood cells, calcium oxalate crystals and their similar objects has important significance. The detection of different components in urine samples represents different types of diseases, which can provide important reference for clinical diagnosis and treatment. For example, an increase in the number of red blood cells in urine can be initially judged as glomerulonephritis, acute cystitis, IgA nephropathy, stone or tumor, etc.; if white blood cells are found in urine, it indicates that the patient may have a urinary system infection disease; if urine contains a large amount of calcium oxalate crystals, accompanied by red blood cells, it may be a sign of kidney stones or bladder stones. Under normal circumstances, in addition to effective components, urine formed elements also contain a large amount of impurities, and part of the impurities have certain similarity in shape and size with effective components, which brings great challenges to manual microscopy and automated examination.

[0004] With the development of artificial intelligence related technologies, segmentation, detection, recognition and other technologies based on deep learning are widely used in the field of medical images. At present, the general steps of automatic detection of urinary sediment are sample acquisition, centrifugal precipitation, microscopic image acquisition, image segmentation, feature extraction, identification and classification. In feature extraction and identification classification, AlexNet, VGGNet, ResNet and other series of deep networks are usually used for feature extraction, and then a classifier is connected for classification, which generally requires a large amount of labeled data to train the neural network. The traditional deep learning method based on batch training usually assumes that the training sample data is independent and identically distributed, and considers that the data sample library has completeness. However, an obvious fact is that the data set used for training the network is difficult to meet the data completeness of the real scene. In the real environment, a large number of impurities are contained in the urinary sediment, and it is relatively tedious and redundant to collect these impurities, and the classifier that has never seen the impurities is usually easy to misclassify them as known effective components, which greatly affects the recognition accuracy of urinary formed elements, thereby affecting the judgment of the patient's condition by the physician. The existing urinary sediment recognition and classification method is mostly based on a closed set model, that is, it is assumed that all categories have been seen during training, and the test category is consistent with the training set category, which is easy to misjudge the impurities that have not appeared in the training set when processing real urine sample data. In view of this, an open set recognition method is needed to solve the classification and recognition problem in an open scene, which mainly includes two subtasks: known class recognition and unknown class rejection, which only contacts a certain category of known class in the training stage, and includes known class and multiple unknown classes in the test stage. At present, in the field of open set recognition, methods such as OpenMax, G-OpenMax have been used, which use EVT extreme value theory to model the features, and change the closed set classifier softmax in the back end into a classifier with open set capability. In addition, variational autoencoder is also introduced into the field of open set recognition, which encodes the input image to obtain the hidden layer representation, and then decodes to reconstruct the image. By constraining the error loss of the reconstructed image, the feature expression ability of the encoder to the input image is improved, so that the hidden layer features can finely express the input, and when encountering unknown class samples, they are rejected because of their large reconstruction error. In view of the problem that the closed set model misclassifies urinary sediment impurities as effective components, the method based on improved conditional variational autoencoder proposed in the present application has the ability to reject unknown impurities while identifying red blood cells, white blood cells and calcium oxalate crystals. SUMMARY

[0005] The application aims to provide an open set recognition method for urine red blood cells, white blood cells and calcium oxalate crystals, and improve the classification accuracy through some improvement measures, so that it is better adapted to the real open scene. The method mainly classifies red blood cells, white blood cells, monohydrate calcium oxalate and dihydrate calcium oxalate in urine formed elements, and can be further extended to other formed elements. The network structure is named Urine_VAE. The experimental results prove the effectiveness of the method, and the method has high accuracy for urine red blood cells, white blood cells and calcium oxalate crystals, and good applicability in unknown class impurity recognition.

[0006] To achieve the above-mentioned purpose, the technical solution adopted by the application is:

[0007] An open set recognition method for urine red blood cells, white blood cells and calcium oxalate crystals, comprising the following steps:

[0008] Step 1: Collecting urine formed element images and constructing a data set;

[0009] The urine sample after centrifugal precipitation is collected by a microscope for image acquisition, and is segmented into images containing single targets. After being labeled by professional physicians, the training set, the validation set and the test set are divided according to the ratio of 5:1:1.

[0010] The urine formed elements include red blood cells (RBC), white blood cells (WBC), monohydrate calcium oxalate (CAOT), dihydrate calcium oxalate (CAOX) and other 3 kinds of impurities similar to effective components (including amorphous crystal (AMOR), bubble (QPQP), and leukocyte-like (ZWBC)). Finally, the formed elements are 7 kinds in total, of which 4 kinds of effective components are used for training, and 7 kinds of components are tested in the test stage to test the open set recognition ability of the model.

[0011] Step 2: Data set preprocessing;

[0012] In order to improve the model recognition ability and enhance its generalization, the training set is subjected to combined data enhancement operation. Since the targets in the urine sediment image have the characteristics of arbitrary direction and diversified angle, horizontal and vertical flipping will not change the semantic features. Therefore, the training set is mainly subjected to horizontal flipping, vertical flipping and random angle rotation.

[0013] The picture size of the formed elements such as red blood cells, white blood cells and calcium oxalate crystals in the training set is between 16-56, and most of them are between 20-36. Therefore, the network input image size is set to 32*32. Finally, the data set is subjected to normalization processing.

[0014] Step 3: Constructing an open set recognition network Urine_VAE for urine formed elements;

[0015] The network adopts a conditional variational autoencoder architecture, and the network structure mainly includes four parts: an encoder E, a decoder D, a known class classifier C, and an unknown class discriminator U. The encoder E extracts features of the input image and maps them to a hidden layer space to obtain a hidden layer representation z based on the original image. The decoder D reconstructs the original image according to the hidden layer representation z and the label true value y. The known class classifier C uses a common softmax layer to classify the hidden layer representation z. The unknown class discriminator U is modeled according to the potential information in the hidden layer representation and the reconstruction error, and is used as a binary classifier to determine whether the test sample is a known class or an unknown class. According to the image size distribution described in step 2, the network input size is set to 32x32.

[0016] Step 4: Set the multi-constraint optimization objective function and network parameters;

[0017] The input of the conditional variational autoencoder network is a single target urine sediment image subjected to combined data enhancement. The network batch learning size is 64, the iteration number is 100 epochs, the learning rate is initialized to 0.001, the identification category is 4 categories, and the data set uses the data set described in step 1. The network loss function includes three parts:

[0018] L total =-(L rec +β·KL+λ·L ce )

[0019] In the formula, the optimization objective function can be divided into three parts, which are reconstruction loss L rec , KL divergence and classification loss L ce . The reconstruction loss is used to measure the difference between the image reconstructed by the decoder according to the hidden layer representation and the original image. The KL divergence is used to measure the difference between the sampling distribution and the prior Gaussian distribution. The classification loss uses the cross-entropy loss function. In the formula, β and λ are empirical coefficients, which are respectively taken as 5 and 100 according to experiments.

[0020] Step 5: Obtain a pre-trained model and migrate it to the urine sediment data set for further training;

[0021] The conditional variational autoencoder network is usually trained from scratch, and the present application uses the transfer learning paradigm to further improve the classification performance of the model. According to the network structure and parameters set in steps 3 and 4, a pre-trained model is obtained by training the CIFAR10 public data set. The training operating system is windows10, the pytorch deep framework is used, and the GPU is GeForce RTX 3060.

[0022] Using the urine sediment data set constructed in step 1, according to steps 3, 4, respectively set network structure and hyperparameters, load the pre-trained model trained on CIFAR10 in step 5 to continue training for 100 epochs until the model converges, realizing the migration of the source domain to the target domain.

[0023] Step 6: Model testing, calculate the recognition accuracy;

[0024] Load the model trained and saved in step 5, test the trained model using the urine sediment data test set obtained in step 1, and calculate the recognition results of red blood cells, white blood cells, calcium oxalate monohydrate, and calcium oxalate dihydrate in the test set and the overall recognition accuracy.

[0025] Further, the specific implementation method of step 2 is:

[0026] First, considering the characteristics of red blood cells, white blood cells, and calcium oxalate crystals under a microscope, such as multiple angles and multiple shapes, the collected urine sediment images are uniformly adjusted to 32x32 according to their size distribution.

[0027] Second, the data set is sequentially subjected to random horizontal flipping and random vertical flipping operations, with a random flipping probability p = 0.5.

[0028] Then, the data set is subjected to a random angle rotation operation, with a random rotation angle of 60° and a random rotation probability p = 0.5.

[0029] Finally, the urine sediment images are normalized according to the zero-mean normalization formula, and all the urine sediment images subjected to horizontal flipping, vertical flipping, random angle rotation, and normalization are divided into model training and testing data sets.

[0030] Further, the specific implementation method of step 3 is:

[0031] The network structure Urine_VAE adopts a variational auto-encoding structure and a conditional Gaussian distribution learning framework, and mainly includes an encoder E, a decoder D, a known class classifier C and an unknown class discriminator U. The encoder part uses an improved VGG network, including 10 convolutional blocks. Each convolutional block contains a convolutional layer, a flattening layer, a mean calculation fully connected layer and a softplus layer. The flattening layer is used to flatten the output of the convolutional layer into a one-dimensional feature vector, and the mean value is calculated through the mean fully connected layer, and the variance is calculated through the softplus layer. The output of the 10th convolutional block in the encoding stage is used to calculate the hidden layer representation z, which is sampled from the Gaussian distribution output by the encoder. The decoder D also contains 10 deconvolutional blocks, which takes the hidden layer representation z as input and outputs the reconstructed image through continuous deconvolution and other operations. Each deconvolutional block contains an inverse flattening layer, a deconvolutional layer, a flattening layer, a mean calculation fully connected layer and a variance calculation fully connected layer. The known class classifier C is composed of a conventional softmax layer, which takes the hidden layer representation z as input and outputs the predicted probability distribution. The unknown class detector U is modeled by the hidden layer representation z and the reconstruction error of the training sample, and is used as a binary classifier to judge whether the test sample is a known class or an unknown class. After the training process is completed, the feature and reconstruction error of the training sample corresponding to the minimum total loss of the validation set are saved, and the mean and variance of the feature of each urinary sediment class are calculated, so as to construct a Gaussian distribution conforming to each class.

[0032] Further, the specific implementation method of step 4 is:

[0033] Firstly, the model training hyperparameters are set. The initial learning rate is 0.001, the learning rate reduction mechanism is set, the training times are 100 epochs, the batch size is set to 64, and the training classes are set to 4 classes (RBC, WBC, CAOT, CAOX).

[0034] Secondly, the multi-constraint optimization objective function is set as:

[0035] L total =-(L rec +β·KL+λ·L ce )

[0036] Wherein, L rec is the reconstruction loss, which is used to measure the difference between the output image of the decoder D and the original image. In order to improve the semantic similarity between the two images, perceptual loss is used as the reconstruction loss. Specifically, the original image and the reconstructed pseudo image are input into the VGG network, and the feature maps output by the network at the first, second and third stages are compared to measure the semantic similarity. The perceptual loss is calculated as follows:

[0037] Lrec = m1·L feat1 + m2·L feat2 + m3·L feat3

[0038] where m1, m2, m3 represent the feature map loss weight of each stage, respectively set to 0.3, 0.3, 0.4. L feat represents the single feature map loss, and the calculation formula is as follows:

[0039]

[0040] wherein represents the reconstructed pseudo image, y represents the original input image, and φ j (j) represents the feature map of the jth stage, C j , H j , W j respectively represent the channel number, height and width value of the jth feature map.

[0041] During the training process, the model makes the conditional posterior distribution q φ (z|x, k) approximate different multivariate Gaussian models where k is the known class index, and the mean of the kth Gaussian distribution is calculated by the mean full connection layer described in step 3. The KL divergence of the latent space is used to measure the difference between the conditional posterior distribution and the approximate prior. The KL divergence calculation process is as follows:

[0042]

[0043] wherein D KL represents the KL divergence, z is the hidden layer representation of the encoding output, and k is the kth known class. x is the input sample, and q φ (z|x, k) represents the conditional posterior distribution of the known input sample and the class label, is a multivariate Gaussian model. μ is the mean, and σ 2 is the variance, and I represents the unit matrix.

[0044] wherein the classification loss L ce adopts cross-entropy loss calculation, and the cross-entropy loss function is as follows:

[0045]

[0046] According to experimental experience, the weight coefficient of the KL divergence is set to 5, and the weight coefficient of the classification loss L ce is set to 100. The loss function is iteratively optimized by the SGD gradient descent algorithm. When training, the arrangement order of all samples in the training sample set and the validation set is shuffled, and they are sent into the defined network model.

[0047] Further, the specific implementation method of step 5 is:

[0048] The network defined in steps 3 and 4 is trained on the CIFAR10 public data set, the number of known categories is set to 4, the epoch is set to 100, the initial learning rate is set to 0.001, and the batchsize is set to 64. The pre-trained model obtained after training is used for further migration.

[0049] Further, the specific implementation method of step 6 is:

[0050] The 7 categories (red blood cells RBC, white blood cells WBC, calcium oxalate CAOT, calcium oxalate CAOX, amorphous crystal AMOR, bubble QPQP, and leukocyte-like ZWBC) of the test set are identified and classified, and the precision, recall and F1-score are used to measure the classification accuracy. The precision is calculated as follows:

[0051]

[0052] The recall is calculated as follows:

[0053]

[0054] The F1-score is calculated as follows:

[0055]

[0056] The present application has the following beneficial effects compared with the prior art:

[0057] The present application provides an open set recognition method for red blood cells, white blood cells and calcium oxalate crystals based on an improved conditional variational autoencoder architecture, which uses multiple constraint methods to make the model learn more discriminative features in the effective components of urine, and uses transfer learning technology to improve the generalization of the model. It can effectively solve the problem of existing urine effective component closed set recognition model, realize known category classification and reject impurities, better cope with real open scenarios, and has certain practical value. DETAILED DESCRIPTION

[0058] Figure 1 The flowchart for the implementation of the present application is shown in the figure;

[0059] Figure 2 Some urine sediment sample images are shown in the figure;

[0060] Figure 3 The urine sediment open set recognition network is shown in the figure;

[0061] Figure 4 The perception loss calculation graph is shown in the figure. DETAILED DESCRIPTION

[0062] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below in combination with the drawings and specific embodiments.

[0063] As shown in the drawings, Figure 1 The open set recognition method for urinary red blood cells, white blood cells and calcium oxalate crystals according to the present application mainly comprises the following steps:

[0064] Step 1: Collecting urine formed element images and constructing a data set;

[0065] The urine sample after centrifugal precipitation is subjected to image acquisition by a microscope and is segmented into images containing single targets. After being labeled by a professional physician, the training set, the validation set and the test set are divided according to the ratio of 5:1:1.

[0066] As shown in the drawings, Figure 2 Part of the sample images of the constructed data set is given. The urine formed elements include red blood cells (RBC), white blood cells (WBC), calcium oxalate monohydrate (CAOT), calcium oxalate dihydrate (CAOX) and other 3 types of impurities similar to the effective components (including amorphous crystals (AMOR), bubbles (QPQP) and leukocyte-like cells (ZWBC)). Finally, a total of 7 types of formed elements are obtained, of which 4 types of effective components are used for training, and 7 types of components are tested in the test stage to test the open set recognition ability of the model. The number of each type in the training set is 5000, and the number of each type in the test set is 1000.

[0067] Step 2: Data set preprocessing;

[0068] By analyzing the scale distribution and angle distribution of the collected urine formed element images, the size of all images in the data set is adjusted to 32x32. In order to make the model have better generalization and prevent overfitting, the combined data augmentation operation is performed on the data set. Specifically, since the collected urine formed element images have the characteristics of arbitrary direction and arbitrary angle, the following operations are performed:

[0069] Random horizontal flip: the image is subjected to horizontal mirror flip operation to change the original direction of the target in the image, and the random flip probability is p=0.5;

[0070] Random vertical flip: the urine sediment image is subjected to vertical mirror flip, which does not change its semantic features, and the random probability is p=0.5.

[0071] Random angle rotation: the image is rotated by a certain angle to change the direction of the target, and the random probability is p=0.5.

[0072] Finally, the enhanced image is subjected to mean normalization, and the normalization formula is as follows:

[0073]

[0074] wherein X i is the i th sample data, μ is the mean of the overall sample data, σ is the standard deviation of the overall sample data, Y i is the sample data obtained after normalization.

[0075] Step 3: constructing a urine formed element open set recognition network Urine_VAE;

[0076] Referring to Figure 3 , the network structure adopts a variational auto-encoding network framework, mainly including an encoder E, a decoder D, a known class classifier C and an unknown class detector U four parts. The encoder part uses an improved VGG network, including 10 convolutional blocks. Each convolutional block contains a convolutional layer, a flattening layer, a mean calculation fully connected layer and a softplus layer. The flattening layer is used to flatten the output of the convolutional layer into a one-dimensional feature vector, and the corresponding mean is calculated through the mean fully connected layer, and the variance is further calculated through the softplus layer. The output of the 10 th convolutional block in the encoding stage is calculated as z = μ + σ ⊙ ε, and ε ~ N(0, I). The decoder D is symmetrical to the encoder E and also contains 10 deconvolutional blocks. D takes the hidden layer representation z as input and outputs the reconstructed image through continuous deconvolution operations. Each deconvolutional block contains inverse flattening, a deconvolutional layer, a flattening layer, a mean calculation fully connected layer and a variance calculation fully connected layer. The known class classifier C is composed of a conventional softmax layer, which takes the hidden layer representation z as input and outputs the predicted probability distribution. The unknown class detector U is modeled by the hidden layer representation z and the reconstruction error of the training sample, and is used as a binary classifier to judge whether the test sample is a known class or an unknown class. After the training process is completed, the model saves the training sample features and reconstruction errors corresponding to the minimum total loss of the validation set, and calculates the mean and variance of the features of each class, so as to construct a Gaussian distribution conforming to each class.

[0077] Step 4: setting a multi-constraint optimization objective function and network parameters;

[0078] The urine sediment open set recognition network adopts a multi-constraint optimization objective function for training, and the total loss of the network is as follows.

[0079] L total = -(L rec + β·KL+ λ·L ce )

[0080] wherein L recdenotes the reconstruction loss, which is used to measure the error between the output image of the decoder D and the original image. Figure 4 As shown in FIG. 1, the present application adopts the perceptual loss as the reconstruction loss in the total loss. The perceptual loss focuses on measuring the similarity between the original image and the reconstructed image at the semantic level. Specifically, the original image X is input into the network, and the reconstructed image is obtained through the process of encoding, decoding, etc. Then the original image X and the reconstructed image are input into the same feature extraction network VGG. The perceptual loss is calculated based on the output feature maps of the three stages of the feature extraction network, so as to measure the feature difference of the three stages and realize the measurement at the semantic similarity level. According to experimental experience, the present application selects the output feature maps of the first three stages of the VGG network and assigns different weights to them. The total loss of the feature maps is:

[0081] rec L feat1 = m1·L feat2 + m2·L feat3 + m3·L

[0082] wherein m1, m2, m3 represent the loss weights of the feature maps of the respective stages, which are respectively set as 0.3, 0.3, 0.4. L feat denotes the loss of a single feature map, and the calculation formula is as follows:

[0083]

[0084] wherein denotes the reconstructed pseudo image, y denotes the original input image, and φ j (·) denotes the feature map of the jth stage, C j , H j , and W j respectively represent the channel number, height and width values of the jth feature map.

[0085] During the training process, the model makes the conditional posterior distribution q φ (z|x,k) approximate different Gaussian models wherein k is the category index. The mean and variance of the Gaussian distribution of the kth category are calculated in step 3. The KL divergence of the latent space is used to measure the difference between the conditional posterior distribution and the approximate prior. The calculation process of the KL divergence is as follows:

[0086]

[0087] wherein D KL denotes the KL divergence, z is the latent layer representation of the encoding output, k is the kth known category. x is the input sample, and q φ (z|x,k) denotes the conditional posterior distribution of the known input sample and the category label, is a multivariate Gaussian model. μ is the mean, σ 2 is the variance, and I represents the identity matrix.

[0088] where the classification loss is calculated by cross-entropy loss. The cross-entropy loss is as follows:

[0089]

[0090] In the total loss function, according to actual experience, the coefficient of the KL divergence loss is 5, and the weight coefficient of the classification loss is 100. The loss function is iteratively optimized by the SGD gradient descent algorithm. When training, the arrangement order of all samples in the training sample set and the validation set is shuffled, and they are sent into the defined network model.

[0091] Step 5: Obtain a pre-trained model and migrate to the urine sediment data set for further training;

[0092] The above network is usually trained by learning from scratch. In order to further improve the classification performance of the model, the present application is trained by using the transfer learning paradigm. First, the network is trained on the public data set CIFAR10, and the visible categories are set to 4 categories and the unknown categories are set to 6 categories. Combined with the network structure and parameters set in steps 3 and 4, a pre-trained model is obtained by training for 100 epochs. The training operating system is windows10, the deep learning framework is pytorch1.8, and the GPU is GeForce RTX 3060.

[0093] Using the urine sediment data set constructed in step 1, the network structure and hyperparameters are set according to steps 3 and 4, respectively, and the pre-trained model obtained in step 5 is loaded to continue training for 100 epochs until the model converges, realizing the transfer from the source domain to the target domain.

[0094] Step 6: Model testing, calculate the recognition accuracy;

[0095] After training, the training sample features and reconstruction errors corresponding to the epoch with the minimum total loss value of the validation set are saved. For each known urine sediment category k, the mean and variance m k = mean i (z i,k ), where z i,k is the hidden layer representation of each correctly classified known class training sample x i,k . The hidden layer representation z = (z1, z2,..., z n ), n is the dimension, which is set to 32. Further, a multivariate Gaussian distribution based on each category k is constructed For each test sample Xt , and the encoded Z = E(X t ) is input into the known class classifier to obtain its possible class label y pred = argmax(C(Z)). The Z is input into the decoder to obtain the reconstructed image Calculate the reconstruction error For each modeled multivariate Gaussian distribution of a class, the probability of a sample falling in the distribution is:

[0096]

[0097] For any known urine sediment class (RBC, WBC, CAOT, CAOX), the probability P k (z) of a test sample falling in each class is calculated. If P k (z) < τ d or the test sample reconstruction error R rec > τ rec , the test sample is identified as an unknown class; otherwise, the label obtained by the class classifier C in the previous step is retained. Wherein, τ d is the threshold of the conditional Gaussian distribution, set to 0.5, and τ rec is the reconstruction error threshold, to ensure that 95% of the training data can be correctly classified into known classes.

[0098] Load the model trained and saved in step 5, test the trained model using the urine sediment data test set obtained in step 1, and calculate the recognition results of red blood cells (RBC), white blood cells (WBC), calcium oxalate monohydrate (CAOT), calcium oxalate dihydrate (CAOX), amorphous crystals (AMOR), bubbles (QPQP), and white blood cell-like (ZWBC) in the test set, and the overall recognition accuracy, wherein AMOR, QPQP and ZWBC are considered as a class; precision, recall and F1-score indicators are used to measure the classification performance during testing. Wherein, precision is calculated as follows:

[0099]

[0100] recall is calculated as follows:

[0101]

[0102] F1-score is calculated as follows:

[0103]

[0104] According to the above steps, the test results for red blood cells (RBC), white blood cells (WBC), calcium oxalate monohydrate (CAOT), calcium oxalate dihydrate (CAOX), amorphous crystals (AMOR), air bubbles (QPQP), and leukocyte-like cells (ZWBC) are as follows:

[0105]

[0106]

[0107] The foregoing merely illustrates the principles of the application and application, and is not to be taken in a limiting sense. Those skilled in the art will readily recognize various modifications of the examples that can be made without departing from the spirit and scope of the application.

Claims

1. An open set recognition method of urinary red blood cells, white blood cells and calcium oxalate crystals, characterized by, The method comprises the following steps: Step 1: collect urine formed element images and build a data set; Step 2: data set preprocessing; Step 3: build a urine formed element open set identification network Urine_VAE; The urine formed element separation set recognition network Urine_VAE takes a conditional variational auto-encoding network as a main framework, and includes four parts: an encoder E, a decoder D, a known category classifier C and an unknown category detector U; wherein the encoder E uses an improved VGG network, and includes 10 convolution blocks, each of which contains a convolution layer, a flatten layer, a mean calculation full connection layer and a softplus layer; the flatten layer is used to flatten the convolution output feature map into a one-dimensional feature vector, the mean calculation full connection layer is used to calculate the corresponding feature mean, and the softplus layer is used to calculate the sample variance; the output of the last convolution block in the encoding stage is used to calculate the hidden layer representation , , is the sample data overall mean, is the sample data overall standard deviation; the decoder D is symmetrical to the encoder E, and also includes 10 deconvolution blocks; the D takes the hidden layer representation as input, and outputs a reconstructed image through continuous deconvolution operations; wherein each deconvolution block contains an inverse flatten layer, a deconvolution layer, a flatten layer, a mean calculation full connection layer and a softplus layer; the known category classifier is composed of a conventional softmax layer, takes the hidden layer representation as input, and outputs a predicted category probability distribution; the unknown category detector U is composed of the hidden layer representation and the reconstruction error of the training sample, is modeled, and is used as a binary classifier to judge whether a test sample belongs to a known category or an unknown category; specifically, after the training is completed, the model saves the training sample features and the reconstruction error corresponding to the minimum total loss of the verification set, calculates the feature mean and variance of each category, and constructs a Gaussian distribution conforming to each known category. Step 4: set a multi-constraint optimization objective function and network training parameters; Step 5: obtain a pre-trained model and migrate to the urine sediment data set for further training; Step 6: model testing and calculation of identification accuracy.

2. The open set identification method of red blood cells, white blood cells and calcium oxalate crystals in urine according to claim 1, characterized in that: In step 1, the collected urine formed element images are segmented into images containing only single targets after being collected by a microscope, and are labeled by professional physicians, and are divided into a training set, a validation set and a test set according to a ratio of 5:1:1; the categories are red blood cells RBC, white blood cells WBC, calcium oxalate monohydrate crystals CAOT, calcium oxalate dihydrate crystals CAOX and 3 types of impurities similar to formed elements, the 3 types of impurities similar to formed elements including amorphous crystals AMOR, bubbles QPQP and white blood cell-like ZWBC; the data set contains 7 types of formed elements in total, of which 4 types are known classes and 3 types are unknown classes.

3. The open set identification method of red blood cells, white blood cells and calcium oxalate crystals in urine according to claim 1, characterized in that: Step 2 is specifically: performing a combined data enhancement operation on the urine sediment open set data set built in step 1, including random horizontal flipping, random vertical flipping and random angle rotation, and the random flipping probability is set to 0.5; the image size in the training set is unified to 32x32, and finally, the data set after completion of the enhancement is subjected to a normalization operation, and the formula is as follows: In the formula, is the first sample data, is the normalized sample data.

4. The open set identification method of red blood cells, white blood cells and calcium oxalate crystals in urine according to claim 1, characterized in that: The multi-constraint optimization objective function in step 4 is as follows: The total loss of the network includes three parts: the reconstruction loss, the KL divergence and the classification loss. The reconstruction loss, the KL divergence and the classification loss are represented by According to the experiment, The value of is 5, The value of is 100; the perceptual loss is used as the reconstruction loss, which focuses on measuring the similarity between the original image and the reconstructed image at the semantic level; the KL divergence is used to measure the difference between two distributions, and in the training process, the model makes the conditional posterior distribution Approximate different Gaussian models Wherein, is the category index; the calculation formula of the KL divergence is as follows: where, denotes the KL divergence, is the hidden layer representation for the encoded output, k is the class index; is the input urine sediment sample, denotes the conditional posterior distribution of known input samples and class labels, is a multivariate Gaussian model; is the mean, is the variance, denotes the identity matrix; the classification loss The softmax cross-entropy loss is calculated as follows: 。 5. The open set identification method of red blood cells, white blood cells and calcium oxalate crystals in urine according to claim 1, characterized in that: The reconstruction loss in step 4 Adopt perceptual loss; specifically, the original input image and the pseudo image reconstructed by the decoding output are input into the same VGG network, the feature maps output by the first three stages of the VGG network are selected for comparison, the similarity between the feature maps is calculated, and the weights of the three-stage feature similarities are respectively set to 0.3, 0.3, and 0.4; the perceptual reconstruction loss is calculated as follows: wherein , , denote the feature map loss weight of each stage, respectively set to 0.3, 0.3, 0.

4. denote the single feature map loss, the calculation formula is as follows: wherein denotes the reconstructed pseudo image, denotes the original input image, denotes the feature map of the st stage, denotes the number of channels, height and width values of the st feature map, respectively.

6. The open set identification method of red blood cells, white blood cells and calcium oxalate crystals in urine according to claim 1, characterized in that: In step 5, the conditional variational auto-encoding network is trained in a scratch training mode, and in order to improve the classification accuracy of the model for urine formed elements, a transfer learning paradigm is used for training; first, the model is trained on the CIFAR10 public data set for 100 epochs, and the known categories are set to 4 categories and the unknown categories are set to 6 categories; then the pre-trained model is loaded onto the urine sediment open set data set built in step 1 for further training for 100 epochs until convergence.

7. The open set recognition method of red blood cells, white blood cells and calcium oxalate crystals in urine according to claim 1, characterized in that: The test set constructed in step 1 is used to test the model recognition accuracy in step 6, including 4 known classes: red blood cells RBC, white blood cells WBC, calcium oxalate CAOT, and calcium oxalate CAOX, and 3 unknown class samples: amorphous crystals AMOR, bubbles QPQP, and white blood cell-like ZWBC; the two-stage test method is adopted in the test process, that is, firstly, the hidden layer representation of the test sample is preliminarily classified by using the known class classifier, and then the test sample is reclassified by using the reconstruction error threshold and the probability threshold If the reconstruction error of the test sample is greater than or the probability of falling in the known class distribution is less than the threshold , it is determined as an unknown class; otherwise, the known class classification result is kept unchanged.

Images