Rare blood type identification method, medium and equipment based on adversarial transfer learning

Abstract

The present invention proposes a rare blood type identification method, medium and device based on adversarial transfer learning. First, the basic CNN model is improved by introducing an attention mechanism, adding multi-scale convolution and performing spatial pyramid pooling on the multi-scale convolution feature map, and the trained CNN model is used as the shared feature extraction part of the adversarial transfer learning model; then, a fully connected layer is added to the CNN model as a source domain classifier and a target domain classifier; secondly, considering the sparsity, category imbalance and feature complexity of rare blood type data, a comprehensive adversarial loss function is designed; finally, the adversarial transfer learning model is trained for the detection of rare blood types. The present invention can effectively extract blood type-related features and improve the accuracy of blood type recognition; only image recognition of blood type samples is required, and no serological experiments are required, which is simple to operate and easy to popularize; only a one-time blood type recognition model needs to be established, reducing the difficulty and cost of operation.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and specifically relates to a rare blood type identification method, medium and equipment based on adversarial transfer learning, which can be applied to clinical blood transfusion, organ transplantation and other fields. Background Art

[0002] ABO blood type is the earliest and most common blood type known to humans. Accurate blood type identification is an important guarantee for safe and effective blood transfusion. It also has important clinical significance for neonatal hemolytic disease detection and transplant matching. In addition to the ABO and Rh blood type systems, rare blood types such as B(A), CisAB and other ABO subtypes and Bombay-like blood types occasionally appear. Although the incidence is low, it brings clinical troubles to ABO blood typing and blood matching for such patients, and is prone to cause transfusion accidents. Therefore, it is particularly important to propose an accurate and effective method for identifying rare blood types.

[0003] Traditional blood typing mainly relies on serology, slide method, test tube centrifugation method, microcolumn gel method, etc., but there are limitations such as cumbersome, time-consuming, and requiring professional operation, and the identification of some special and rare blood types is not accurate enough. The subsequent gene sequencing method is costly and complex to operate, making it difficult to popularize. In recent years, many machine learning algorithms based on automatic recognition and classification of blood type images have emerged, such as support vector machines, decision trees, random forests, K nearest neighbors, and deep learning algorithms, which can improve the accuracy and efficiency of blood typing. However, these methods are supervised learning algorithms that require a large amount of labeled data for training. In the field of blood typing analysis, obtaining a large amount of accurate standard data is time-consuming and costly. Especially for some rare blood types, the training data is extremely limited, which makes it more difficult for the learning model to accurately identify and classify. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides a rare blood type identification method, medium and device based on adversarial transfer learning. Pre-training is performed based on a large amount of common blood type image data, and adversarial transfer learning is used to fine-tune the model based on very limited rare blood type image data, and a rare blood type identification method based on adversarial transfer learning is proposed.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] A rare blood type identification method based on adversarial transfer learning is characterized by comprising the following steps:

[0007] Collect image data of samples with known blood types and perform data preprocessing;

[0008] Constructing a CNN model, wherein the CNN model introduces an attention mechanism and multi-scale convolution, and performs spatial pyramid pooling on the multi-scale convolution feature map; training the CNN model based on the preprocessed image data, and using the trained CNN model as a generator of an adversarial transfer learning model for feature extraction;

[0009] Add source domain classifier and target domain classifier after the CNN model as adversarial discriminators of the adversarial transfer learning model to output the category probability distribution of source domain data and target domain data;

[0010] Design an adversarial loss function to minimize the distribution difference between source domain data and target domain data in the feature space;

[0011] Based on rare blood type data, an adversarial transfer learning model is trained. During the training process, the adversarial loss function is minimized to obtain a trained adversarial transfer learning model.

[0012] The trained adversarial transfer learning model is used to identify rare blood type samples, and the category probability distribution output by the target domain classifier is the rare blood type identification result.

[0013] To optimize the above technical solutions, the specific measures taken also include:

[0014] Furthermore, the data preprocessing includes data enhancement operation and normalization processing.

[0015] Furthermore, the basic architecture of the CNN model is an input layer, three convolutional layers and three maximum pooling layers arranged alternately, and one fully connected layer, wherein an attention mechanism is introduced between the first convolutional layer and the first maximum pooling layer, and multi-scale convolution and spatial pyramid pooling are introduced between the third maximum pooling layer and the fully connected layer.

[0016] The attention mechanism is composed of an average pooling layer, a fully connected layer, a ReLU activation function, a fully connected layer, and a softmax activation function in sequence, and the processing process is as follows:

[0017] The input feature map X is averaged and pooled to obtain the global feature vector z:

[0018] z = GlobalAvgPool(X);

[0019] In the formula, GlobalAvgPool represents the global average pooling operation;

[0020] Calculate the attention weight vector w:

[0021] w=softmax(W2·ReLU(W1·z));

[0022] Where W1 and W2 represent the neuron weights of the two fully connected layers in the attention mechanism respectively;

[0023] Finally, we get the attention weighted feature map X a , which is then input into the second convolutional layer:

[0024] X a =w⊙X.

[0025] Furthermore, the multi-convolution scale uses two convolution kernels of different sizes, k1 and k2, to convolve the output feature X of the third maximum pooling layer. c Perform multi-scale convolution operations to obtain two feature maps f1 and f2:

[0026]

[0027] Perform weighted fusion of the two feature maps f1 and f2:

[0028] f fusion1 =σ(α1·f1+α2·f2);

[0029] In the formula, f fusion1 represents the fused feature map, α1 and α2 represent the channel attention weights, and α1+α2=1.0, α1, α2≥0; σ is the Sigmoid activation function;

[0030] At the same time, the two feature maps f1 and f2 are spatially pyramid pooled:

[0031] f fusion2 =concat(f1, f2, pool(f1), pool(f2)));

[0032] In the formula, f fusion2 It represents the feature map after spatial pyramid pooling, concat represents the concatenation operation of the feature map, and pool represents the spatial pyramid pooling operation;

[0033] Finally, f fusion1 、f fusion2 Two feature maps are weighted fused:

[0034] f fusion =β·f fusion1 +(1-β)·f fusion2 ;

[0035] In the formula, f fusion represents the fused feature map, β represents f fusion1 The weight of .

[0036] Furthermore, both the source domain classifier and the target domain classifier are fully connected layers arranged after the fully connected layer of the CNN model.

[0037] Furthermore, the adversarial loss function is specifically:

[0038]

[0039] Where, L adv is a comprehensive adversarial loss function for adversarial transfer learning model training, L class is the class-weighted adversarial loss function for the class imbalance problem, L gen Generate adversarial loss function for samples with data scarcity problem, L feat is the feature space adversarial loss function for complex feature problems; λ1, λ2, and λ3 are the loss terms L class , L gen and L feat The influence weights of x are λ1, λ2, λ3∈[0,1] and λ1+λ2+λ3=1.0; i is the i-th input sample; N is the number of samples; w i is the category weight of the i-th sample; D(·) is the adversarial discriminator; D(x i ) is the output of the adversarial discriminator; G(x i ) is the adversarial sample generated by the generator; D(G(x i )) is the output of the adversarial discriminator for the generated sample; represents the mapping function from input data x to feature space; p src (x) is the source domain data distribution; p tgt (x) is the target domain data distribution; f(·) represents the Lipschitz continuous function; ||f||L represents the Lipschitz constant.

[0040] Furthermore, in the process of training the adversarial transfer learning model based on rare blood type data, if the change error of the adversarial loss function is less than 0.05, the training ends.

[0041] Accordingly, the present invention proposes a computer-readable storage medium storing a computer program, characterized in that the computer program enables a computer to execute the rare blood type identification method based on adversarial transfer learning as described above.

[0042] Accordingly, the present invention proposes an electronic device, characterized in that it includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the rare blood type identification method based on adversarial transfer learning as described above is implemented.

[0043] The beneficial effects of the present invention are: (1) The rare blood type identification method based on adversarial transfer learning proposed in the present invention can effectively extract blood type related features and improve the accuracy of blood type recognition; (2) The present invention only requires image recognition of blood type samples, and does not require serological experiments. The operation is simple and easy to popularize; (3) The present invention only requires a one-time establishment of a blood type recognition model, which reduces the difficulty and cost of operation and provides a guarantee for the safety of clinical blood transfusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a flow chart of the rare blood type identification method based on adversarial transfer learning proposed in the present invention.

[0045] Figure 2 This is a schematic diagram of the CNN basic model constructed by the present invention.

[0046] Figure 3 Schematic diagram of the adversarial transfer learning model designed for the present invention.

[0047] Figure 4 These are three performance comparison diagrams of the present invention and other mainstream blood typing algorithms. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0049] In one embodiment, the present invention proposes a rare blood type identification method based on adversarial transfer learning, such as Figure 1 As shown, the specific operation steps include the following.

[0050] Step 1: Collect a large amount of image data of known blood type samples, including various rare blood types. Use OpenCV to perform a series of data enhancements such as mirror flipping, rotation, scaling, translation, brightness adjustment, etc. on these image data, and perform normalization preprocessing operations to convert the data to the range of [0, 1].

[0051] Among them, x represents the original blood type sample image data, x max 、x min Represent the maximum and minimum values ​​of the image pixels respectively.

[0052] Step 2: If Figure 2 As shown in , first, a CNN basic model is constructed, which includes an input layer, three convolutional layers, three maximum pooling layers, and one fully connected layer. Figure 3As shown in the figure, an attention mechanism is introduced between the first convolutional layer and the pooling layer. The attention mechanism consists of an average pooling layer, a fully connected layer, a ReLU activation function, a fully connected layer, and a softmaxt activation function. The input feature map X is subjected to global average pooling to obtain a global feature vector z:

[0053] z = GlobalAvgPool(X); Calculate the attention weight vector w:

[0054] W=softmax(W2·ReLU(W1·z));

[0055] Among them, W1 and W2 represent Figure 3 Neuron weights of two fully connected layers in the attention mechanism.

[0056] Get the attention weighted feature map X a Input to the second convolutional layer:

[0057] X a =w⊙X.

[0058] Then, if Figure 3 As shown in Figure 2, after the last pooling layer, two convolution kernels of different sizes, k1 (3×3) and k2 (5×5), are used to pool the output feature X. c Perform multi-scale convolution operations to obtain two feature maps f1 and f2:

[0059]

[0060] Perform weighted fusion of the two feature maps:

[0061] f fusion1 =σ(α1·f1+α2·f2);

[0062] Among them, α1 and α2 represent channel attention weights, and α1+α2=1.0, α1, α2≥0; σ is the Sigmoid activation function. At the same time, the two feature maps are spatially pyramid pooled to capture multi-scale spatial information:

[0063] f fusion2 =concat(f1, f2, pool(f1), pool(f2)));

[0064] Among them, concat represents the concatenation operation of feature maps, and pool represents the spatial pyramid pooling operation.

[0065] f fusion1 、f fusion2 The two feature maps are weightedly fused to balance the importance of the two features, obtain a more comprehensive and rich feature representation, and improve the performance and generalization ability of the model in the task.

[0066] f fusion =β·f fusion1 +(1-β)·f fusion2 .

[0067] Step 3: The preprocessed blood type image dataset is randomly divided into a training set and a test set at a ratio of 4:1. The optimizer uses the Adam optimization algorithm and uses mini-batch training to iterate in batches based on the training set. Figure 2 For the CNN network shown, if the epoch is greater than the set threshold, proceed to step 4, otherwise proceed to step 3. The trained model is the shared feature extractor of the adversarial transfer learning model, i.e., the generator.

[0068] Step 4: If Figure 3 As shown in the figure, fully connected layers with similar structures are added after the fully connected layer of the trained CNN model as source domain classifiers and target domain classifiers, which serve as adversarial discriminators of the adversarial transfer learning model. The source domain classifier is trained based on the source domain data, and the adversarial loss function in step 5 makes the features extracted by the feature extractor difficult to distinguish between the source domain and the target domain, which helps the feature extractor to be more universal; the target domain classifier is trained using the features from the feature extractor, and its output is a softmax layer, which is used to output the category probability distribution of the target domain data, which is the final blood typing result.

[0069] Step 5: Considering the sparsity, class imbalance and feature complexity of rare blood type data, a comprehensive adversarial loss function L is designed. adv , which is used to minimize the difference between the source domain classifier and the target domain classifier, thereby reducing the difference between the distribution of source domain data and target domain data.

[0070]

[0071] Among them, λ1, λ2 and λ3 are the loss terms L closs , L gen and L feat The influence weights of L, λ1, λ2, λ2∈[0,1] and λ1+λ2+λ3=1.0; class L is a class-weighted adversarial loss function for class imbalance, which aims to balance the influence between different classes. gen Generate adversarial loss functions for samples targeting data scarcity problems, aiming to expand rare blood type datasets; L feat It is a feature space adversarial loss function for complex feature problems, which aims to minimize the distribution difference between source domain data and target domain data in the feature space; i is the i-th input sample; N is the number of samples; wi is the category weight of the i-th sample; D(·) is the adversarial discriminator; D(x i ) is the output of the adversarial discriminator; G(x i ) is the adversarial sample generated by the generator; D(G(x i )) is the output of the adversarial discriminator for the generated sample; represents the mapping function from input data x to feature space; p src (x) is the source domain data distribution; p tgt (x) is the target domain data distribution; f(·) represents the Lipschitz continuous function; ||f||L represents the Lipschitz constant.

[0072] Step 6: Select some (10 cases in this example) rare blood type (such as B (A), CisAB and other ABO subtypes) data sets, iteratively train the adversarial transfer learning model through adversarial training between feature extractors (generators) and adversarial discriminators, minimize the adversarial loss function, and improve the model's identification performance for rare blood types. If the loss function change error is less than 0.05, the training ends and proceeds to step 7; otherwise, proceed to step 6.

[0073] Step 7: Input the rare blood type test set to evaluate the blood type identification performance of the model.

[0074] Evaluation of the effect of the present invention: Accuracy, recall and precision are used as evaluation indicators to evaluate the performance of the proposed adversarial transfer learning model on rare blood type test samples.

[0075] Among them, TP, TN, FP, and FN are the number of true positives, false negatives, false positives, and false negatives, respectively. The larger the values ​​of the three indicators, the higher the accuracy of model identification.

[0076] The blood typing performance of the model of the present invention was compared with that of AlexNet, VGG16, blood typing serology test tube method, and PCR-SSP method as shown in Table 1. The three patients had no history of blood transfusion or transplant matching. To clarify their blood types, samples were drawn for further ABO blood typing gene testing. All three patients signed the "Informed Consent Form" for further testing.

[0077] Table 1 Test patient sample information

[0078] patient gender age disease ABO blood typing performance Patient 1 female 28 Pregnant woman (39+4 weeks) planning to undergo cesarean section Positive AB type, negative B type Patient 2 male 31 Complete rupture of the anterior cruciate ligament of the knee Positive AB type, negative B type Patient 3 male 58 Rectal malignancy Positive AB type, negative B type

[0079] After experiments, the results of the blood type serological test tube method showed that all three samples were B(A) blood type, and the PCR-SSP method determined that the phenotypes of the three samples were all B(A)02 / O1. Figure 3 This is a comparison of the accuracy, recall rate, and precision of blood typing between the present invention and two mainstream CNN methods, AlexNet and VGG16. It can be seen that the traditional blood typing serology test tube method and PCR-SSP method can accurately determine the type of rare blood types, but the steps are complicated and time-consuming; the accuracy of blood typing identification by the model proposed in the present invention is significantly improved compared with the two intelligent methods AlexNet and VGG16, reaching more than 80%, but due to the very limited training set of rare blood types, there is still a lot of room for improvement in the accuracy of the model.

[0080] In another embodiment, the present invention proposes a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the rare blood type identification method based on adversarial transfer learning as described in Example 1.

[0081] In another embodiment, the present invention proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for identifying rare blood types based on adversarial transfer learning as described in Example 1 is implemented.

[0082] In the embodiments disclosed in the present application, the computer storage medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. The computer storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. More specific examples of computer storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0083] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0084] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A rare blood type identification method based on adversarial transfer learning, characterized in that: The steps include: Collect image data of samples with known blood types and perform data preprocessing; Constructing a CNN model, wherein the CNN model introduces an attention mechanism and multi-scale convolution, and performs spatial pyramid pooling on the multi-scale convolution feature map; The CNN model is trained based on the preprocessed image data. The trained CNN model is used as the generator of the adversarial transfer learning model for feature extraction. Add source domain classifier and target domain classifier after the CNN model as adversarial discriminators of the adversarial transfer learning model to output the category probability distribution of source domain data and target domain data; Design an adversarial loss function to minimize the distribution difference between source domain data and target domain data in the feature space; Based on rare blood type data, an adversarial transfer learning model is trained. During the training process, the adversarial loss function is minimized to obtain a trained adversarial transfer learning model. The trained adversarial transfer learning model is used to identify rare blood type samples, and the category probability distribution output by the target domain classifier is the rare blood type identification result; The basic architecture of the CNN model is an input layer, three convolutional layers and three maximum pooling layers arranged alternately, and one fully connected layer. An attention mechanism is introduced between the first convolutional layer and the first maximum pooling layer, and multi-scale convolution and spatial pyramid pooling are introduced between the third maximum pooling layer and the fully connected layer. The attention mechanism is composed of an average pooling layer, a fully connected layer, a ReLU activation function, a fully connected layer, and a softmax activation function in sequence, and the processing process is as follows: The input feature map X is averaged and pooled to obtain the global feature vector z: z = GlobalAvgPool(X); In the formula, GlobalAvgPool represents the global average pooling operation; Calculate the attention weight vector w: w=softmax(W2·ReLU(W1·z)); Where W1 and W2 represent the neuron weights of the two fully connected layers in the attention mechanism respectively; Finally, we get the attention weighted feature map X a , which is then input into the second convolutional layer: X a =w⊙X; The multi-scale convolution uses two convolution kernels of different sizes, k1 and k2, to compute the output feature X of the third maximum pooling layer. c Perform multi-scale convolution operations to obtain two feature maps f1 and f2: Perform weighted fusion of the two feature maps f1 and f2: f fusion1 =σ(α1·f1+α2·f2); In the formula, f fusion1 represents the fused feature map, α1 and α2 represent the channel attention weights, and α1+α2=1.0, α1, α2≥0; σ is the Sigmoid activation function; At the same time, the two feature maps f1 and f2 are spatially pyramid pooled: f fusion2 =concat(f1,f2,pool(f1),pool(f2))); In the formula, f fusion2 It represents the feature map after spatial pyramid pooling, concat represents the concatenation operation of the feature map, and pool represents the spatial pyramid pooling operation; Finally, f fusion1 、f fusion2 Two feature maps are weighted fused: f fusion =β·f fusion1 +(1-β)·f fusion2 ; In the formula, f fusion represents the fused feature map, β represents f fusion1 The weight of The adversarial loss function is specifically: Where, L adv is a comprehensive adversarial loss function for adversarial transfer learning model training, L class is the class-weighted adversarial loss function for the class imbalance problem, L gen Generate adversarial loss function for samples with data scarcity problem, L feat is the feature space adversarial loss function for complex feature problems; λ1, λ2, and λ3 are the loss terms L class , L gen and L feat The influence weights of x are λ1,λ2,λ3∈[0,1] and λ1+λ2+λ3=1.0; i is the i-th input sample; N is the number of samples; w i is the category weight of the i-th sample; D(·) is the adversarial discriminator; D(x i ) is the output of the adversarial discriminator; G(x i ) is the adversarial sample generated by the generator; D(G(x i )) is the output of the adversarial discriminator for the generated sample; represents the mapping function from input data x to feature space; p src (x) is the source domain data distribution; p tgt (x) is the target domain data distribution; f(·) represents the Lipschitz continuous function; fL represents the Lipschitz constant.

2. The rare blood type identification method based on adversarial transfer learning as claimed in claim 1, characterized in that: The data preprocessing includes data enhancement operation and normalization processing.

3. The rare blood type identification method based on adversarial transfer learning as claimed in claim 1, characterized in that: The source domain classifier and the target domain classifier are both fully connected layers arranged after the fully connected layer of the CNN model.

4. The rare blood type identification method based on adversarial transfer learning as claimed in claim 1, characterized in that: In the process of training the adversarial transfer learning model based on rare blood type data, if the change error of the adversarial loss function is less than 0.05, the training ends.

5. A computer-readable storage medium storing a computer program, characterized in that: The computer program enables a computer to execute the rare blood type identification method based on adversarial transfer learning as described in any one of claims 1 to 4.

6. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for identifying rare blood types based on adversarial transfer learning as described in any one of claims 1 to 4 is implemented.

Images