A multi-label pelvic organ prolapse detection method based on Vision Transformer

Abstract

The application discloses a multi-label pelvic organ prolapse detection method based on a Vision Transformer, and belongs to the technical field of image processing and application, and aims to solve the problems of time consumption and inaccuracy in manual pelvic organ prolapse diagnosis, and develop a multi-label deep learning method to shorten the reading time of doctors, reduce the misdiagnosis rate, and improve the diagnosis efficiency of diseases. The method comprises the following steps: selecting pelvic magnetic resonance images provided by a cooperative hospital as original samples, dividing the samples into a training set, a verification set and a test set, and performing a series of preprocessing operations; a feature extraction module for medical images and a multi-label classification grading model based on the Vision Transformer are designed to automatically classify and detect three organ diseased areas in the eye pelvic floor image. The multi-label pelvic organ prolapse detection method based on the Vision Transformer can extract more target information, so as to realize accurate grading detection of the pelvic organ prolapse condition in the pelvic floor magnetic resonance image.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing and application technology, specifically relating to a multi-label pelvic organ prolapse detection method based on deep learning. Background Technology

[0002] Pelvic organ prolapse (POP) is defined as descent of one or more of the following: the anterior vaginal wall, posterior vaginal wall, uterus (cervix), or vaginal apex (vaginal fornix or cuff scar after hysterectomy). It has a severely negative impact on a patient's quality of life. According to global statistics, 2.9% to 53% of women have some form of POP. The prevalence of POP among working-age women is as high as 47%. The probability of a woman undergoing surgery for pelvic organ prolapse in her lifetime is 11%. In most cases, POP is asymptomatic, indicating its high prevalence in the population.

[0003] In recent years, deep learning technology has been widely applied in medical auxiliary diagnosis and has achieved great success. However, although traditional CNNs have great potential in medical image recognition, medical images differ from natural images, possessing unique noise, artifacts, and clinically important recognition regions. Therefore, we need to design models with specific interpretability and recognition structures. Currently, there is very little research on using deep learning technology to assist in the diagnosis of pelvic organ prolapse, both domestically and internationally. Existing research directions are mostly based on traditional network structures, which leads to the loss of much edge detail information during convolution, resulting in unsatisfactory detection results. Summary of the Invention

[0004] Objective: This invention applies artificial intelligence technology to medical auxiliary diagnosis. Focusing on clinical auxiliary research using MRI images of female pelvic organs, it proposes a multi-label method for detecting pelvic organ prolapse based on Vision Transformer. This method establishes a dataset based on MRI images of female pelvic organs labeled by specialists, constructs a medical image feature extraction module and a multi-label classification model, and aims to predict organ prolapse with high accuracy. Based on the structural characteristics of the network model and the requirements of medical image classification tasks, a new classification and detection method is proposed to achieve accurate detection of pelvic organ prolapse.

[0005] Technical solution:

[0006] 1. A multi-label pelvic organ prolapse detection method based on Vision Transformer includes a feature acquisition module and a multi-label training module. This method uses a dataset of pelvic MRI images annotated by specialists for multi-label model training. The trained multi-label pelvic organ prolapse model can be used to classify MRI images of affected pelvises, and finally, the Grad-CAM algorithm is used to mark the regions that have the greatest impact on the model's classification results to assist in diagnosis. The specific steps are as follows:

[0007] (1) The pelvic MRI images constitute a set of pelvic organ prolapse detection datasets. The pelvic organ prolapse detection datasets serve as original data samples. The feature acquisition module performs preprocessing operations on the images in the original data samples to ensure that a set of trainable feature matrices can be formed based on the images.

[0008] (2) The multi-label training module is trained using the pelvic MRI image dataset. The multi-label training module includes a label mask embedding network, a Vision Transformer, and an independent feedforward network. The label mask embedding network uses a ternary encoding method to construct corresponding label vectors for the diseases present in the training set images, which are then processed by the Vision Transformer along with the image features acquired by the feature acquisition module. The Vision Transformer uses a feature map block partitioning method to process the features of the training set images, and its encoder obtains an image feature block association relationship through a multi-head attention mechanism with multiple layers. The independent feedforward network obtains the pelvic organ prolapse detection result based on the output of the Vision Transformer through a unidirectional propagation feedforward fully connected neural network.

[0009] (3) The classification model is trained using the training module. The pelvic organ prolapse detection dataset is input into the network for training. During training, the model parameters are adjusted by setting the learning rate, loss function method, and learning iteration cycle, and finally a pelvic organ prolapse detection model is formed.

[0010] (4) The Grad-CAM algorithm is used to mark the regions that have the greatest impact on the classification results based on the trained multi-label pelvic organ prolapse model, so as to provide interpretable evidence and assist clinical diagnosis.

[0011] The feature acquisition module is:

[0012] The method of this invention requires an input image size of M×N×3, where M and N are positive integers, and an original image size of H×W, where H and W are positive integers. In the image processing stage, the image is first scaled to M×((N / H)×W), and then the shorter side is padded with gray edges to convert its size to M×N. The preprocessed image uses a convolutional block consisting of two 3×3 convolutional layers to generate features of size H / 4×W / 4×C, where C is the number of channels. After passing through the three sets of convolutional layers, the input image features of size M×N×3 are extracted.

[0013] The aforementioned tag mask embedding network is:

[0014] The method of this invention applies to all label vectors l present in the training set. i Add a "state" vector s to each i ∈R d Among them, s i There are three possible scenarios: PR (property deviation is known to exist), NU (property deviation is known not to exist), and UN (property deviation is unknown). PR, NU, and UN are all learnable vectors of size d*3, with UN always being all zeros. As a label "embedded" into the image features, it is extracted as input to the encoder.

[0015] The feature image segmentation method of the Vision Transformer is as follows:

[0016] First, the image features of size M×N×3 are upsampled to M'×N'×3 through an upsampling operation. The M'×N'×3 image is then input into the Vision Transformer, which divides the input image into (M' / Patch)×(N' / Patch) feature image blocks of size Patch×Patch. A trainable positional information parameter of size 1×((M' / Patch)×(N' / Patch))×(3×Patch×Patch) is added to each feature image block.

[0017] The multi-head attention mechanism is as follows:

[0018] The multi-head attention mechanism is used to input the feature image patches as units. Matrix operations are used to calculate the relationships between the feature image patches and generate new image features of size ((M' / Patch)×(N' / Patch))×(3×Patch×Patch). This multi-head attention mechanism is iterated 12-16 times. The image features generated by the multi-head attention mechanism are then transformed to obtain a feature image attention map of size 2048×8×8.

[0019] The independent feedforward network is:

[0020] A three-layer fully connected neural network is used, where each neuron in each layer is connected to all neurons in the next layer, but there are no feedback connections. The feature maps obtained by the Vision Transformer are passed through the independent feedforward network to obtain a matrix of size K×1 representing the organ prolapse prediction results, where K is the number of labels contained in the training set images;

[0021] The Grad-CAM algorithm is as follows:

[0022] Grad-CAM combines the gradient information and category activation map concepts of the multi-label pelvic organ prolapse model to generate a heatmap of key regions with spatial information, thereby assisting clinical understanding of the important regions that the model focuses on when performing classification tasks. Taking the pelvic organ prolapse detection task as an example, the Grad-CAM map for category c... It can be represented as Where u and v represent the width and height A of the feature map. k This represents the k-th channel of feature map A. The formula... The calculation formula is: in The model prediction result y c The gradient of the activation value at (i, j) in the k-th channel of feature map A;

[0023] 2. The classification and detection method according to claim 1, characterized in that,

[0024] The loss function of the multi-label pelvic organ prolapse model is the cross-entropy loss function, and its purpose is to minimize the following objective function:

[0025]

[0026] Where N tr This indicates the upper limit of the number of training iterations. Let represent n unknown labels randomly sampled from all possible labels y, and CE denote the cross-entropy loss function. This indicates the calculation of the known label y. k The expected value of the probability distribution.

[0027] 3. The classification and detection method according to claim 1, characterized in that,

[0028] The multi-label pelvic organ prolapse model has a learning iteration cycle of 120 cycles. To accelerate the training speed and prevent the weights from being destroyed in the early stage of training, frozen training is used for cycles 0-30, and normal training is used for cycles 30-120. The learning rate is 1e-5 and the weight decay is 1e-4.

[0029] Beneficial effects:

[0030] The training was conducted using an expert-annotated dataset of pelvic organ prolapse, ensuring the authority of the training.

[0031] ResNet50 and multi-scale visual convolutional networks were used as feature extraction networks, and the model was pre-trained on the public dataset ImageNet using transfer learning. The deep convolutional network ensured that the model could extract sufficiently complete regional features.

[0032] The label mask training method, which uses ternary coding, integrates target labels with medical image features, thereby enhancing the correlation between the image and the lesion.

[0033] The multi-head attention mechanism in Vision Transformer can greatly preserve the detailed information of the lesion region contour by linking the relationship between the image and the label. Attached Figure Description

[0034] Figure 1 Schematic diagram of feature acquisition module structure A

[0035] Figure 2 Schematic diagram of feature acquisition module structure B

[0036] Figure 3 A schematic diagram of a multi-label classification model constructed using Vision Transformer.

[0037] Figure 4 Schematic diagram of Grad-CAM results Detailed Implementation

[0038] Example: The multi-label pelvic organ prolapse detection method based on Vision Transformer provided by this invention detects pelvic organ prolapse. The specific operation is as follows:

[0039] 1. Pelvic MRI Image Dataset

[0040] The pelvic MRI image dataset contains sagittal MRI sequences of 662 subjects taken at rest and at maximum Valsalva. The training set contains 464 images, the validation set contains 92 images, and the test set contains 100 images. The clinical characteristics of the enrolled patients are shown in Table 1. The labels for each MRI image of organ prolapse were manually annotated by a pelvic floor specialist.

[0041] Table 1 Clinical characteristics of the enrolled patients

[0042]

[0043] 4. Data Preprocessing

[0044] This invention collects sagittal views of MRI from a PACS workstation and stores them as DICOM metadata on the workstation for annotation and further analysis. Before inputting the DICOM images into the network, all images are converted to JPG format. During this conversion, the bit depth of the image is reduced from 12 bits to 8 bits, and grayscale values ​​are used to replicate each channel to convert a single-channel DICOM image into a three-channel RGB image. This conversion process follows relevant regulations and guidelines for medical image processing. All MRI slices are uniformly processed to ensure they are 512×512×3 pixels in both height and width.

[0045] 5. Feature Acquisition Module

[0046] This invention designs two feature acquisition modules with different structures for extracting raw MRI features and region of interest (ROI) features, respectively. Structure A employs a three-stage feature layer for ROI feature extraction. In Structure A, the model receives an input image of size H×W×3 and uses a stem block consisting of two 3×3 convolutional layers to generate features of size H / 4*W / 4*C2, where C2 is the number of channels in the second stage. In the second and third stages, we use feature maps based on the output of the previous stage.

[0047] The input is used as the input for the i-th stage.

[0048] In the second and third stages, visual labels for different receptive fields are obtained by using two overlapping convolutional kernels of different sizes (3*3 and 7*7) on the input feature map from the previous stage. These labels are called feature tokens. Then, the feature tokens of different sizes are fed into the Transformer respectively.

[0049] In the Transformer, the model introduces a global-to-local feature interaction module, which consists of a depth residual bottleneck block composed of 1×1 convolutions, 3×3 depthwise convolutions, and 1×1 convolutions. In the i-th stage, the output of the Transformer encoder interacts with the local feature L. i They merge together to form RoiFeature i =Concat([L i G i,0 G i,1 ]), subsequently, RoiFeature i After downsampling via a 1×1 convolution, the input for stage i+1 is obtained. Here, RoiFeature iFor the output of stage i, Concat is the aggregation function, and G is the output of stage i. i,0 G i,1 These are the outputs of the two Transformer encoders in the i-th stage.

[0050] Structure B, based on the ResNet-101 model, is used to extract visual features from the raw MRI. The method of this invention replaces the 7*7 average pooling layer in the original model with a 2D average pooling layer with a kernel size of 7, a stride of 1, and padding of 0. For a raw MRI with an input size of 512×512, the shape of the output feature OrinFeature is 1×2048×16×16.

[0051] 6. Training a multi-label pelvic organ prolapse model

[0052] Based on the MRI feature matrix Z∈R extracted by the feature acquisition module h×w×d Retrieve a set of z from it i ∈R d The range of i is from 1 to P (where P = h × w), representing a small sub-region mapped back to the original image space. For each MRI, we construct a set of label vectors L = {l1, l2, ..., l...} γ},l i ∈R d These represent labels that may exist in MRI. The method of this invention employs a state embedding technique, allowing the model to incorporate known information, thereby improving the accuracy of predicting unknown conditions. For a given label vector l i We add a "state" vector s to this label. i ∈R d Therefore, there is Among them, s i There are three possible scenarios: PR (property known to exist), NU (property known not to exist), and UN (property unknown). PR, NU, and UN are all learnable vectors of size d×3, with UN always being all zeros. During model training, a random variable s is added to each different property label to represent the unknown. i Learn from the unknown using the remaining known labels.

[0053] The model will As the label is "embedded" into the MRI features to form the input of the encoder. Let... This is the set of matrices input to the Transformer. In the Transformer, the weight relationships between different label vectors are learned through the Transformer's multi-head self-attention mechanism. As shown in equations (1), (2), and (3), firstly, the attention weight α between embeddings i and j is calculated.ij Then, all embedding vectors are weighted and summed according to the calculated attention weights to obtain... After nonlinear transformation, h' is obtained i We set up a 3-layer encoder for the model, h' i As input to the second and third layer encoders.

[0054]

[0055]

[0056]

[0057] Where Wk is the key weight matrix, W q It is the query weight matrix, W v It is a value weight matrix, W r and W o b1 and b2 are the transformation matrix and bias vectors, respectively. We represent the output of the third-layer encoder as H' = {z'1, ..., z'}. h×w ,l'1,...l' γ}

[0058] After modeling feature and label dependencies using a Transformer, the classifier uses an independent feedforward network (FFNi) for the final label prediction (Equation 4).

[0059]

[0060] Wherein, the weight of label i It is a 1*d vector, and σ is the simoid function.

[0061] 4. Analysis of Processing Results

[0062] The method of this invention employs four widely used evaluation metrics to assess model performance, including precision (Equation 5), recall (Equation 6), Kappa value (Equation 7), and area under the curve (AUC).

[0063]

[0064]

[0065]

[0066] TP, FP, and FN represent the number of true positives, false positives, and false negatives, respectively. TP is defined as a correct prediction of organ prolapse. FP is defined as a false prediction of organ prolapse. FN is defined as no detection of organ prolapse. The Kappa score is a statistic used to evaluate classifier consistency; it measures the degree of consistency between the classifier's predicted labels and the actual labels. o p represents the "actual consistency" among observers. e Representing "consistency of expectations," AUC stands for Area Under the ROC curve, which uses the false positive rate as the horizontal axis and the true positive rate as the vertical axis. Furthermore, a network visualization technique called Grad-CAM is used to analyze predictions and make the neural network interpretable. Depending on the specific prediction category, this technique highlights the most active regions in the input image, showing which parts are likely most important for the prediction.

[0067] The results of this invention on the pelvic organ prolapse test set are: Precision: 86.43%, Recall: 76.57%, Kappa: 76.44%, AUC: 93.16%. Extensive use demonstrates that the multi-label pelvic organ prolapse detection method based on Vision Transformer proposed in this invention has high classification and detection performance. This is of significant importance in the medical field.

[0068] As described above, although the present findings have been shown and expressed with reference to specific preferred embodiments, they should not be construed as limiting the invention itself. Various changes in form and detail may be made without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

1. A multi-label pelvic organ prolapse detection method based on Vision Transformer, including a feature acquisition module and a multi-label training module. This method uses a dataset of pelvic MRI images labeled by specialists for multi-label model training. The trained multi-label pelvic organ prolapse model can be used to classify MRI images of affected pelvis, and finally, the Grad-CAM algorithm is used to mark the regions that have the greatest impact on the model's classification results to assist in diagnosis. The specific steps are as follows: (1) The pelvic MRI images constitute a set of pelvic organ prolapse detection datasets. The pelvic organ prolapse detection datasets serve as original data samples. The feature acquisition module performs preprocessing operations on the images in the original samples to form training set images and ensures that a set of trainable feature matrices can be formed based on the images. (2) The multi-label training module is trained using the pelvic MRI image dataset. The multi-label training module includes a label mask embedding network, a Vision Transformer, and an independent feedforward network. The label mask embedding network uses a ternary encoding method to construct corresponding label vectors for the diseases present in the training set images. These vectors, along with the image features acquired by the feature acquisition module, are then processed by the Vision Transformer. The Vision Transformer uses a feature map block partitioning method to process the image features of the training set. Its encoder obtains an image feature block association relationship through a multi-head attention mechanism with multiple layers. The independent feedforward network obtains the pelvic organ prolapse detection result based on the output of the Vision Transformer through a unidirectional propagation feedforward fully connected neural network. (3) The classification model is trained using the training module. The pelvic organ prolapse detection dataset is input into the network for training. During training, the model parameters are adjusted by setting the learning rate, loss function method, and learning iteration cycle, and finally a pelvic organ prolapse detection model is formed. (4) The Grad-CAM algorithm is used to mark the regions that have the greatest impact on the classification results based on the trained multi-label pelvic organ prolapse model, so as to provide interpretable evidence and assist clinical diagnosis. The feature acquisition module is: The method of this invention requires an input image size of M×N×3, where M and N are positive integers, and an original image size of H×W, where H and W are positive integers. In the image processing stage, the image is first scaled to M×((N / H)×W), and then the shorter side is padded with gray edges on both sides to convert its size to M×N. The preprocessed image is used to generate features of size H / 4×W / 4×C using a convolutional block consisting of two 3×3 convolutional layers, where C is the number of channels. After passing through the three sets of convolutional layers, the input image features of size M×N×3 are extracted. The aforementioned tag mask embedding network is: The method of this invention applies to all label vectors l present in the training set. i Add a "state" vector s to each i ∈R d , where s i There are three possible scenarios: PR (property retrieval is known to exist), NU (property retrieval is known to not exist), and UN (property retrieval is unknown). PR, NU, and UN are all learnable vectors of size d*3, where UN is always set to all zeros. The labels are "embedded" into the image features and extracted as input to the encoder. The feature image segmentation method of the Vision Transformer is as follows: First, the image features of size M×N×3 are upsampled to M'×N'×3 through an upsampling operation. The M'×N'×3 image is then input into the Vision Transformer, which divides the input image into (M' / Patch)×(N' / Patch) feature image blocks of size Patch×Patch. A trainable positional information parameter of size 1×((M' / Patch)×(N' / Patch))×(3×Patch×Patch) is added to each feature image block. The multi-head attention mechanism is as follows: The multi-head attention mechanism is input in units of the feature image blocks. The relationship between each feature image block is calculated through matrix operations, and a new image feature of size ((M' / Patch)×(N' / Patch))×(3×Patch×Patch) is generated. The multi-head attention mechanism is iterated 12-16 times. The image feature generated in the multi-head attention mechanism is transformed to obtain the feature image attention map of size 2048×8×8. The independent feedforward network is: A three-layer fully connected neural network, wherein the neurons in each layer are connected to all neurons in the next layer, but there are no feedback connections. The feature maps obtained by the Vision Transformer are passed through the independent feedforward network to obtain a matrix of size K×1 representing the organ prolapse prediction results, where K is the number of labels contained in the training set images. The Grad-CAM algorithm is as follows: Grad-CAM combines the gradient information of the multi-label pelvic organ prolapse model with the concept of category activation maps to generate a heatmap of key regions with spatial information. This helps clinicians understand the important regions that the model focuses on when performing classification tasks. Taking the pelvic organ prolapse detection task as an example, the Grad-CAM map for category c... It can be represented as Where u and v represent the width and height A of the feature map. k The formula represents the k-th channel of feature map A. The calculation formula is ,in The model prediction result y c The gradient of the activation value at (i, j) on the k-th channel of feature map A.

2. The detection method according to claim 1, characterized in that, The loss function of the multi-label pelvic organ prolapse model is the cross-entropy loss function, and its purpose is to minimize the following objective function: Where N tr This indicates the upper limit of the number of training iterations. Let CE represent n unknown labels randomly sampled from all possible labels y, and let CE denote the cross-entropy loss function. This indicates the calculation of the known label y. k The expected value of the probability distribution.

3. The detection method according to claim 1, characterized in that, The multi-label pelvic organ prolapse model has a learning iteration cycle of 120 cycles. To accelerate the training speed and prevent the weights from being destroyed in the early stage of training, frozen training is used for cycles 0-30, and normal training is used for cycles 30-120. The learning rate is 1e-5 and the weight decay is 1e-4.

Images