Knee osteoarthritis grading method based on quantum to classical transfer learning

Abstract

The present invention relates to the field of computer technology, and in particular to a method for grading knee osteoarthritis based on quantum-to-classical transfer learning, comprising the steps of: designing a hybrid quantum convolutional network structure HQCNN, encoding data into quantum state extraction information, and using the information to distinguish the category of data; according to the label conditions of MRI images and OAI images, and the relationship between MRI image classification and OAI grading tasks, dividing quantum classical QC transfer learning into two stages: pre-training and fine-tuning, the pre-training stage uses a trained quantum neural network model to learn quantum state evolution and quantum gate operations, and the fine-tuning stage applies the pre-trained model to a new task, and adjusts and trains the model in the new task; performing dimensionality reduction processing on image data, encoding the pre-processed image data into quantum data through a data encoding circuit, and using a hybrid quantum convolutional network structure HQCNN for image classification. The present invention solves the technical problem of technical bottlenecks.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a knee osteoarthritis grading method based on quantum to classical transfer learning. Background Art

[0002] Osteoarthritis is a common type of arthritis that is an important factor limiting the mobility and limb function of the elderly. The cause of osteoarthritis is unknown, and its late-stage characteristics are cartilage wear, bone deformity, and synovitis. Imaging examination is the most important tool for detecting osteoarthritis, which can detect early signs of osteoarthritis and slow down the progression of the disease through behavioral interventions (such as exercise and weight loss programs). MRI (magnetic resonance imaging) can reflect the three-dimensional structure of the knee joint, but the number of MRI detection equipment is small and the cost is high, which is not suitable for routine diagnosis of osteoarthritis. On the contrary, X-ray examination has the characteristics of safety, economy, and wide popularity, and has always been used as a routine standard for osteoarthritis examination. Kellgren-Lawrence grading, also known as Kellgren-Lawrence (KL), is one of the most commonly used clinical scales for assessing the severity of osteoarthritis through plain X-ray films. However, the consistency of visual diagnosis by radiologists is low, which will introduce great inconsistency in decision-making. Doctors usually examine the scanned knee X-ray images and then perform KL grading on both knee joints in a short time. The accuracy of diagnosis depends largely on the doctor's experience and carefulness. In addition, as shown in the attached Figure 1 As shown in the literature, the criteria for KL grading are very vague. For example, the criteria for KL grade 1 are possible bone graft labrum and joint space narrowing (JSN). Even the same physician may give different KL grades when examining the same knee at different time points. The intra-rater reliability of KL is between 0.67 and 0.73.

[0003] In recent years, deep learning has become a major research direction in the assessment of osteoarthritis severity. Due to the high prevalence of osteoarthritis, there is an urgent need for methods to accurately detect its presence and quantify its severity. Fully automatic osteoarthritis severity grading can provide objective and repeatable predictions. In the past decade, several methods have been developed for knee joint detection and KL grade classification. For example: a new CNN model is designed for the KL grading task, and a weighted combination of cross entropy loss and mean square error loss is optimized to obtain a recognition rate of 63.6%; the osteoarthritis assessment task is divided into two stages: detection and classification, in which Faster R-CNN is used for detection and deep convolutional networks are used for classification; an automatic algorithm based on deep learning is developed to jointly use the posterior-anterior (PA) and lateral (LAT) views of knee X-rays to assess the severity of knee osteoarthritis according to the KL grading system; a CNN knee osteoarthritis KL grade classification model under the attention mechanism is proposed, which first uses ResNet to extract knee joint features from X-rays, and then combines them with the features extracted by the convolutional attention module to automatically predict the KL grade.

[0004] In addition, quantum AI / ML algorithms are particularly suitable for diagnostic applications. Many potentially correlated variables lead to high-dimensional feature spaces, and the interactions between variables lead to complex interdependencies, correlations, and patterns. Quantum AI / ML algorithms are able to penetrate such data structures in a way that surpasses purely classical methods. As a result, a wide range of quantum applications are being explored in this field, including enhancing image edge detection, segmentation, and classification processing steps, such as: in the classification of retinal color fundus and chest X-ray images, orthogonal QNNs are studied and the training of classical neural networks is accelerated using quantum circuits; based on computed tomography (CT) and positron emission tomography (PET) data, QFT-based enhanced image reconstruction algorithms are developed; for breast cancer classification, QSVCs and quantum kernel Gaussian processes and QNNs based on transfer learning are applied; rheumatoid arthritis is detected by classifying heat maps with QSVCs trained with quantum kernel alignment; MRI images of Alzheimer's disease are classified using QNNs, and COVID-19 is classified using QNNs using chest X-ray and CT lung images. However, due to the technical bottlenecks caused by high coherence requirements, quantum noise, and the limited number of qubits, it is not feasible to use full quantum algorithms. Summary of the invention

[0005] The present invention provides a method for grading knee osteoarthritis based on quantum to classical transfer learning, which solves the technical problem that the full quantum algorithm cannot perform knee osteoarthritis grading.

[0006] The basic scheme provided by the present invention is: a knee osteoarthritis grading method based on quantum to classical transfer learning, comprising the steps of:

[0007] S1. Design a hybrid quantum convolutional network structure HQCNN for multi-classification of MRI images, encode the data into quantum states to extract information, and use the information to distinguish the categories of data. The quantum convolutional network consists of quantum convolutional layers and classical convolutional layers. The quantum convolutional layer consists of an encoder, a random circuit, and a decoder. The classical convolutional layer consists of a classical convolutional layer, a global pooling layer, and a dense layer.

[0008] S2. According to the labeling of MRI images and OAI images, and the relationship between MRI image classification and OAI grading tasks, a quantum-to-classical QC transfer learning scheme is selected, and quantum-classical QC transfer learning is divided into two stages: pre-training and fine-tuning. In the pre-training stage, the trained quantum neural network model is used to learn quantum state evolution and quantum gate operations. In the fine-tuning stage, the pre-trained model is applied to the new task, and the model is adjusted and trained in the new task.

[0009] S3. Perform dimensionality reduction on the image data, encode the preprocessed image data into quantum data through the data encoding circuit, and use the hybrid quantum convolutional network structure HQCNN for image classification.

[0010] The working principle and advantages of the present invention are as follows: in this scheme, firstly, a hybrid quantum convolutional neural network HQCNN is designed, and the hybrid quantum convolutional neural network HQCNN is used for MRI imaging to obtain a quantum convolutional network structure with stronger generalization ability, and secondly, a quantum-to-classical transfer learning method is used for the hierarchical diagnosis of OAI images, thereby improving the classification effect; based on the pre-trained weights, most or all parameters can be updated, and the encoding characteristics of the pre-trained model are used in feature extraction to train the weights of the last layer to improve the extraction ability of the quantum convolutional layer QCL; the technical bottlenecks caused by high coherence requirements, quantum noise and limited number of quantum bits are overcome, thereby improving the feasibility of using a full quantum algorithm for grading and classification.

[0011] The present invention overcomes the technical bottlenecks caused by high coherence requirements, quantum noise and limited quantum bit numbers, improves the feasibility of grading and classification using the full quantum algorithm, and solves the technical problem that the full quantum algorithm cannot be used for grading knee osteoarthritis.

[0012] Furthermore, in S2, the quantum to classical QC transfer learning scheme includes:

[0013] A. First stage: Select the quantum image format of Real Ket, divide an image into 4 blocks, and number each block from left to right. Starting from the top row, these blocks are divided into 4 blocks again and numbered in the same way until the smallest block with only 4 pixels is obtained;

[0014] B. Second stage: Use the pre-trained HQCNN model as the feature extractor. The classic layer of the HQCNN model uses the xception network and adopts the residual learning strategy to learn from the multiple layers of QCNN.

[0015] C. The third stage: perform a linear transformation from 512 features to 64 features on the classical layers, pass the classical output features of the classical convolutional layers to the final fully connected layer, and create a two-dimensional target output class prediction.

[0016] The beneficial effect is that an important criterion for using a pre-trained QCNN model is to use appropriate image preprocessing. The image corresponds to a data point that needs to be encoded. The image needs to be reshaped first to reduce the number of quantum bits in the circuit. Before normalizing the pixels, the pixels are readjusted to (H, W, C) = (128, 128, 1). Normalization refers to the process of converting pixel values ​​0-255 into real numbers in the range of 0.0-1.0. This can reduce the amount of calculation while improving the calculation accuracy.

[0017] Further, in S1, the quantum convolution layer performs sequence parsing on the image to extract local information, features, and patterns from the image. The quantum convolution layer consists of a quantum circuit that encodes pixel data on n×n qubits, where n represents the kernel size. A random quantum circuit is applied to the qubits and measured to produce a feature matrix.

[0018] Furthermore, in S1, during the encoder encoding process, the pixel data corresponding to the filter size is stored in the form of quantum bits, the image is encoded into the quantum circuit, and the data points are encoded into a quantum circuit, including: ground state encoding, angle encoding, amplitude encoding, IQP encoding, and Hamiltonian evolution encoding.

[0019] Furthermore, in S1, the quantum convolution layer includes quantum kernels, which use q circuits to segment the input image into patches containing local information, and detect and extract spatial details and features from the image.

[0020] The beneficial effect is that the basic idea of ​​the quantum convolutional network structure HQCNN designed in this way is to encode data into quantum states and use information to distinguish the categories of data, so as to extract information faster, which can be well used for multi-classification of MRI images. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a schematic diagram of KL grading of the knee joint according to an embodiment of the present invention.

[0022] Figure 2 Schematic diagram of the structure of a hybrid quantum convolutional neural network according to an embodiment of the present invention.

[0023] Figure 3 Schematic diagram of the structure of the quantum convolution layer of an embodiment of the present invention.

[0024] Figure 4 Flow chart of a general transfer learning solution according to an embodiment of the present invention.

[0025] Figure 5 Flow chart of the quantum classical transfer learning solution of an embodiment of the present invention.

[0026] Figure 6 The ROC scores of the training and test sets of each class of the MRI dataset are divided according to the embodiment of the present invention.

[0027] Figure 7 The ROC scores of the training and test sets of each class of the MRI dataset after division according to the embodiment of the present invention.

[0028] Figure 8 The ROC Score of the training set and test set of each class of the OAI dataset is divided according to the embodiment of the present invention.

[0029] Fig. 9 This is the MRI classification report of an embodiment of the present invention.

[0030] Fig.10 This is the OAI classification report of the embodiment of the present invention.

[0031] Fig.11 This is the MRI Confusion Matrix according to an embodiment of the present invention.

[0032] Fig.12 It is the train history before and after the quantum classical migration in the embodiment of the present invention.

[0033] Fig.13 It is the train history of each model in the embodiment of the present invention.

[0034] Fig.14 This is the HQCNN Confusion Matrix of an embodiment of the present invention.

[0035] Fig.15 This is the QC Confusion Matrix according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The following is a further detailed description through specific implementation methods:

[0037] Example 1

[0038] In order to better describe the structure of the hybrid quantum convolutional neural network (HQCNN), as shown in the attached Figure 2 As shown, we first briefly outline some basic features of convolutional neural networks (CNNs). The classic convolutional layer consists of interleaved convolutional layers and pooling layers and ends with a fully connected layer. The main purpose of the convolutional layer is to extract features from the input data using feature maps (or filters), which is the most computationally intensive step in the convolutional neural network (CNN). A pooling layer is usually added after the convolutional layer to reduce the dimensionality of the data and prevent overfitting. The basic idea of ​​the quantum convolutional neural network structure is to encode the data into quantum states to extract information faster, and then use the information to distinguish the categories of the data. The hybrid quantum convolutional neural network consists of quantum convolutional layers and classical convolutional layers. The quantum convolutional layer has three parts: encoder, random circuit, and decoder. After obtaining the output from the quantum convolutional layer (QCL), the information is passed to the classical convolutional layer of the architecture. The classical convolutional layer consists of three different layers, namely classical convolutional layers, global pooling layers, and dense layers.

[0039] The quantum convolution layer performs sequence parsing on the image, extracting local information, features, and patterns from the image, which makes an important contribution to the classification of the image. The hybrid quantum classical method provides a quantum simulation for the classical convolution layer and names it the quantum convolution layer. Similar to the classical convolution layer, the proposed quantum convolution layer extracts high-level spatial features from the input image. The layer consists of a quantum circuit that encodes pixel data on n×n qubits (where n represents the kernel size), applies a random quantum circuit on these qubits, and then measures them to produce a feature matrix. The internal structure of the quantum convolution layer is shown in the attached figure. Figure 3 shown.

[0040] During the encoding process, the pixel data corresponding to the filter size is stored in the form of quantum bits. Encoding images into quantum circuits is a challenging task in the field of quantum machine learning. There are several ways of quantum coding to encode data points into a quantum circuit, including ground state coding, angle coding, amplitude coding, IQP coding, and Hamiltonian evolution coding. A simple encoding idea is to convert a data point into an angle in the quantum state (in fact, quantum bits are state vectors that represent data points as angles and reduce them to different directions of the state vector). Each quantum bit relies on its classical data point to represent its configuration. Angle coding is to use the revolving gate R x ,R y ,R z The classical information is encoded by the rotation angle.

[0041] The rotation matrix is ​​defined as follows:

[0042]

[0043]

[0044]

[0045] These are all valid operators on the Pauli spin matrix. Essentially, the gate R x (θ) rotates the state vector in space by an angle θ relative to the X-axis. Similarly, R y Door and R z The gates are related to the Y and Z axes respectively, and the angle θ directly corresponds to the intensity value of a pixel. After mapping all data points to qubits, the operation of qubits and quantum gates can be used to design random q circuits. The angle encoding encodes N classical data onto N qubits.

[0046]

[0047] Among them, |x> is the classical data vector to be encoded. However, since a qubit can not only load angle information but also phase information, a classical data of length N can be encoded on □N□ qubits.

[0048]

[0049] Among them, the two data are encoded into the rotation angle cos(πx 2i-1 )|0> and phase information middle.

[0050] A series of uniform quantum transformations (implemented by the gates defined earlier) and measurements of qubits are essential to design a quantum circuit. The quantum convolution layer contains the quantum kernel, which essentially partitions the input image into smaller patches containing local information using q-circuits in order to detect and extract meaningful spatial details and features from the image. An idea to have a simple depth and use smaller qubits is to integrate a random q-circuit in the quantum convolution layer. The random circuit is constructed by randomly selected single-qubit gates and two-qubit gates, and the rotations applied on these gates are also randomly selected using Numpy's random method. We design a parametric quantum circuit, which is composed of interleaved single-qubit layers and two-qubit layers. The single-qubit layer consists of Ry gates, each containing a tunable parameter. The two-qubit layer consists of CNOT gates on nearest neighbor qubit pairs. Using Hadamard gates, the qubits are initialized with a balanced superposition of |0> and |1>, and then rotated according to the input parameters. The main purpose of the embedding operation is to initialize the qubits by properly balancing the |0> and |1> qubit values ​​using Hadamard gates and rotation gates based on the input factors. The quantum coding framework is responsible for establishing the association between the classical information input x and its associated quantum state |X>. In general, quantum coding implements quantum embedding, which converts the classical input vector into a quantum state vector. The resulting quantum state is given by the following equation.

[0051]

[0052] The quantum coding circuit develops a new quantum state, as shown in (7)

[0053]

[0054] The quantum variational layer consists of a series of rotational layers consisting of rotational gates, followed by an entanglement layer consisting of CNOT gates to satisfy the training process. During operation, the CNOT gates enforce the entanglement between any adjacent quantum wires by quantum operations, resulting in entanglement of the qubits of each quantum circuit. At the same time, the rotation angle of the rotational gate is adjustable. During training, data is transmitted through the variational layer, which consists of alternating rotational gates and entanglement gates (control and non-control gates). The best performance model of the proposed work includes a quantum depth of 4 in this variational quantum circuit, shown as (8).

[0055] X=L 4o…… L 2o L1(8)

[0056] This component of the QCL is responsible for all measurements that occur in this layer. Decoding means measuring quantum data, and the data is converted into classical form. The measured data provides relevant information about the image, which can then be fed into a neural network to classify the type of image. Pauli spin matrices find their use in this component. They are the most intuitive and simplest way to decode quantum information and generate the corresponding classical information. The data decoder component measures the data along the Z axis by using the Pauli-Z matrix defined earlier. Depending on the selection of the measurement operator, our common measurements are divided into computational basis measurements, projection measurements, Pauli measurements, etc. Pauli measurement is a projection measurement in which the observable M is selected as the Pauli operator. Taking Pauli-Z measurement as an example, let us consider the Z operator:

[0057]

[0058] It can be seen that Z satisfies That is, Z is Hermite. Z has two eigenvalues ​​+1 and -1, and the corresponding eigenvectors are: |0> and |1>. Therefore, the spectral decomposition form of Z is:

[0059]

[0060] Use Z to do projection measurement. If the measurement result is +1, we can conclude that the state of the quantum bit is projected onto the +1 characteristic subspace V of the Z operator. +1 This indicates that the measured state is projected to |0>. Similarly, if the measurement result is -1, it can be concluded that the quantum bit is projected to the -1 characteristic subspace V -1 This indicates that the measured state is projected into |1>.

[0061] After the quantum transformation, the data is in an arbitrary quantum state, and the output quantum state needs to be measured to obtain classical information for subsequent processing. The quantum decoding (measurement) operation is a nonlinear transformation that implements a function similar to the classical activation function. The classical data obtained by measurement will be used in the subsequent classical part of the convolutional neural network, where each qubit is measured, and then a vector is generated and input into the corresponding quantum feature map. One qubit generates a quantum feature map, which corresponds to one channel. When the sliding window of the input image is completed, the quantum feature map of the image is formally generated and serialized, and saved to disk for classifier training.

[0062] Traditionally, deep neural networks require large labeled datasets and powerful computing resources to solve challenging computer vision problems such as feature extraction and classification. Transfer can be used to learn new tasks by transferring knowledge from related learning tasks. From a pre-trained model, we perform two tasks: (1) fine-tuning and (2) feature extraction during fine-tuning. Essentially, the model is retrained, based on the pre-trained weights, updating most or all parameters. In feature extraction, the weights of the last layer (classifier) ​​are trained using the encoded features of the pre-trained model. New datasets with the same number of output classes are predicted and reconstructed, which means that the pre-trained QCNN model acts as a fixed feature extractor. In this case, a pre-trained quantum network acts as a feature extractor, producing an output vector of numerical values ​​associated with the input. The extracted features are then further processed using a classical network to solve a specific problem of interest. General transfer learning schemes are shown in the attached figure. Figure 4 As shown:

[0063] Step 1: Take a network A pre-trained on the dataset DA and the given task TA;

[0064] Step 2: Delete some of the last layers, and the resulting truncated network A' can be used as a feature extractor;

[0065] Step 3: Connect a new trainable network B at the end of the pre-trained network A';

[0066] Step 4: Keep the weights of A' unchanged and train the final block B using a new dataset DB and a new task of interest TB.

[0067] Quantum-to-classical transfer learning involves using a pre-trained quantum circuit as a feature extractor and post-processing its output variables using a classical neural network. In this case, only the final classical part will be trained to the specific problem of interest. In our problem, it can be transformed into the following table: Figure 5 . An important criterion for using a pre-trained QCNN model is to use appropriate image preprocessing. The image corresponds to a data point that needs to be encoded, and it first needs to reshape the image to reduce the number of qubits in the circuit. In addition, the pixels are rescaled to (H, W, C) = (128, 128, 1) before being normalized. Normalization refers to the process of converting pixel values ​​0-255 into real numbers in the range of 0.0-1.0. The advantage of doing this is that it can reduce the amount of calculation while improving the calculation accuracy.

[0068] Table 1: Quantum Classical transfer Learning scheme

[0069]

[0070] A. Phase 1: Qubit Lattice, Real Ket and FRQI are three main quantum image formats, among which, in the Real Ket quantum image format, an image is divided into 4 blocks, each block is numbered from left to right, starting from the top row, these blocks are again divided into 4 blocks and numbered in the same way until a minimum block of only 4 pixels is obtained, and the grayscale values ​​of these four pixels are mapped to the probability amplitude of each component of a quantum state with 2 qubits. Equation (9) describes the quantum state, where i1=1 is the index of the upper left pixel, i1=2 is the index of the upper right pixel, i1=3 is the lower left pixel, and i1=4 is the lower right pixel.

[0071]

[0072]

[0073] C i Store the mapping value for each pixel and satisfy

[0074] We now consider a larger block consisting of 4 inner sub-blocks. To identify which sub-block we are dealing with, a new label is needed, called i2, with the same conventions as defined for the inner blocks. The new image shows 2 2 ×2 2 pixels, and by The real vector representation in is:

[0075]

[0076] in, All pixel values ​​are stored.

[0077] This block structure can be extended to any number of steps up to size 2n×2n. By gradually expanding, a 2 n ×2 n The image can be mapped to the quantum state as shown below:

[0078]

[0079]

[0080] B. Second stage: Feature extraction is to extract valuable features from the image. Now QCNN no longer extracts manual features, but automatically extracts important features from the image. In this work, we use the pre-trained HQCNN model proposed in this paper as a feature extractor. The classic layer of the HQCNN model uses the xception network, which adopts a residual learning strategy that is effective enough to learn from multiple layers of QCNN. The residual block of the xception architecture is described as follows:

[0081] g=F(h,W+h)(12)

[0082] Where h is the input layer, g is the output layer, and the F function is represented by residual mapping.

[0083] C. The third stage: The feature dimension extracted from the HQCNN model is performed on the classical layers, that is, there is a linear transformation from 512 features to 64 features. The classical output features of the classical convolutional layer are passed to the final fully connected layer, creating a 2D target output class prediction, which is the target class predicted by the model for the 5-classification problem.

[0084]

[0085] The two datasets used in this protocol are from Kaggle. The brain tumor dataset includes MRI images of brain tumors of 233 patients, including 926 glioma tumor slices, 937 meningioma tumor slices, 500 tumor-free slices, and 901 pituitary tumor slices, for a total of 3264 images. The distribution of the MRI dataset is shown in Table 2. The images used to evaluate knee X-rays are from the Osteoarthritis Initiative, a multicenter, longitudinal, prospective observational study of knee osteoarthritis aimed at identifying biomarkers for the onset and progression of OA I. 4,796 participants aged 45 to 79 years participated in this test. In our study, a public dataset was used. The knee dataset contains 9516 X-ray images, including 3857 level 0 images, 1770 level 1 images, 2578 level 2 images, 1286 level 3 images, and 295 level 4 images. The distribution of the OAI dataset is shown in Table 3.

[0086] Table 2: MRI data and distribution

[0087]

[0088] Table 3: OAI data and distribution

[0089]

[0090] Adversarial validation is a simple but effective method. Its essence is to construct a classification model to predict the probability that a sample is a training set or a test set. If the classification effect of this model is good (usually AUC is above 0.7), it means that there is a big difference between the training set and the test set, as shown in the following figure. Figure 6 The MRI data set before division is shown. If the auc score is close to 0.5, it means that the distribution of the training set and the test set is consistent, as shown in the attached Figure 7 Divide the MRI dataset into Figure 8 The OAI dataset before partitioning is shown.

[0091] We need to consider the differences in data distribution. The inconsistency between the distribution of training data and the distribution of data that actually needs to be predicted is likely to lead to poor model performance, which is usually called dataset shift. In order to better improve the generalization ability of the model, we use adversarial validation to verify the original dataset. Figure 6 As shown in the figure, we consider re-dividing the train set and test set. There is no dataset shift in the OAI dataset. Figure 7 As shown. Adversarial verification has the following steps:

[0092] Step 1: Separate the index of each category of tumor from the training set and the test set;

[0093] Setp 2: Prepare a new label, set the tumor in the train set to 1, and the tumor in the test set to 0. Then, concatenate the tumors from the training set and the test set;

[0094] Step 3: Shuffle the concatenated data set, and then re-divide each category of tumor according to train:test=9:1;

[0095] Step 4: Prepare a Random Forest Classifier with parameters set to bootstrap = True, oob_score = True, criterion = 'entropy', and perform training.

[0096] Step 5: Use the trained classifier to predict the test set and calculate their ROC scores.

[0097] From the statistics in Table 1, we can see that Grade0, Grade1 and Grade2 images occupy the vast majority of the dataset, while the number of Grade3 and Grade4 images is quite small. The largest number of Grade0 accounts for 39.4%, while the smallest number of Grade4 accounts for only 0.03%. Therefore, the data processed by this scheme is a typical unbalanced dataset. In order to ensure the consistency of label distribution in the training set and the validation set, the StratifiedKFold method is used to divide the dataset. StratifiedKFold is a stratified sampling cross-division method to ensure that the proportion of samples of each category in the training set and the test set is the same as that in the original dataset. The dataset is divided as follows: (1) 1 / 10 holdout set for ensemble. (2) building 5folds cross validation sets using rest of 9 / 10. Table 4 is the 10-fold cross validation of the OAI dataset.

[0098] Table 4: 10-fold cross validation of the OAI dataset

[0099]

[0100]

[0101] We know that choosing a classifier with better performance requires the use of excellent evaluation metrics. To understand the generalization ability of the model, it must be evaluated using objective metrics, which is the importance of performance evaluation. In machine learning, classification tasks involving more than two classes are called multi-class classification. We must be very careful when choosing evaluation metrics for multi-class, single-label classifiers. First, it is different from binary class metrics. Second, it must be considered whether the dataset is single-label or multi-label. In multi-class, there is only one label, while in multi-label, there are multiple labels. Since OAI is multi-class, we chose a measurement metric based on the concept of multi-class, single label.

[0102] Accuracy

[0103] Accuracy is one of the most popular metrics in multi-class classification. It reflects the correct prediction of sample scores and can be calculated directly from the confusion matrix. The calculation formula is as follows:

[0104]

[0105] The model predicted a true positive as positive, but it was actually positive. The model predicted a false positive as positive, but it was actually negative. This calculation also works for multi-class and is very simple. To calculate the accuracy just add these elements on the diagonal of the confusion matrix and divide by the total number of labels. Fig. 9 Having described the HQC-CNN confusion matrix, we can also calculate the accuracy of HQC-CNN.

[0106] Macro precision

[0107] Precision indicates how much we can trust the model when it predicts a positive outcome for an individual. The macro precision of the multi-class algorithm is calculated in two steps. First, equation (15) is used to calculate the precision of each class, where k represents the class.

[0108]

[0109] Secondly, the arithmetic mean of each class is calculated using formula (16), where K represents the number of classes.

[0110]

[0111] Macro recall

[0112] The multi-class macro recall, like macro precision, requires a two-step calculation. First, equation (17) is used to calculate the recall of each class, where k represents the class,

[0113]

[0114] Secondly, the arithmetic mean of each class is calculated using formula (18), where K represents the number of classes,

[0115]

[0116] Macro F1 Score

[0117] The macro F1 score is the harmonic mean of macro precision and macro recall, taking into account both the accuracy and recall of the classification model. Therefore, before calculating the macro f1 score, you must first calculate the macro precision and macro recall. Therefore, before calculating the macro f1 score, you must first calculate the macro precision and macro recall. The calculation formula for the macro F1 score is as follows:

[0118]

[0119] In order to observe each class, the precision, recall, f1 and support of the class in the two datasets are accurately calculated, where suppprt refers to the number of classes belonging to the class in the original real data. The classification report of the MRI dataset is as follows Fig. 9 , the classification report of the OAI dataset is as attached Fig.10 .

[0120] Comparative test

[0121] The first stage of the experiment requires explanation of weight initializer, loss function, and optimizer. Glorot uniform initializer, also called Xavier uniform initializer, is used to initialize the convolution kernel weights so that the variance of each layer is as equal as possible, so as to achieve better information flow in the network. We know that for regression models, the commonly used loss function is the mean square error function, while for classification models that predict probability, the most commonly used loss function is cross entropy. Therefore, we uniformly use Categorical crossentropy as the loss function. Categorical crossentropy, also known as Softmax loss, is a Softmax activation plus a cross entropy loss. The optimizer chooses the most commonly used Adam. In addition, we set the learning rate to 0.001 based on experience. Table 5 shows the comparison between HQC-CNN and other models. The results show that the HQC-CNN we proposed is superior to other classic models.

[0122] Table 5: Comparison of HQC-CNN with other models

[0123]

[0124] Classification performance is often represented by scalar values, such as accuracy, sensitivity, and specificity. The confusion matrix measures the model in a more intuitive matrix form, and can also show the classification of each category to achieve a more comprehensive measurement. The confusion matrix, also known as the error matrix, is a standard format for representing accuracy evaluation. It is represented in the form of a matrix with N rows and N columns, where N is the number of target classes. We can think of the confusion matrix as a summary table of the number of correct and incorrect predictions generated by the classification model for the classification task. We roughly measure the number of accurate classifications by observing the values ​​of the diagonal to determine the accuracy of the model. Appendix Fig.11 It is the confusion matrix of the prediction results of HQCNN for the MRI dataset.

[0125] The second phase of the experiment is aimed at the five-classification problem of different types of knee osteoarthritis. The proposed integrated framework for OAI classification is trained according to the optimal hyperparameters of HQCNN. We will analyze the advantages of quantum classical transfer learning for OAI dataset classification from the following aspects. Fig.12 is the train history before and after quantum classical migration, Fig.13 It is the train history that connects different classic models in the HQCNN model.

[0126] Validation accuracy fluctuates: SqueezeNet is the most stable, but its accuracy is always below 39%. Although EfficientNet is relatively stable, its accuracy is always below 56.38%. Xception fluctuates at epoch=4 and then shows a downward trend. The quantum classical transfer learning method (QC) fluctuates at epoch=7, and then becomes relatively stable and shows an upward trend. Although the validation set accuracy of ResNet18 and NasNetMobile is generally increasing, they have been fluctuating greatly and are obviously not stable.

[0127] Convergence speed: The accuracy curve can reflect the convergence speed of the loss function. HQCNN is the fastest in terms of convergence speed. Its accuracy exceeds 90% after less than 2 epochs. Xception is the second and starts to converge quickly after 4 epochs.

[0128] Overfitting: SqueezeNet suffers from serious overfitting. NasNetMobile has good accuracy on the training set, but the accuracy on the validation set and test set is far apart, and the prediction ability is insufficient.

[0129] Table 6: Comparison of different classic models used in HQCNN

[0130]

[0131] Attached Fig.14 is the confusion matrix of HQCNN’s prediction results for the OAI dataset. Fig.15 This is the confusion matrix of the prediction results of QC for the OAI dataset, in which the percentage in each square represents the percentage of the number of correctly predicted classes in the total number of classes, that is, the accuracy of the class, and the number above the percentage is the number of correctly predicted classes. We can see from the diagonal of this confusion matrix that the quantum classical transfer learning method can achieve 100% accuracy in the classification of OAI.

[0132] In summary, the comparative experiments show that quantum classical transfer learning (QCTL) performs well in the OAI classification task, with the advantages of fast training speed, stable performance, and high classification accuracy. It ranks first in accuracy, macro precision, macro recall, and macro f1 score. The comparison results in Table 6 and Table 7 show that the average accuracy index of this method is the best, which is 96.36%. Therefore, the method proposed in this scheme is more reliable for the classification of OAI.

[0133] Table 7: Comparison of the proposed knee osteoarthritis method with the existing technology

[0134]

[0135]

[0136] The above is only an embodiment of the present invention. The common sense such as the known specific structure and characteristics in the scheme is not described in detail here. The ordinary technicians in the relevant field know all the common technical knowledge in the technical field of the invention before the application date or priority date, can obtain all the existing technologies in the field, and have the ability to apply the conventional experimental means before that date. The ordinary technicians in the relevant field can improve and implement this scheme in combination with their own abilities under the enlightenment given by this application. Some typical known structures or known methods should not become obstacles for ordinary technicians in the relevant field to implement this application. It should be pointed out that for those skilled in the art, without departing from the structure of the present invention, several deformations and improvements can be made, which should also be regarded as the protection scope of the present invention, which will not affect the effect of the implementation of the present invention and the practicality of the patent. The protection scope required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. A method for grading knee osteoarthritis based on quantum to classical transfer learning, characterized in that: Includes steps: S1. Design a hybrid quantum convolutional network structure HQCNN for multi-classification of MRI images, encode the data into quantum states to extract information, and use the information to distinguish the categories of data. The quantum convolutional network consists of quantum convolutional layers and classical convolutional layers. The quantum convolutional layer consists of an encoder, a random circuit, and a decoder. The classical convolutional layer consists of a classical convolutional layer, a global pooling layer, and a dense layer. S2. According to the labeling of MRI images and OAI images, and the relationship between MRI image classification and OAI grading tasks, a quantum-to-classical QC transfer learning scheme is selected, and quantum-classical QC transfer learning is divided into two stages: pre-training and fine-tuning. In the pre-training stage, the trained quantum neural network model is used to learn quantum state evolution and quantum gate operations. In the fine-tuning stage, the pre-trained model is applied to the new task, and the model is adjusted and trained in the new task. S3, perform dimensionality reduction processing on the image data, encode the pre-processed image data into quantum data through the data encoding circuit, and use the hybrid quantum convolutional network structure HQCNN for image classification; In S2, the quantum to classical QC transfer learning scheme includes: A. First stage: Select the quantum image format of Real Ket, divide an image into 4 blocks, and number each block from left to right. Starting from the top row, these blocks are divided into 4 blocks again and numbered in the same way until the smallest block with only 4 pixels is obtained; B. Second stage: Use the pre-trained HQCNN model as the feature extractor. The classic layer of the HQCNN model uses the xception network and adopts the residual learning strategy to learn from the multiple layers of QCNN. C. The third stage: perform a linear transformation from 512 features to 64 features on the classical layers, pass the classical output features of the classical convolutional layers to the final fully connected layer, and create a two-dimensional target output class prediction.

2. The method for grading knee osteoarthritis based on quantum to classical transfer learning as claimed in claim 1, characterized in that: In S1, the quantum convolution layer performs sequence parsing on the image to extract local information, features, and patterns from the image. The quantum convolution layer consists of a quantum circuit that encodes pixel data on n×n qubits, where n represents the kernel size. A random quantum circuit is applied to the qubits and measured to produce a feature matrix.

3. The method for grading knee osteoarthritis based on quantum to classical transfer learning as claimed in claim 2, characterized in that: In S1, during the encoder encoding process, the pixel data corresponding to the filter size is stored in the form of quantum bits, encoding the image into a quantum circuit and encoding the data points into a quantum circuit, including: ground state encoding, angle encoding, amplitude encoding, IQP encoding, and Hamiltonian evolution encoding.

4. The method for grading knee osteoarthritis based on quantum to classical transfer learning as claimed in claim 3, characterized in that: In S1, the quantum convolution layer includes quantum kernels and uses q circuits to segment the input image into patches containing local information to detect and extract spatial details and features from the image.

Images