Text recognition method based on quantum transfer learning

Abstract

The invention provides a text classification method based on quantum transfer learning, and belongs to the field of fusion of quantum computing and artificial intelligence. In consideration of complex feature space and nonlinear characteristics in text classification, quantum circuit design, a self-attention mechanism and a quantum transfer learning process are optimized, so that the learning efficiency and classification accuracy of the model are improved. The high-dimensional feature representation and a complex nonlinear decision process are converted into a quantum self-attention network structure, an optimal strategy of text classification is obtained by utilizing a complex quantum kernel self-attention network (CQKSAN) model, and a quantum superposition and entanglement mechanism is introduced to enhance the expression ability of the model. A simulation result shows that compared with a traditional deep learning algorithm and other mixed quantum-classical algorithms, the algorithm provided by the invention has higher classification precision and learning efficiency. The method not only shows the potential of a mixed quantum-classical method, but also provides a basis for further research of a future full-quantum learning system.

Description

Technical Field

[0001] The present invention belongs to the field of integration of quantum transfer learning and artificial intelligence, and specifically relates to a text classification method based on quantum transfer learning, and more specifically, to a quantum transfer learning model combined with a complex quantum kernel self-attention network (CQKSAN) for improving text classification performance. Background Art

[0002] In recent years, quantum computing, as a revolutionary technology, has demonstrated its great potential in many fields through the superposition and entanglement of quantum bits (qubits), especially in solving high-dimensional and complex data problems that are difficult to deal with in classical computing. Due to this characteristic of quantum computing, quantum computers are expected to bring revolutionary changes to the field of artificial intelligence (AI) in the future.

[0003] Machine learning in artificial intelligence has made significant progress through classical methods, especially in fields such as natural language processing (NLP), where classical deep learning models (such as BERT, Transformer, etc.) have performed well in tasks such as text classification, machine translation, and named entity recognition. However, the limitations of these classical methods are that they are inefficient when processing high-dimensional complex data and require a lot of computing resources. In order to overcome these problems, the combination of quantum transfer learning and machine learning has become a very promising solution.

[0004] In the current era of noisy intermediate-scale quantum (NISQ) devices, the number of qubits and error rates still limit the application of pure quantum transfer learning. Therefore, hybrid classical-quantum models have become an effective solution. By using classical neural networks for feature extraction and combining them with quantum circuits for classification, these models can optimize performance within the limitations of current quantum hardware.

[0005] Transfer learning, as an effective method to improve the efficiency and accuracy of machine learning, has been widely used in NLP tasks, especially in dealing with low-resource languages ​​and cross-language adaptation problems. Quantum transfer learning is a fusion of quantum transfer learning and transfer learning, aiming to take advantage of quantum processing and pattern recognition to optimize solutions for specific tasks.

[0006] In existing research, classical to quantum transfer learning methods have demonstrated performance improvements in multiple fields by combining classical feature extractors with quantum classifiers, especially when dealing with small data sets and high-dimensional complex tasks. Models such as quantum self-attention network (QSANN), quantum kernel self-attention mechanism (QKSAM), and quantum mixed state self-attention network (QMSAN) have been proposed and applied in the field of NLP, demonstrating the potential advantages of quantum transfer learning in text classification and language modeling.

[0007] This paper proposes a complex quantum kernel self-attention network (CQKSAN) model based on quantum transfer learning, which improves the efficiency and accuracy of text classification by combining quantum circuits and self-attention mechanisms. Using the framework of quantum transfer learning, CQKSAN fully utilizes the advantages of classical pre-training models and enhances them through the characteristics of quantum transfer learning. In experiments, the CQKSAN model significantly outperforms traditional deep learning models in terms of learning efficiency and accuracy, especially showing faster convergence speed and higher model stability on small sample data sets. Summary of the invention

[0008] The purpose of the present invention is to provide a complex quantum kernel self-attention network (CQKSAN) model based on quantum transfer learning, which is used to solve the problems of insufficient efficiency and accuracy in existing text classification tasks.

[0009] In order to achieve the above object, the present invention provides the following scheme:

[0010] The present invention provides a text classification method based on a complex quantum kernel self-attention network of quantum transfer learning, comprising the following steps:

[0011] S1: Build a pre-trained classic neural network (such as BERT) as a feature extractor, and extract features from the input text data through the pre-trained model.

[0012] S2: The extracted features are processed through quantum circuits, which optimize the feature representation capability based on quantum superposition and entanglement characteristics. The classical features are mapped to quantum states using angle encoding.

[0013] S3: Apply the self-attention mechanism in quantum circuits to transform input features into high-dimensional quantum states, calculate queries, keys, and values, and execute the self-attention mechanism through quantum circuits to obtain better feature weight distribution.

[0014] S4: Using the complex quantum kernel method (CQKSAN) model, the similarity between different input samples is calculated through quantum feature mapping and kernel function to complete text classification.

[0015] S5: Through the quantum transfer learning framework, we combine the advantages of classical neural networks and quantum circuits to improve the learning efficiency and accuracy of the model.

[0016] Furthermore, in S1, a pre-trained classical neural network (such as BERT) is constructed as a feature extractor, and features are extracted from the input text data through the pre-trained model, specifically:

[0017] The pre-trained model is the BERT model, which performs context-aware feature extraction on text data through a bidirectional encoder and is fine-tuned in downstream tasks to adapt to specific text classification tasks. The pre-trained BERT model provides powerful context-aware sentence representations that can be fine-tuned for a variety of downstream NLP tasks. Fine-tuning requires adjusting multiple hyperparameters, whose values ​​directly affect the performance of the model. The BERTbase model consists of 12 hidden layers, each with 768 hidden units and 12 attention heads, for a total of 110 million parameters. The BERTbase model accepts input sequences of up to 512 tokens, each represented by a specific embedding. These embeddings are used for a variety of NLP tasks, including syntactic analysis. The tokens in the input sequence can be case-sensitive or case-insensitive, resulting in two different pre-trained models. The bidirectionality of BERT enables it to encode rich contextual information internally and adapt to specific tasks in the fine-tuning stage, making it a base model in recent years.

[0018] Before the token sequence is fed into BERT, a special [CLS] token is added to the front of the sequence to obtain the vector representation of the entire sequence. Another special token [SEP] is placed at the end of each sentence. Given an input sequence t = (t 1 ,t 2 ,…,t m ), the output of BERT is h=(h 0 ,h 1 ,h 2 ,…,h m ),in is the final hidden state of the [CIS] tag, providing a summary representation of the entire input sequence, while h 1 ,h 2 ,…,h m is the final hidden state of the other tokens. In order to classify the input token sequence into K different text categories, the final hidden state h 0 It is input to the classification layer, and then a softmax operation is performed to convert the score of each text category into a probability: P = softmax (CW T ).in is the parameter matrix of the classification layer. In this way, BERT can effectively perform various NLP tasks.

[0019] Furthermore, in S2, the extracted features are processed by quantum circuits, which optimize the feature representation capability based on the characteristics of quantum superposition and entanglement. The classical features are mapped to quantum states using angle coding, specifically:

[0020] Quantum circuits use rotation gates to encode the angles of quantum bits, which is a common way to map classical information to quantum states. i Through the revolving door y (θ) is mapped to the quantum state: Among them, θ i By the classic feature h i In this way, the classical vector extracted by BERT is embedded in the quantum state. Multiple quantum bits are entangled through quantum entanglement gates (such as CNOT gates), thereby enhancing the ability to express features and using quantum superposition and entanglement properties to encode data more efficiently.

[0021] First, features are extracted from the input text through quantum processing: the BERT model, and then processed through a classical neural network layer to obtain the real part of the quantum circuit input. The real part of the input is encoded into a quantum state using ansatz. The quantum circuit then performs the following operations: Apply a rotation gate (RY) to encode the real part into a quantum state. Depending on the ansatz type, different quantum circuits are applied: 1.HEAnsatz: Apply a Hadamard gate to initialize the qubit to a superposition state. Apply a rotation gate (RZ) to each qubit based on the parameters. Apply a controlled rotation gate (CRY) between adjacent qubits. 2.QAOAAnsatz: Apply a Hadamard gate to initialize the qubit to a superposition state. Apply a rotation gate (RY) to each qubit. Apply a controlled NOT gate (CNOT) gate between adjacent qubits, followed by a rotation gate (RZ) and another set of CNOT gates.

[0022] Furthermore, in S3, the self-attention mechanism is applied in the quantum circuit to transform the input features into high-dimensional quantum states, calculate the query, key and value, and execute the self-attention mechanism through the quantum circuit to obtain better feature weight distribution, specifically:

[0023] Query, key and value calculation. In the classic self-attention mechanism, the attention distribution of query, key and value is calculated by the following formula In the quantum version, the quantum states of the query, key, and value are computed using quantum circuits, with each input quantum state represented as ψ i =α|0>+β|1>. The quantum inner product of query and key is used to calculate the attention distribution: The attention value is calculated through quantum circuits and combined with the classical attention mechanism to improve the accuracy and efficiency of the self-attention mechanism.

[0024] Furthermore, in S4, the complex quantum kernel method (CQKSAN) model is used to calculate the similarity between different input samples through quantum feature mapping and kernel function to complete text classification, specifically:

[0025] The quantum kernel function is calculated by the inner product of the two quantum states after the quantum feature mapping: κ(ψ(x i ),ψ(x j ))=|<ψ(x i )|ψ(x j )>| 2 The inner product is used to measure the similarity between samples. After feature mapping through quantum circuits, the quantum kernel function can better capture the complex relationship between high-dimensional data. CQKSAN achieves higher classification accuracy in text classification tasks through the optimized quantum kernel function.

[0026] Furthermore, in S5, the quantum transfer learning framework combines the advantages of classical neural networks and quantum circuits to improve the learning efficiency and accuracy of the model. Specifically:

[0027] Combining the feature extraction capabilities of classic pre-trained models (such as BERT) and the efficient processing capabilities of quantum circuits, the model training process and convergence speed are optimized. The final generated quantum state is measured to obtain the output value of the quantum state. The classic loss function is calculated by cross entropy: Among them, y i is a label, p i is the probability predicted by the model. In this framework, the classical model is responsible for preliminary feature extraction, while the quantum circuit encodes and enhances these features, ultimately improving the accuracy of text classification. Quantum transfer learning accelerates the convergence of the quantum model through the pre-trained classical model, and uses quantum transfer learning to improve the classification effect.

[0028] Compared with the prior art, the present invention has achieved the following beneficial technical effects:

[0029] 1. The text classification method based on quantum transfer learning proposed in this invention combines the high efficiency of quantum transfer learning and the feature extraction ability of the classical pre-training model, greatly improving the efficiency of text classification.

[0030] 2. Through the complex quantum kernel method, the model of the present invention can process high-dimensional feature representation, significantly improving the accuracy of text classification and the robustness of the model. It shows better learning ability when processing small-scale data sets, significantly shortens the convergence time of the model, and has higher generalization ability, which is suitable for a variety of natural language processing tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1A flowchart of a text recognition method algorithm based on quantum transfer learning provided by an embodiment of the present invention;

[0032] Figure 2 A quantum circuit framework diagram for text recognition based on quantum transfer learning provided by an embodiment of the present invention;

[0033] Figure 3 A framework diagram of a complex quantum kernel self-attention network (CQKSAN) model for text recognition based on quantum transfer learning provided in an embodiment of the present invention;

[0034] Figure 4 A diagram of simulation results of a complex quantum kernel self-attention network (CQKSAN) for text recognition based on quantum transfer learning provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] The purpose of the present invention is to provide a complex quantum kernel self-attention network (CQKSAN) model based on quantum transfer learning, which is used to solve the problems of insufficient efficiency and accuracy in existing text classification tasks.

[0037] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0038] Figure 1 A flowchart of a text recognition method algorithm based on quantum transfer learning provided by an embodiment of the present invention, such as Figure 1 As shown, the following steps are included:

[0039] S1: Build a pre-trained classic neural network (such as BERT) as a feature extractor, and extract features from the input text data through the pre-trained model:

[0040] The pre-trained model is the BERT model, which performs context-aware feature extraction on text data through a bidirectional encoder and is fine-tuned in downstream tasks to adapt to specific text classification tasks. The pre-trained BERT model provides powerful context-aware sentence representations that can be fine-tuned for a variety of downstream NLP tasks. Fine-tuning requires adjusting multiple hyperparameters, whose values ​​directly affect the performance of the model. The BERTbase model consists of 12 hidden layers, each with 768 hidden units and 12 attention heads, for a total of 110 million parameters. The BERTbase model accepts input sequences of up to 512 tokens, each represented by a specific embedding. These embeddings are used for a variety of NLP tasks, including syntactic analysis. The tokens in the input sequence can be case-sensitive or case-insensitive, resulting in two different pre-trained models. The bidirectionality of BERT enables it to encode rich contextual information internally and adapt to specific tasks in the fine-tuning stage, making it a base model in recent years.

[0041] Before the token sequence is fed into BERT, a special [CLS] token is added to the front of the sequence to obtain the vector representation of the entire sequence. Another special token [SEP] is placed at the end of each sentence. Given an input sequence t = (t 1 ,t 2 ,…,t m ), the output of BERT is h=(h 0 ,h 1 ,h 2 ,…,h m ),in is the final hidden state of the [CIS] tag, providing a summary representation of the entire input sequence, while h 1 ,h 2 ,…,h m is the final hidden state of the other tokens. In order to classify the input token sequence into K different text categories, the final hidden state h 0 It is input to the classification layer, and then a softmax operation is performed to convert the score of each text category into a probability: P = softmax9CW T ).in is the parameter matrix of the classification layer. In this way, BERT can effectively perform various NLP tasks.

[0042] S2: The extracted features are processed through quantum circuits. Quantum circuits optimize the feature representation capability based on quantum superposition and entanglement characteristics. Use angle encoding to map classical features to quantum states:

[0043] Quantum circuits use rotation gates to encode the angles of quantum bits, which is a common way to map classical information to quantum states. i Through the revolving door y (θ) is mapped to the quantum state: Among them, θ i By the classic feature h i In this way, the classical vector extracted by BERT is embedded in the quantum state. Multiple quantum bits are entangled through quantum entanglement gates (such as CNOT gates), thereby enhancing the ability to express features and using quantum superposition and entanglement properties to encode data more efficiently.

[0044] Specifically, Figure 2 As shown, in this embodiment, features are first extracted from the input text by quantum processing: the BERT model, and then processed by a classical neural network layer to obtain the real part of the quantum circuit input. The real part of the input is encoded as a quantum state using ansatz. The quantum circuit then performs the following operations: a rotation gate (RY) is applied to encode the real part as a quantum state. Depending on the ansatz type, different quantum circuits are applied HEAnsatz: a Hadamard gate is applied to initialize the quantum bit to a superposition state. A rotation gate (RZ) is applied to each quantum bit based on a parameter. A controlled rotation gate (CRY) is applied between adjacent quantum bits. QAOAAnsatz: a Hadamard gate is applied to initialize the quantum bit to a superposition state. A rotation gate (RY) is applied to each quantum bit. A controlled NOT gate (CNOT) gate is applied between adjacent quantum bits, followed by a rotation gate (RZ) and another set of CNOT gates.

[0045] S3: Apply the self-attention mechanism in quantum circuits to transform input features into high-dimensional quantum states, calculate queries, keys, and values, and execute the self-attention mechanism through quantum circuits to obtain better feature weight distribution.

[0046] Query, key and value calculation. In the classic self-attention mechanism, the attention distribution of query, key and value is calculated by the following formula In the quantum version, the quantum states of the query, key, and value are computed using quantum circuits, with each input quantum state represented as ψ i =α|0>+β|1>. The quantum inner product of query and key is used to calculate the attention distribution: The attention value is calculated through quantum circuits and combined with the classical attention mechanism to improve the accuracy and efficiency of the self-attention mechanism.

[0047] S4: Using the complex quantum kernel method (CQKSAN) model, the similarity between different input samples is calculated through quantum feature mapping and kernel function to complete text classification:

[0048] Specifically, Figure 3 As shown in the model structure diagram, in this embodiment, the quantum kernel function is calculated by the inner product of the two quantum states after the quantum feature mapping: κ(ψ(x i ),ψ(x j ))=|<ψ(x i )|ψ(x j )>| 2 The inner product is used to measure the similarity between samples. After feature mapping through quantum circuits, the quantum kernel function can better capture the complex relationship between high-dimensional data. CQKSAN achieves higher classification accuracy in text classification tasks through the optimized quantum kernel function.

[0049] S5: Through the quantum transfer learning framework, we combine the advantages of classical neural networks and quantum circuits to improve the learning efficiency and accuracy of the model:

[0050] The final generated quantum state is measured to obtain the output value of the quantum state. The classic loss function is calculated by cross entropy: Among them, y i is a label, p i is the probability predicted by the model. In this framework, the classical model is responsible for preliminary feature extraction, while the quantum circuit encodes and enhances these features, ultimately improving the accuracy of text classification. Quantum transfer learning accelerates the convergence of the quantum model through the pre-trained classical model, and uses quantum transfer learning to improve the classification effect.

[0051] Specifically, Figure 4 The simulation results of text recognition under different algorithms are described. The complex quantum kernel self-attention network for text recognition based on quantum transfer learning can achieve the best recognition performance. When processing the same data set, the accuracy rate reaches 97.97% of the peak training accuracy, which is better than the 1.12% of the BERT model.

[0052] The above is a detailed introduction to the text recognition method based on quantum transfer learning provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other.

[0053] The contents not described in detail in the specification of the present invention belong to the prior art known to the professional and technical personnel in this field.

Claims

1. A text recognition method based on quantum transfer learning, characterized in that: The following steps are involved: S1: Build a pre-trained classic neural network (such as BERT) as a feature extractor, and extract features from the input text data through the pre-trained model; S2: The extracted features are processed through quantum circuits. The quantum circuits optimize the feature representation capability based on the characteristics of quantum superposition and entanglement. Use angle encoding to map classical features to quantum states; S3: Apply self-attention mechanism in quantum circuits to transform input features into high-dimensional quantum states, calculate queries, keys and values, and perform self-attention mechanism through quantum circuits to obtain better feature weight distribution; S4: Using the complex quantum kernel method (CQKSAN) model, the similarity between different input samples is calculated through quantum feature mapping and kernel function to complete text classification; S5: Through the quantum transfer learning framework, we combine the advantages of classical neural networks and quantum circuits to improve the learning efficiency and accuracy of the model.

2. The text recognition method based on quantum transfer learning according to claim 1, characterized in that: In S1, a pre-trained classical neural network (such as BERT) is constructed as a feature extractor as follows: The pre-trained model is the BERT model, which performs context-aware feature extraction on text data through a bidirectional encoder and is fine-tuned in downstream tasks to adapt to specific text classification tasks; the pre-trained BERT model provides powerful context-aware sentence representations that can be fine-tuned for various downstream NLP tasks; Fine-tuning requires adjusting multiple hyperparameters, whose values ​​directly affect the performance of the model; the BERTbase model consists of 12 hidden layers, each with 768 hidden units and 12 attention heads, for a total of 110 million parameters; the BERTbase model accepts input sequences of up to 512 tokens, each represented by a specific embedding; these embeddings are used for a variety of NLP tasks, including syntactic parsing; the tokens in the input sequence can be case-sensitive or case-insensitive, resulting in two different pre-trained models; BERT's bidirectionality enables it to encode rich contextual information internally and adapt to specific tasks during the fine-tuning phase, making it a base model in recent years; Before the token sequence is fed into BERT, a special [CLS] tag is added to the front of the sequence to obtain the vector representation of the entire sequence; another special tag [SEP] is placed at the end of each sentence. Given an input sequence t = (t1, t2, …, t m ), the output of BERT is h=(h0,h1,h2,…,h m ),in is the final hidden state of the [CIS] tag, providing a summary representation of the entire input sequence, while h1,h2,…,h m is the final hidden state of the other tags; in order to classify the input tag sequence into K different text categories, the final hidden state h0 is input to the classification layer, and then a softmax operation is performed to convert the score of each text category into a probability: P = softmax (CW T );in is the parameter matrix of the classification layer; in this way, BERT can effectively perform various NLP tasks.

3. The text recognition method based on quantum transfer learning according to claim 1, characterized in that: In S2, the feature data output by the classic BERT is converted into quantum states as follows: Quantum circuits use the rotation gate of quantum bits to encode angles, which is a common way to map classical information to quantum states; the classical feature h i Through the revolving door y (θ) is mapped to the quantum state: Among them, θ i By the classic feature h i In this way, the classical vector extracted by BERT is embedded into the quantum state; multiple quantum bits are entangled through quantum entanglement gates (such as CNOT gates), thereby enhancing the feature expression capability and using quantum superposition and entanglement properties to encode data more efficiently; First, through quantum processing: the BERT model extracts features from the input text, which is then processed through a classical neural network layer to obtain the real part of the quantum circuit input; the real part of the input is encoded as a quantum state using ansatz; the quantum circuit then performs the following operations: a rotation gate (RY) is applied to encode the real part as a quantum state; depending on the ansatz type, different quantum circuits are applied:

1. HEAnsatz: Apply a Hadamard gate to initialize the quantum bit to a superposition state; Apply a rotation gate (RZ) to each quantum bit based on the parameters; Apply a controlled rotation gate (CRY) between adjacent quantum bits; 2. QAOAAnsatz: Apply a Hadamard gate to initialize the quantum bit to a superposition state; Apply a rotation gate (RY) to each quantum bit; Apply a controlled NOT gate (CNOT) gate between adjacent quantum bits, followed by a rotation gate (RZ) and another set of CNOT gates.

4. The text recognition method based on quantum transfer learning according to claim 1, characterized in that: In S3, the input features are converted into high-dimensional quantum states, the query, key and value are calculated, and the self-attention mechanism is executed through quantum circuits to obtain better feature weight distribution, specifically: Query, key and value calculation; in the classic self-attention mechanism, the attention distribution of query, key and value is calculated by the following formula In the quantum version, the quantum states of the query, key, and value are computed using quantum circuits, with each input quantum state represented as ψ i =α|0>+β|1>; the quantum inner product of query and key is used to calculate the attention distribution: The attention value is calculated through quantum circuits and combined with the classical attention mechanism to improve the accuracy and efficiency of the self-attention mechanism.

5. The text recognition method based on quantum transfer learning according to claim 1, characterized in that: In S4, the complex quantum kernel method (CQKSAN) model is used to calculate the similarity between different input samples through quantum feature mapping and kernel function to complete text classification, specifically: The quantum kernel function is calculated by the inner product of the two quantum states after the quantum feature mapping: κ(ψ(x i ),ψ(x j ))=|<ψ(x i )|ψ(x j )>| 2 ; The inner product is used to measure the similarity between samples. After feature mapping through quantum circuits, the quantum kernel function can better capture the complex relationship between high-dimensional data; CQKSAN achieves higher classification accuracy in text classification tasks through the optimized quantum kernel function.

6. The text recognition method based on quantum transfer learning according to claim 1 is characterized in that: In S5, the quantum transfer learning framework combines the advantages of classical neural networks and quantum circuits to improve the learning efficiency and accuracy of the model. The specific methods are as follows: The final generated quantum state is measured to obtain the output value of the quantum state; the classic loss function is calculated by cross entropy: Among them, y i is a label, p i is the probability predicted by the model; in this framework, the classical model is responsible for preliminary feature extraction, while the quantum circuit encodes and enhances these features in quantum states, ultimately improving the accuracy of text classification; quantum transfer learning accelerates the convergence of the quantum model through the pre-trained classical model, and uses quantum computing to improve the classification effect.

Images