Quantum fusion-based neural network model distillation method and device

Abstract

This invention provides a method and apparatus for distilling neural network models based on quantum fusion, belonging to the field of model compression technology. The method for distilling neural network models based on quantum fusion of this invention constructs a quantum mechanical-like mathematical computation framework to map the single-dimensional real-number features of traditional neural networks into complex-number domain states. By utilizing the rich dimensions of complex space, it effectively preserves the subtle structural information and directional differences in the feature data that are easily lost. By simulating the quantum entanglement channel mechanism, it establishes the feature interaction relationship between the teacher model and the student model, and constructs a loss function with a global optimization perspective and sensitivity to rapid response for model parameter updates, thereby improving the convergence speed of the student model. The resulting lightweight model can achieve prediction accuracy and robustness close to that of large-scale teaching models while significantly reducing computational overhead.

Description

Technical Field

[0001] This invention relates to the field of model compression technology, and in particular to a method and apparatus for distilling neural network models based on quantum fusion. Background Technology

[0002] With the development of deep learning technology, deep neural networks have achieved remarkable results in fields such as computer vision and natural language processing. However, high-performance models are usually accompanied by a huge number of parameters and computational overhead, making them difficult to deploy on mobile or edge devices. To address this, knowledge distillation (KD) technology has emerged, which aims to transfer knowledge from a large model (teacher model) to a lightweight small model (student model).

[0003] Current mainstream knowledge distillation methods, such as those based on Softmax temperature scaling, primarily align the probability distributions of teacher and student outputs using KL divergence. However, traditional probability distribution alignment often focuses only on the logical relationships at the output layer, neglecting the rich structural information contained in the intermediate layer features, leading to the loss of implicit knowledge. Secondly, existing feature fusion methods mostly employ linear weighting or Euclidean distance metrics, making it difficult to capture the complex nonlinear interactions between teacher and student features.

[0004] Therefore, there is an urgent need for an efficient knowledge distillation method that can handle nonlinear characteristic relationships in order to solve the above-mentioned technical problems. Summary of the Invention

[0005] This invention provides a method and apparatus for distilling neural network models based on quantum fusion, which addresses the shortcomings of existing technologies in identifying feature relationships, thereby improving the convergence speed, prediction accuracy, and robustness of student models.

[0006] This invention provides a method for distilling neural network models based on quantum fusion, comprising: Acquire training data samples and input the training data samples into the pre-trained teacher network model and the student network model to be trained, respectively, to obtain teacher feature data and student feature data; The teacher feature data and the student feature data are subjected to quantum state encoding processing to map the real number domain feature data to the complex number domain data, and the teacher quantum state vector and the student quantum state vector are constructed respectively. Based on the teacher quantum state vector and the student quantum state vector, a simulated quantum entanglement channel is constructed to generate entangled state data representations containing the interaction information of the two networks. Based on the preset quantum superposition rule, and combining the teacher quantum state vector, student quantum state vector, and entangled state data representation, a fused quantum state is generated; The fused quantum state is simulated using a preset observable operator to calculate the expected measurement value, and a distillation loss function is constructed based on the expected measurement value. The gradient is calculated based on the distillation loss function to update the parameters of the student network model.

[0007] According to the quantum fusion-based neural network model distillation method provided by the present invention, the quantum state encoding processing of the teacher feature data and the student feature data specifically includes: Construct an encoder containing complex projection layers; The complex projection layer projects the input features of the teacher feature data and the student feature data into mixed features of the dimensionality; The hybrid feature is divided into real vectors and imaginary vectors; The real and imaginary vectors are normalized to construct a complex vector that satisfies the normalization condition, which serves as the quantum state vector.

[0008] According to the present invention, a distillation method for a neural network model based on quantum fusion is provided, wherein constructing a simulated quantum entanglement channel based on the teacher quantum state vector and the student quantum state vector specifically includes: Calculate the similarity matrix between the teacher quantum state vector and the conjugate transpose of the student quantum state vector; The similarity matrix is ​​then subjected to temperature scaling. Normalizing the scaled matrix yields an entanglement operator that represents the degree of feature correlation between the teacher network model and the student network model.

[0009] According to the present invention, a neural network model distillation method based on quantum fusion is provided, wherein the simulated quantum measurement of the fused quantum state is performed using a preset observable operator, specifically including: Construct an observable matrix representing the direction of objective optimization; The inner product of the fused quantum state vector and the observable matrix is ​​calculated to obtain the expected value; The negative real part of the expected value is used as part of the distillation loss function.

[0010] According to the quantum fusion-based neural network model distillation method provided by the present invention, the total loss function used when updating the parameters of the student network model is determined based on the basic loss of the student network model on a specific task, the quantum distillation loss calculated based on the measurement expectation value, and the regularization loss of the constraint entanglement strength.

[0011] According to the present invention, a neural network model distillation method based on quantum fusion is provided, the method being applied to image processing; the teacher network model and the student network model are convolutional neural network models or visual Transformer models; The training data samples are image data, and the feature data are intermediate layer feature maps or output layer classification probability vectors of the images.

[0012] According to the present invention, a distillation method for a neural network model based on quantum fusion is provided, which is applied to natural language processing; the training data sample is a text sequence, and the feature data is a word embedding vector or a context semantic vector of the text.

[0013] The present invention also provides a distillation apparatus based on a quantum fusion neural network model, comprising: The acquisition module is used to acquire training data samples and input the training data samples into the pre-trained teacher network model and the student network model to be trained, respectively, to obtain teacher feature data and student feature data. The encoding module is used to perform quantum state encoding processing on the teacher feature data and the student feature data, mapping the real number domain feature data to complex number domain data, and constructing teacher quantum state vectors and student quantum state vectors respectively. The first processing module is used to construct a simulated quantum entanglement channel based on the teacher's quantum state vector and the student's quantum state vector, and generate an entangled state data representation containing the interaction information of the two networks' features. The second processing module is used to generate a fused quantum state based on a preset quantum superposition rule, combined with the teacher quantum state vector, the student quantum state vector, and the entangled state data representation. The parameter update module is used to perform simulated quantum measurement on the fused quantum state using a preset observable operator, calculate the expected measurement value, and construct a distillation loss function based on the expected measurement value; it also calculates the gradient based on the distillation loss function and updates the parameters of the student network model. The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the quantum fusion-based neural network model distillation method as described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the distillation method of a neural network model based on quantum fusion as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the quantum fusion-based neural network model distillation method as described above.

[0016] The present invention provides a quantum fusion-based neural network model distillation method and apparatus. By constructing a quantum mechanical-like mathematical computation framework, it maps the single-dimensional real-number features of traditional neural networks to complex-number domain states. By utilizing the rich dimensions of complex space, it effectively preserves the subtle structural information and directional differences in the feature data that are easily lost. By simulating the quantum entanglement channel mechanism, it establishes the feature interaction relationship between the teacher model and the student model, and constructs a loss function with a global optimization perspective and sensitivity to rapid response for model parameter updates. This improves the convergence speed of the student model. The resulting lightweight model achieves prediction accuracy and robustness close to that of large-scale teaching models while significantly reducing computational overhead. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a schematic flowchart of the distillation method for a neural network model based on quantum fusion provided by the present invention; Figure 2 This is a schematic diagram of the structure of the distillation device based on a quantum fusion neural network model provided by the present invention; Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0020] The following is combined Figures 1-3 The present invention describes a quantum fusion-based neural network model distillation method and apparatus.

[0021] It should be noted that the core of this invention lies in utilizing the computing power of classical computers to construct a quantum-mechanical-like mathematical model framework. In the description of this invention, terms such as quantum state, entanglement, and measurement, unless specifically referring to physical qubits, all refer to complex field vector operations, matrix transformations, and probability statistics processes running on classical processors such as CPUs and GPUs.

[0022] like Figure 1 As shown, the quantum fusion-based neural network model distillation method of this invention can be applied to computer systems, and mainly includes steps 110, 120, 130, 140 and 150.

[0023] Step 110: Obtain training data samples and input them into the pre-trained teacher network model and the student network model to be trained, respectively, to obtain teacher feature data and student feature data.

[0024] Teacher network models refer to pre-trained models with complex structures, a large number of parameters, and high accuracy. Student network models refer to lightweight models with simplified structures, a small number of parameters, and areas for optimization.

[0025] Feature data is a numerical matrix generated in the intermediate layers of a neural network when the network model processes input data. It represents the network's understanding of the data, such as texture, shape, and semantics.

[0026] After starting the data loading program, training data samples can be read in batches from the storage medium. These samples can be images, text, or speech signals. The same training data sample can be simultaneously input into both the teacher network in evaluation mode and the student network in training mode. The two networks compute in parallel: the teacher network outputs the numerical vectors of its deep feature layers as teacher feature data; the student network outputs the numerical vectors of its corresponding layers as student feature data. Alternatively, considering the high computational cost of the teacher network, it can be pre-run to process all training data, storing the extracted teacher feature data in a hard disk database. When training the student network, only student feature data is computed in real time, and the corresponding teacher feature data is retrieved from the database for matching.

[0027] Understandably, by extracting intermediate layer features rather than just the final result, student models can learn the thought process of teacher models, rather than just the output content.

[0028] In one example, in an image processing scenario, the system can read data in batches from a pre-built disk database, such as the ImageNet local dataset, using a data loader. For instance, after randomly cropping and normalizing the images, tensors are generated. These tensors from the same batch are simultaneously input into the teacher and student models loaded in GPU memory. The 1024-dimensional real vector output from the penultimate layer of the teacher model (e.g., before Logits) can be extracted as the teacher's feature data. Similarly, the 512-dimensional vector from the corresponding layer of the student model can be extracted and up-aligned to 1024 dimensions using a fully connected layer to serve as the student's feature data.

[0029] In another example, in an edge computing scenario, a camera captures video stream data in real time. The system can extract keyframes from the video frames and use them directly as training samples. The teacher model can be deployed in the cloud, and the student model can be deployed at the edge. The cloud computes and transmits back teacher feature data, such as a feature map, while the edge computes student feature data in real time. At this point, the feature data is a three-dimensional tensor (number of channels, height, width), which is then compressed into a one-dimensional feature vector through global average pooling for subsequent processing.

[0030] Step 120: Perform quantum state encoding processing on teacher feature data and student feature data, mapping the feature data in the real number field to the data in the complex number field, and constructing teacher quantum state vectors and student quantum state vectors respectively.

[0031] Quantum state encoding is the process of mapping the real-number features output by a traditional neural network into complex-number forms through specific mathematical transformations. In analog systems, complex vectors are treated as quantum states.

[0032] Complex data consists of a data structure containing both real and imaginary parts. Classical neural networks deal with real numbers, while the states described by quantum mechanics are typically located in complex Hilbert space, allowing for the construction of mappings from real space to complex space.

[0033] A learnable linear projection layer can be used to expand the input N-dimensional features to 2N dimensions. Defining the first N dimensions as the real part of a complex number and the last N dimensions as the imaginary part of a complex number allows the neural network to automatically learn how to distribute the features across the complex plane.

[0034] In some embodiments, quantum state encoding processing is performed on teacher feature data and student feature data, specifically including: constructing an encoder containing a complex projection layer; projecting the input features of the teacher feature data and student feature data into mixed features of the dimensionality through the complex projection layer; dividing the mixed features into real vectors and imaginary vectors; normalizing the real vectors and imaginary vectors to construct a complex vector that satisfies the normalization condition, which serves as the quantum state vector.

[0035] In deep learning frameworks for computer systems, a submodule of complex projection layers can be constructed. This module can be a fully connected layer or a convolutional layer, with the number of output nodes set to twice the number of input nodes.

[0036] Suppose the input features are vectors of dimension D. After passing through the projection layer described above, the data becomes a mixed feature vector of dimension 2D. In this process, the neural network can automatically learn how to distribute the original features into this 2D space; that is, the neural network adaptively seeks the optimal complex representation.

[0037] When the system performs a slicing operation, it divides a 2D vector in half from the middle. The first half (index 0 to D-1) is designated as the real part vector, and the second half (index D to 2D-1) is designated as the imaginary part vector.

[0038] In quantum mechanics, quantum states must satisfy the normalization condition, meaning the total probability of a particle appearing is 1. Mathematically, this corresponds to the modulus (L2 norm) of a complex vector being 1.

[0039] In one example, global normalization can be used. The magnitude of the entire complex vector can be calculated by first calculating the sum of squares of the real parts, then the sum of squares of the imaginary parts, adding them together and taking the square root. Then, divide the real and imaginary vectors by this magnitude respectively.

[0040] In another example, component normalization can be used. For each feature dimension, its magnitude can be calculated and normalized separately, so that the complex number of each dimension lies on the unit circle.

[0041] It should be noted that normalization is not only used to mimic physical principles. In deep learning, it can also serve as a regularization mechanism, thereby eliminating the instability caused by the magnitude of feature values, preventing gradient explosion, and allowing the model to focus on learning the direction and relative relationship of features, rather than the absolute magnitude of the values.

[0042] Of course, in some embodiments, amplitude-phase encoding or trigonometric function embedding can also be used for mapping, and there are no restrictions here.

[0043] Step 130: Based on the teacher's quantum state vector and the student's quantum state vector, construct a simulated quantum entanglement channel to generate an entangled state data representation containing the interaction information of the two networks.

[0044] It should be noted that entanglement channels are not physical connections, but rather a computational mechanism used to capture deep, non-local correlations between features of two models.

[0045] Since traditional deep learning frameworks process real numbers, they can input a real feature vector of length N into a specific mapping module. This module expands the N-dimensional vector to 2N dimensions, where the first N dimensions are defined as the real part of the complex number, representing the intensity (amplitude) of the feature; and the last N dimensions are defined as the imaginary part of the complex number, representing the phase (direction or periodicity information) of the feature.

[0046] In this embodiment, a simulated quantum entanglement channel is constructed based on the teacher quantum state vector and the student quantum state vector. Specifically, this includes: calculating the similarity matrix between the conjugate transposes of the teacher quantum state vector and the student quantum state vector; performing temperature scaling on the similarity matrix; and performing normalization on the scaled matrix to obtain an entanglement operator representing the degree of feature correlation between the teacher network model and the student network model.

[0047] It can obtain the normalized teacher quantum state vector and student quantum state vector. It can perform the conjugate transpose operation, which involves inverting the sign of the imaginary part of the student vector (making it negative), transposing it, and then calculating the dot product between the teacher vector and the processed student vector.

[0048] Furthermore, each value in the similarity matrix obtained in the previous step can be divided by a preset scalar coefficient, which is the temperature parameter T. In one example, T can be set to be greater than 1, such as T=5, which reduces the differences between similarity values, resulting in a smoother probability distribution. This allows the student model to focus on the overall macroscopic structure of the teacher model, rather than rote memorizing a particular feature point. In another example, T can be set to be less than 1, such as T=0.1, which amplifies the numerical differences, allowing the student model to focus on the key features that the teacher model considers most important.

[0049] Understandably, the temperature parameter provides a way to adjust the scope of attention, allowing the distillation process to flexibly adjust its strategy according to the training phase, such as first using high temperature for rough training and then using low temperature for fine-tuning.

[0050] Finally, the entanglement operator can be obtained through normalization. Softmax operation and exponential normalization can be performed on the scaled matrix to transform arbitrary values ​​into a probability distribution. In this case, the final matrix is ​​the entanglement operator. The entanglement operator is not merely data, but also an operational instruction, indicating how to adjust the student's state based on the teacher's state.

[0051] In another example, based on the tensor product, according to the definition of quantum mechanics, the tensor product of the teacher's quantum state vector and the student's quantum state vector can be calculated, thereby generating a joint state vector with extremely high dimensionality. To reduce computational cost, tensor network techniques or low-rank approximations can be used to represent the entangled state representation.

[0052] In one example, an association matrix can be constructed by calculating the dot product similarity or cosine similarity between the teacher's and student's quantum state vectors based on the soft entanglement of the similarity matrix. The association matrix is ​​used to represent the conditional probability that if the teacher focuses on feature A, the student should focus on feature B.

[0053] It is understandable that by introducing the complex number field, the feature representation space is expanded from a line (real number axis) to a surface (complex plane). This encoding method can accommodate more feature dimensions, not only representing the strength of features, but also representing the orthogonality or interference relationship between features through the imaginary part, thus preserving the subtle difference information that is easily lost in real number networks.

[0054] Step 140: Based on the preset quantum superposition rule, and combining the teacher's quantum state vector, the student's quantum state vector, and the entangled state data representation, a fused quantum state is generated.

[0055] A fused quantum state is a new, unified state vector formed by mixing information from teachers, students, and the relationships between them according to the principle of quantum superposition.

[0056] In one example, a set of weighting parameters can be set, such as 60% for teachers, 30% for students, and 10% for entanglement terms. These three parts of data can then be directly added in the complex domain. Since complex addition involves adding real parts to real parts and imaginary parts to imaginary parts, this simulates wave interference, which can enhance some features and cancel out others.

[0057] In another example, nonlinear gating superposition can be used by introducing a quantum gate-like control function that dynamically adjusts the fusion ratio based on the current training stage. For example, more teacher states (guiding direction) can be retained in the early stages of training, while more entangled states (fine-tuning details) can be retained in the later stages.

[0058] It should be noted that the fusion operation is not a simple splicing, but rather a simulation of the superposition and interference of wave functions, so that the final fused state can simultaneously retain the teacher's correct guidance and the students' own diversity, while eliminating the noise information that is inconsistent between the two.

[0059] Step 150: Perform simulated quantum measurement on the fused quantum state using a preset observable operator, calculate the expected measurement value, and construct a distillation loss function based on the expected measurement value; calculate the gradient based on the distillation loss function and update the parameters of the student network model.

[0060] In physics, an observable operator represents a measuring instrument; in mathematics, it can be a Hermitian matrix representing the ideal state that the system is expected to achieve, such as correct classification or feature matching.

[0061] The expected value of measurement refers to the average value obtained after measuring the fusion state, which reflects the degree of quality of the current system state.

[0062] In one example, an operator matrix representing a perfect match can be constructed. The inner product of the fused quantum state vector and this matrix is ​​calculated, and the result is a complex number. The square of its modulus or its real part is taken as the score.

[0063] In another example, the training objective can also be defined as the energy of the system. The average energy of the fused states under this matrix can be calculated by constructing a Hamiltonian matrix representing the error energy.

[0064] Finally, the calculated score or energy can be transformed into a loss function, and the gradient of this loss function with respect to the student network parameters can be calculated by automatic differentiation. The weights of the student network can then be updated using the gradient descent algorithm.

[0065] In this embodiment, the loss function constructed using quantum measurement principles exhibits more sensitive gradient response characteristics compared to traditional Euclidean distance or divergence. Especially when the student model closely approximates the teacher model, traditional methods tend to exhibit gradient vanishing, while the measured expectation value can still provide an effective optimization direction, thereby accelerating convergence.

[0066] The quantum fusion-based neural network model distillation method provided by the present invention constructs a quantum mechanical-like mathematical computation framework, mapping the single-dimensional real-number features of traditional neural networks to complex-number domain states. By utilizing the rich dimensions of complex space, it effectively preserves the subtle structural information and directional differences in the feature data that are easily lost. By simulating the quantum entanglement channel mechanism, it establishes the feature interaction relationship between the teacher model and the student model, constructs a loss function with a global optimization perspective and sensitivity to rapid response for model parameter updates, improves the convergence speed of the student model, and finally generates a lightweight model that can achieve prediction accuracy and robustness close to that of a large-scale teaching model while significantly reducing computational overhead.

[0067] In some embodiments, a preset observable operator is used to simulate quantum measurement of the fused quantum state, specifically including: constructing an observable matrix representing the target optimization direction; calculating the inner product of the fused quantum state vector and the observable matrix to obtain the expected value; and using the negative real part of the expected value as part of the distillation loss function.

[0068] For classification tasks, a diagonal matrix can be constructed as the observable matrix, where the diagonal elements corresponding to the correct class are set to 1, and the rest to 0, representing the measurement operator of the physical quantity of correct classification. Alternatively, an inverse matrix representing the distribution of teacher model features can be constructed as the observable matrix to represent the deviation between students and teachers.

[0069] The fused quantum state vector can be multiplied on the left by the observable matrix, and the result can be multiplied by the conjugate transpose of the fused quantum state vector to obtain a complex value, namely the expectation value. It represents the average result that can be obtained by performing this measurement in the current state.

[0070] Because the loss function in deep learning must be a real number, and the model is optimized by minimizing the loss, if the expected value represents the accuracy, its negative value (the negative real part) is taken as the loss. If the expected value represents the energy error, its real part is taken directly as the loss.

[0071] Understandably, this loss function based on expected value has global smoothness compared to traditional point-to-point error calculation. It can consider the weighted average of all possible feature states, and therefore has stronger robustness to noisy data.

[0072] In some embodiments, the total loss function used when updating the parameters of the student network model is determined based on the student network model’s base loss on a specific task, the quantum distillation loss calculated based on the expected value of the measurement, and the regularization loss of the constraint entanglement strength.

[0073] The basic loss is the loss incurred by the student model when performing its own task. For example, in image classification, if the student model predicts an image as a cat but the actual label is a dog, the resulting cross-entropy loss is significant. Considering the basic loss ensures that the student model can at least complete the basic task.

[0074] Quantum distillation loss is the loss generated by imitating the teacher's thinking. It can be a calculated measurement expectation loss, used to characterize the complex distribution of student characteristics that converges with the teacher's.

[0075] Entanglement strength regularization loss is used to prevent overfitting and requires limiting the complexity of entanglement. Specifically, the entropy or L1 norm of the entanglement operator matrix can be calculated. If the entanglement is too complex, the loss is increased, thereby prompting the model to learn the simplest and most essential feature associations, thus conforming to Occam's razor principle.

[0076] It is understandable that by adjusting the weight coefficients of these three parts, the relationship between independent learning and imitation learning can be balanced, while preventing the model from overfitting.

[0077] In some possible implementations, the quantum fusion-based neural network model distillation method of this invention is applied to image processing; the teacher network model and the student network model are convolutional neural network models or visual Transformer models; the training data samples are image data, and the feature data are intermediate layer feature maps of the image or output layer classification probability vectors.

[0078] This embodiment can be applied to intelligent photo categorization on mobile devices or real-time facial recognition for security cameras.

[0079] In one example, the teacher model could be an attention-based model, which excels at capturing global relationships in an image. For instance, determining whether a person is riding a bicycle requires simultaneous attention to both the person and the bicycle, but this is extremely slow and cannot be run on a mobile phone.

[0080] The student model can use a lightweight convolutional neural network, which is fast to compute, but is limited by the size of the convolutional kernel, and can usually only see local features, making it difficult to understand global relationships.

[0081] The specific form of the feature data can be an intermediate layer feature map. Both the teacher model and the student model output three-dimensional tensors. Each pixel or channel vector in the feature map can be regarded as a microscopic particle. Through quantum state encoding, the real value of each pixel is transformed into a complex wave function. Through entangled channels, the structural information of the global attention map unique to the teacher model is transferred to the student model in the form of quantum entanglement. In this way, although the small model on the mobile phone has a simple structure, it learns to see the whole image like the large model, thus significantly improving the recognition accuracy in complex backgrounds such as occlusion and low light.

[0082] In some possible implementations, the quantum fusion-based neural network model distillation method of this invention is applied to natural language processing; the training data samples are text sequences, and the feature data are word embedding vectors or context semantic vectors of the text.

[0083] This embodiment can be applied to offline voice command parsing or low-latency machine translation for smartwatches.

[0084] Training data can be a text sequence, such as "Set my alarm for tomorrow morning." Feature data can be word embedding vectors or contextual semantic vectors.

[0085] In natural language processing, the semantic intensity of word vectors, such as the emotional intensity of "like," can be mapped to the real part (amplitude) of a complex number, while the positional information of word vectors in a sentence can be mapped to the imaginary part (phase) of a complex number. Words in a sentence often exhibit long-distance dependencies (e.g., "although...but..."), and teacher models are adept at capturing these relationships. By constructing simulated quantum entanglement channels, these long-distance syntactic dependencies can be directly mapped to the parameters of the student model.

[0086] In this case, the problem that traditional small models are prone to losing context and cannot understand complex long sentences can be solved.

[0087] To further illustrate the operating environment of this invention, the following section briefly describes the execution logic on GPU hardware. Since the embodiments of this invention extensively utilize complex matrix operations, the PyTorch framework can be used in the specific code implementation. Instead of using two separate real matrices for simulation, the underlying cuComplex data type and cuBLAS linear algebra library of the unified computing device architecture are directly invoked. For example, when performing entangled matrix multiplication, the GPU's double-precision complex matrix multiplication instructions can be directly called. This allows the aforementioned complex simulated quantum computing process to be completed on the graphics card in a very short time, thus meeting the efficiency requirements for model training in industrial applications.

[0088] The following describes the quantum fusion-based neural network model distillation apparatus provided by the present invention. The quantum fusion-based neural network model distillation apparatus described below and the quantum fusion-based neural network model distillation method described above can be referred to and correspond to each other.

[0089] like Figure 2 As shown, the quantum fusion-based neural network model distillation device of this invention includes an acquisition module 210, an encoding module 220, a first processing module 230, a second processing module 240, and a parameter update module 250.

[0090] The acquisition module 210 is used to acquire training data samples and input the training data samples into the pre-trained teacher network model and the student network model to be trained, respectively, to obtain teacher feature data and student feature data. Encoding module 220 is used to perform quantum state encoding processing on teacher feature data and student feature data, mapping the feature data in the real number field to the data in the complex number field, and constructing teacher quantum state vectors and student quantum state vectors respectively; The first processing module 230 is used to construct a simulated quantum entanglement channel based on the teacher's quantum state vector and the student's quantum state vector, and generate entangled state data representation containing the interaction information of the two networks. The second processing module 240 is used to generate a fused quantum state based on a preset quantum superposition rule, combined with the teacher's quantum state vector, the student's quantum state vector, and entangled state data representation. The parameter update module 250 is used to simulate quantum measurement of the fused quantum state using a preset observable operator, calculate the expected value of the measurement, and construct a distillation loss function based on the expected value of the measurement; and calculate the gradient based on the distillation loss function to update the parameters of the student network model.

[0091] The quantum fusion-based neural network model distillation device provided in this embodiment of the invention constructs a quantum mechanical-like mathematical computation framework, mapping the single-dimensional real-number features of traditional neural networks to complex-number domain states. By utilizing the rich dimensions of complex space, it effectively preserves the subtle structural information and directional differences in the feature data that are easily lost. By simulating the quantum entanglement channel mechanism, it establishes the feature interaction relationship between the teacher model and the student model, constructs a loss function with a global optimization perspective and sensitivity to rapid response for model parameter updates, improves the convergence speed of the student model, and finally generates a lightweight model that can achieve prediction accuracy and robustness close to that of a large-scale teaching model while significantly reducing computational overhead.

[0092] Figure 3 illustrates a schematic diagram of the physical structure of an electronic device. As shown in Figure 3, the electronic device may include: a processor 310, a communication interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other through the communication bus 340. The processor 310 can call logic instructions in the memory 330 to execute a distillation method for a neural network model based on quantum fusion. This method includes: acquiring training data samples and inputting them into a pre-trained teacher network model and a student network model to be trained, respectively, to obtain teacher feature data and student feature data; performing quantum state encoding on the teacher and student feature data, mapping the real-domain feature data to complex-domain data, and constructing teacher quantum state vectors and student quantum state vectors respectively; constructing a simulated quantum entanglement channel based on the teacher and student quantum state vectors to generate an entangled state data representation containing interaction information between the two networks; generating a fused quantum state based on a preset quantum superposition rule, combining the teacher quantum state vector, student quantum state vector, and entangled state data representation; performing simulated quantum measurement on the fused quantum state using a preset observable operator, calculating the expected measurement value, and constructing a distillation loss function based on the expected measurement value; and calculating the gradient based on the distillation loss function to update the parameters of the student network model.

[0093] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0094] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the quantum fusion-based neural network model distillation method provided by the above methods. The method includes: acquiring training data samples and inputting the training data samples into a pre-trained teacher network model and a student network model to be trained, respectively, to obtain teacher feature data and student feature data; performing quantum state encoding processing on the teacher feature data and student feature data, mapping the feature data in the real number domain to the complex number domain data, and constructing teacher quantum state vectors and student quantum state vectors, respectively; constructing a simulated quantum entanglement channel based on the teacher quantum state vector and student quantum state vector, and generating an entangled state data representation containing the interaction information of the two network features; generating a fused quantum state based on a preset quantum superposition rule, combining the teacher quantum state vector, student quantum state vector, and entangled state data representation; performing simulated quantum measurement on the fused quantum state using a preset observable operator, calculating the measurement expectation value, and constructing a distillation loss function based on the measurement expectation value; and calculating the gradient based on the distillation loss function and updating the parameters of the student network model.

[0095] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the distillation method for a quantum fusion-based neural network model provided by the above methods. The method includes: acquiring training data samples and inputting the training data samples into a pre-trained teacher network model and a student network model to be trained, respectively, to obtain teacher feature data and student feature data; performing quantum state encoding processing on the teacher feature data and student feature data, mapping the feature data in the real number domain to data in the complex number domain, and constructing teacher quantum state vectors and student quantum state vectors, respectively; constructing a simulated quantum entanglement channel based on the teacher quantum state vector and student quantum state vector, and generating an entangled state data representation containing the interaction information of the two network features; generating a fused quantum state based on a preset quantum superposition rule, combining the teacher quantum state vector, student quantum state vector, and entangled state data representation; performing simulated quantum measurement on the fused quantum state using a preset observable operator, calculating the measurement expectation value, and constructing a distillation loss function based on the measurement expectation value; and calculating the gradient based on the distillation loss function and updating the parameters of the student network model.

[0096] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0097] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A distillation method for a neural network model based on quantum fusion, characterized in that, include: Acquire training data samples and input the training data samples into the pre-trained teacher network model and the student network model to be trained, respectively, to obtain teacher feature data and student feature data; The teacher feature data and the student feature data are subjected to quantum state encoding processing to map the real number domain feature data to the complex number domain data, and the teacher quantum state vector and the student quantum state vector are constructed respectively. Based on the teacher quantum state vector and the student quantum state vector, a simulated quantum entanglement channel is constructed to generate entangled state data representations containing the interaction information of the two networks. Based on the preset quantum superposition rule, and combining the teacher quantum state vector, student quantum state vector, and entangled state data representation, a fused quantum state is generated; The fused quantum state is simulated using a preset observable operator to calculate the expected measurement value, and a distillation loss function is constructed based on the expected measurement value. The gradient is calculated based on the distillation loss function to update the parameters of the student network model.

2. The distillation method for a neural network model based on quantum fusion according to claim 1, characterized in that, The quantum state encoding process for the teacher feature data and the student feature data specifically includes: Construct an encoder containing complex projection layers; The complex projection layer projects the input features of the teacher feature data and the student feature data into mixed features of the dimensionality; The hybrid feature is divided into real vectors and imaginary vectors; The real and imaginary vectors are normalized to construct a complex vector that satisfies the normalization condition, which serves as the quantum state vector.

3. The distillation method for a neural network model based on quantum fusion according to claim 1, characterized in that, The construction of a simulated quantum entanglement channel based on the teacher's quantum state vector and the student's quantum state vector specifically includes: Calculate the similarity matrix between the teacher quantum state vector and the conjugate transpose of the student quantum state vector; The similarity matrix is ​​then subjected to temperature scaling. Normalizing the scaled matrix yields an entanglement operator that represents the degree of feature correlation between the teacher network model and the student network model.

4. The distillation method for a neural network model based on quantum fusion according to claim 1, characterized in that, The step of simulating quantum measurement of the fused quantum state using a preset observable operator specifically includes: Construct an observable matrix representing the direction of objective optimization; The inner product of the fused quantum state vector and the observable matrix is ​​calculated to obtain the expected value; The negative real part of the expected value is used as part of the distillation loss function.

5. The distillation method for a neural network model based on quantum fusion according to claim 1, characterized in that, The total loss function used when updating the parameters of the student network model is determined based on the student network model's base loss on a specific task, the quantum distillation loss calculated based on the measured expectation value, and the regularization loss of the constraint entanglement strength.

6. The distillation method for a neural network model based on quantum fusion according to claim 1, characterized in that, The method is applied to image processing; the teacher network model and the student network model are convolutional neural network models or visual Transformer models. The training data samples are image data, and the feature data are intermediate layer feature maps or output layer classification probability vectors of the images.

7. The distillation method for a neural network model based on quantum fusion according to claim 1, characterized in that, The method is applied to natural language processing; the training data samples are text sequences, and the feature data are word embedding vectors or context semantic vectors of the text.

8. A distillation apparatus based on a quantum fusion neural network model, characterized in that, include: The acquisition module is used to acquire training data samples and input the training data samples into the pre-trained teacher network model and the student network model to be trained, respectively, to obtain teacher feature data and student feature data. The encoding module is used to perform quantum state encoding processing on the teacher feature data and the student feature data, mapping the real number domain feature data to complex number domain data, and constructing teacher quantum state vectors and student quantum state vectors respectively. The first processing module is used to construct a simulated quantum entanglement channel based on the teacher's quantum state vector and the student's quantum state vector, and generate an entangled state data representation containing the interaction information of the two networks' features. The second processing module is used to generate a fused quantum state based on a preset quantum superposition rule, combined with the teacher quantum state vector, the student quantum state vector, and the entangled state data representation. The parameter update module is used to perform simulated quantum measurement on the fused quantum state using a preset observable operator, calculate the expected measurement value, and construct a distillation loss function based on the expected measurement value; calculate the gradient based on the distillation loss function, and update the parameters of the student network model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, When the processor executes the program, it implements the quantum fusion-based neural network model distillation method as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the quantum fusion-based neural network model distillation method as described in any one of claims 1 to 7.

Images