A method for sentiment classification of teaching review short texts based on distributed approximate homomorphic encryption

Abstract

The present invention provides a method for sentiment classification of teaching evaluation short texts based on distributed approximate homomorphic encryption, which relates to the field of educational technology. The method uses distributed approximate homomorphic encryption to encrypt input teaching evaluation short text data; the encrypted short text data is input into an ALBERT‑Mixplus fusion model deployed in the cloud for feature extraction and sentiment classification; the sentiment classification result in the encrypted state is decrypted on the local user side, and the final classification result is output. Through the use of distributed approximate homomorphic encryption, the present invention enables data and models to complete the required computing tasks in an encrypted environment during the cloud sentiment classification task. The ALBERT‑Mixplus fusion model fully utilizes the lightweight feature extraction advantage of the ALBERT model, and combines the powerful feature extraction capabilities of BiGRU and MCNN, so that sentiment classification still has efficient performance in an encrypted environment, improving the accuracy and processing efficiency of sentiment classification.

Description

Technical Field

[0001] The present invention relates to the field of educational technology, and in particular to a method for sentiment classification of teaching review short texts based on distributed approximate homomorphic encryption. Background Art

[0002] In the process of teaching evaluation in colleges and universities, teachers and students often generate a large amount of text data, which contains subjective evaluations of teaching quality, course content, teaching methods, etc. These teaching evaluation data not only have important analytical value, but can also be used to improve teaching methods and enhance the quality of education. However, these data often contain sensitive personal information, such as students' evaluation of teachers and teachers' views on courses, so special attention should be paid to privacy protection during data processing and analysis.

[0003] With the development of information technology, more and more universities choose to deploy data storage and computing tasks in the local cloud to ensure data security. However, due to the large amount and complex structure of university teaching evaluation data, in order to improve computing efficiency and reduce costs, many universities need to use the computing resources of the public cloud to complete complex tasks such as sentiment classification. In this case, how to process and store sensitive data in the cloud and ensure that data privacy is not leaked has become an urgent problem to be solved.

[0004] Homomorphic encryption is a technology that can perform calculations on data in an encrypted state. It allows addition and multiplication operations to be performed directly on ciphertext without decryption, thereby achieving data privacy protection in the cloud. Fully Homomorphic Encryption (FHE) is an encryption technology that can support any number of addition and multiplication homomorphic operations, and is considered an important breakthrough in the field of privacy protection. At present, FHE technology has achieved remarkable results in fields such as image encryption, and can effectively prevent data from being stolen or tampered with in the cloud.

[0005] However, in the field of natural language processing, especially in text sentiment classification methods based on pre-trained models (such as Transformer and BERT), the application of fully homomorphic encryption (FHE) technology still faces many challenges. Although fully homomorphic encryption (FHE) supports addition and multiplication operations and can be used to construct polynomial functions, it has limited support for non-polynomial operations. In particular, in text sentiment classification tasks, many key operations such as maximum (MAX), minimum (MIN), and functions such as Softmax, GELU, and LayerNorm cannot be directly calculated through FHE. These non-polynomial functions play a key role in self-attention models such as Transformer and BERT. Their existence makes it difficult for these models to directly apply FHE for encryption calculations, resulting in the inability to effectively protect data privacy in sentiment classification tasks in the cloud.

[0006] In addition, short texts of university teaching evaluation have unique characteristics. These short texts are often highly subjective and diverse, and often contain distinctive network cultural characteristics, such as emoticons, abbreviations, and network slang, which leads to a lot of noise in the data. In addition, these short texts usually lack contextual information, and the correlation between individual texts is weak, which brings great challenges to the sentiment classification task.

[0007] Although the existing deep learning-based sentiment classification methods for teaching and review short texts have improved the accuracy of sentiment classification to a certain extent, they still have shortcomings when processing these complex short texts. The current mainstream sentiment classification methods still need to be improved in terms of semantic feature extraction and context information capture capabilities. It is difficult to effectively capture the long-distance correlation information and deep semantic connotations between short texts, which leads to the fact that the accuracy and robustness of sentiment classification still have room for improvement.

[0008] In summary, in the field of natural language processing, especially in the sentiment classification task of short texts for university teaching evaluation, how to use fully homomorphic encryption technology to achieve secure computing in the cloud and how to improve the model's feature extraction and semantic analysis capabilities for complex short texts are two important issues in current research and application. Summary of the invention

[0009] In view of this, the present invention proposes a teaching evaluation short text sentiment classification method based on distributed approximate homomorphic encryption, which aims to ensure the privacy and security of teaching evaluation data while improving the accuracy and processing efficiency of sentiment classification. Aiming at the needs of sentiment classification of teaching evaluation texts in colleges and universities, this method effectively combines the distributed approximate homomorphic encryption (BHE, BERT-based Homomorphic Encryption) algorithm and the ALBERT-Mixplus fusion model, which improves the accuracy and efficiency of sentiment classification tasks while protecting the privacy of college teaching evaluation data, so as to realize the encryption processing and accurate classification of massive teaching evaluation short text data.

[0010] The present invention is implemented by the following technical solutions:

[0011] In a first aspect, the present invention provides a method for sentiment classification of teaching review short texts based on distributed approximate homomorphic encryption, the method comprising the following steps:

[0012] Use the distributed approximate homomorphic encryption (BHE) algorithm to encrypt the input teaching evaluation short text data;

[0013] The encrypted short text data is input into the ALBERT-Mixplus fusion model deployed in the cloud for feature extraction and sentiment classification, wherein the ALBERT-Mixplus fusion model includes:

[0014] The meta-information extraction layer uses a lightweight pre-trained ALBERT model to extract features from the input text through a multi-head self-attention mechanism and learn general semantic knowledge;

[0015] The fusion feature extraction layer further extracts the features output by the ALBERT model through the bidirectional gated recurrent unit (BiGRU), multi-scale convolutional neural network (MCNN-MCO) and self-attention mechanism (MCNN-SA);

[0016] The splicing output layer splices the features extracted by the fusion feature extraction layer, and calculates the probability of sentiment classification through the approximate Appro-Softmax function;

[0017] The encrypted sentiment classification results are decrypted on the local user side and the final classification results are output.

[0018] As a further solution of the present invention, a distributed approximate homomorphic encryption (BHE) algorithm is based on the CKKS (Homomorphic Encryption Scheme for Approximate Arithmetic) encryption scheme and is called by the SEAL (Simple Encrypted Arithmetic Library) encryption algorithm library.

[0019] As a further solution of the present invention, in the teaching evaluation short text sentiment classification method based on distributed approximate homomorphic encryption, the calculation of the ALBERT-Mixplus fusion model is executed in the cloud, and the encrypted short text data is classified and predicted; wherein, the calculation of the ALBERT-Mixplus fusion model includes standard fine-tuning of the pre-trained model and LN-feature extraction; the pre-trained ALBERT-Mixplus fusion model is subjected to standard fine-tuning, and the non-polynomial function in the pre-trained model is replaced with the ReLU activation function and the Appro-Softmax function; the pre-trained ALBERT-Mixplus fusion model is optimized by the labeled teaching evaluation short text data, and the model parameters are optimized by the Original LN calculation module and the mean square error (MSE), so as to obtain an ALBERT-Mixplus fusion model that fully supports homomorphic encryption operations.

[0020] As a further solution of the present invention, in the ALBERT-Mixplus fusion model, the number of Encoder layers of the ALBERT model is set to 6, and the multi-scale convolutional neural network (MCNN-MCO) uses convolution kernels of different sizes for convolution operations, combines three convolution layers with convolution kernel sizes of 2, 3, and 4 respectively, and combines its own attention mechanism (MCNN-SA) after the convolution layer output to further extract key emotional features.

[0021] As a further solution of the present invention, in order to meet the problem of non-polynomial function calculation in the fusion sentiment classification model based on the BHE fully homomorphic encryption (FHE) system, in the ALBERT-Mixplus fusion sentiment classification model structure, ReLU is selected as the activation function, and the pooling operation is discarded in the MCNN, and the Max() calculation is placed on the user side. At the same time, the approximate Appro-Softmax function supported by BHE is used as the classifier. The final result of the sentiment classification is calculated by the distributed approximate homomorphic encryption (BHE) algorithm and completed in an encrypted state.

[0022] As a further solution of the present invention, ReLU is selected as the activation function, and when the MAX comparison operation in the ReLU function is performed, a local-public cloud distributed fully homomorphic encryption (FHE) computing architecture is constructed, wherein the ReLU fully homomorphic includes the following steps: the cloud side passes the ciphertext data input of the encrypted short text data to the local side for local Max() calculation; once the connection is received, the user device on the local side calls the local Max function to obtain the result, and uses the private key to decrypt the ciphertext into plaintext data on the local side. After the plaintext data after the Max() operation on the local side is encrypted with the public key, the re-encrypted result is returned to the cloud side.

[0023] As a further solution of the present invention, the approximate Appro-Softmax function is implemented using an estimation network of addition and multiplication, and the calculation formula of the approximate Appro-Softmax function is:

[0024]

[0025] In the formula, x i represents the i-th element in the input vector, x j represents the jth element of the input vector, ∑ j It means summing the scores of all categories; ReLU works by splitting the Max operation to the user side, and a three-layer linear neural network T is used to approximate the inverse operation.

[0026] As a further solution of the present invention, the distributed approximate homomorphic encryption (BHE) algorithm uses piecewise polynomial fitting to approximate LayerNorm. The calculation formula of LayerNorm is:

[0027]

[0028] In the formula, x is the input variable, μ is the mean, 2 is the variance, ξ is a constant, and the LayerNorm approximation is used to ensure that the model is compatible with FHE; the approximate version of LayerNorm is expressed as:

[0029]

[0030] Among them, a i are the polynomial coefficients obtained through training, x is the input vector, and Appro-LN is used as the LN-Extraction module to learn valuable semantic information from LN Layers. By simplifying and replacing LayerNorm, the distributed approximate homomorphic encryption (BHE) algorithm implements the LayerNorm operation under the homomorphic encryption framework.

[0031] As a further solution of the present invention, the ALBERT-Mixplus fusion model performs feature extraction and sentiment classification, including the following steps:

[0032] The local (user) side constructs a distributed word vector for the text sentence;

[0033] Encrypt the word vector using the local (user) encryption public key;

[0034] The encrypted word vectors are transmitted to the cloud, where the BHE-FHE model is used to calculate the sentiment classification task.

[0035] The activation function of the local (user) side calculation model, including the ReLU activation function and Max() calculation;

[0036] After receiving the ciphertext data returned by the local (user) side, the cloud continues to perform the sentiment classification task of the model;

[0037] Finally, the local (user) side uses the encrypted private key to decrypt the sentiment classification results.

[0038] As a further solution of the present invention, the local (user) side encrypts the distributed calculation results of the word vector and transmits them to the cloud. The cloud uses a distributed approximate homomorphic encryption algorithm to perform classification task calculations, and the activation function is calculated on the local (user) side.

[0039] The present invention also includes a computer device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor executes the teaching and review short text sentiment classification method based on distributed approximate homomorphic encryption.

[0040] The present invention also includes a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the teaching and review short text sentiment classification method based on distributed approximate homomorphic encryption.

[0041] Compared with the prior art, the method for sentiment classification of teaching review short texts based on distributed approximate homomorphic encryption provided by the present invention has the following beneficial effects:

[0042] 1. It has the effects of privacy protection and data security. By using the distributed approximately homomorphic encryption (BHE) algorithm, the present invention can realize the transmission, calculation and prediction of text data in a fully encrypted environment. It not only ensures the privacy protection of data when it is deployed in the public cloud, but also enables the sentiment classification model to be safely applied in scenarios with high requirements for data privacy such as colleges and universities. Especially in the scenario where local data storage is combined with cloud computing, the method of the present invention provides an efficient and secure solution for the distributed application of AI+ business systems.

[0043] 2. The model has compatibility and scalability. In the present invention, the ALBERT MixPlus text sentiment classification model fully combines the characteristics of the teaching review short text, and uses the double-layer structure of the meta-information extraction layer and the fusion feature extraction layer to effectively improve the model's feature extraction ability for the teaching review text. By increasing the ratio of contextual semantic associations and keyword weights in the fusion feature vector, the model can more accurately predict the true teaching review semantics. At the same time, the ALBERT MixPlus model is fully compatible with the distributed approximate homomorphic encryption (BHE) algorithm, ensuring that a high classification performance can be maintained in an encrypted environment.

[0044] 3. Achieved performance improvement. Compared with the traditional teaching review short text sentiment classification method, the method of the present invention has been improved in multiple performance indicators. By introducing the combined structure of BiGRU and MCNN, the model can better capture global semantic information and local sentiment features, improving the accuracy and robustness of sentiment classification. In addition, by abandoning the pooling operation, the method of the present invention better adapts to the special needs of the fully homomorphic encryption environment, so that sentiment classification can also be efficiently performed on encrypted data.

[0045] In summary, the sentiment classification method for teaching review short texts based on distributed approximate homomorphic encryption provides a comprehensive solution for efficient and secure sentiment classification by ensuring data privacy, improving classification performance and expanding adaptability. It has significant application value and broad promotion prospects. It is not only suitable for AI+ business systems in colleges and universities, but can also be extended to other fields with strict requirements on data privacy and security, such as medical data analysis, financial risk control, etc. The powerful adaptability and extensibility of the present invention enable this method to perform well in a variety of complex application scenarios, providing safe and efficient sentiment classification services.

[0046] These and other aspects of the present invention will be more clearly understood in the following description of the embodiments. It should be understood that the above general description and the following detailed description are only exemplary and explanatory and cannot limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the following briefly introduces the drawings required for use in the exemplary embodiments or related technical descriptions. The drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation to the present invention. In the drawings:

[0048] Figure 1 The present invention is a flowchart of a method for sentiment classification of teaching and review short texts based on distributed approximately homomorphic encryption according to an embodiment of the present invention.

[0049] Figure 2 The overall architecture diagram of the teaching and review short text sentiment classification method based on distributed approximate homomorphic encryption according to an embodiment of the present invention.

[0050] Figure 3 This is a workflow diagram of the BHE approximate homomorphic algorithm in the teaching and review short text sentiment classification method based on distributed approximate homomorphic encryption in an embodiment of the present invention.

[0051] Figure 4 This is a framework diagram of the ALBERT-Mixplus meta-information extraction layer in the teaching and review short text sentiment classification method based on distributed approximate homomorphic encryption in an embodiment of the present invention.

[0052] Figure 5 This is a framework diagram of the multi-head self-attention mechanism in the teaching and review short text sentiment classification method based on distributed approximate homomorphic encryption in an embodiment of the present invention.

[0053] Figure 6 This is a graphical diagram of the multi-head self-attention mechanism in the ALBERT-Mixplus meta-information extraction layer in the teaching and review short text sentiment classification method based on distributed approximately homomorphic encryption in an embodiment of the present invention.

[0054] Figure 7 This is a structural diagram of the MCNN of the ALBERT-Mixplus fusion feature extraction layer in the teaching and review short text sentiment classification method based on distributed approximate homomorphic encryption in an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0056] In some of the processes described in the specification and claims of the present invention and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0057] The following will be combined with the drawings in the exemplary embodiments of the present invention to clearly and completely describe the technical solutions in the exemplary embodiments of the present invention. Obviously, the exemplary embodiments described are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0058] In order to ensure the privacy and security of teaching evaluation data while improving the accuracy and processing efficiency of sentiment classification, in response to the needs of sentiment classification of teaching evaluation texts in colleges and universities, the present invention provides a teaching evaluation short text sentiment classification method based on distributed approximate homomorphic encryption, which effectively combines the distributed approximate homomorphic encryption (BHE) algorithm with the ALBERT-Mixplus fusion model. While protecting the privacy of university teaching evaluation data, it improves the accuracy and efficiency of sentiment classification tasks, so as to realize the encryption processing and accurate classification of massive teaching evaluation short text data.

[0059] The technical solution of the present invention is further described below in conjunction with specific embodiments:

[0060] See also Figure 1 As shown, a method for sentiment classification of teaching and review short texts based on distributed approximate homomorphic encryption is provided in one embodiment of the present invention, comprising the following steps:

[0061] Step S10: using a distributed approximately homomorphic encryption (BHE) algorithm to encrypt the input teaching evaluation short text data;

[0062] Step S20: input the encrypted short text data into the ALBERT-Mixplus fusion model deployed in the cloud for feature extraction and sentiment classification, wherein the ALBERT-Mixplus fusion model includes:

[0063] The meta-information extraction layer uses a lightweight pre-trained ALBERT model to extract features from the input text through a multi-head self-attention mechanism and learn general semantic knowledge;

[0064] The fusion feature extraction layer further extracts the features output by the ALBERT model through the bidirectional gated recurrent unit (BiGRU), multi-scale convolutional neural network (MCNN-MCO) and self-attention mechanism (MCNN-SA);

[0065] The output layer is concatenated to concatenate the features extracted by BiGRU and MCNN, and the probability of sentiment classification is calculated by approximating the Appro-Softmax function.

[0066] Step S30: decrypt the encrypted emotion classification result at the local user side and output the final classification result.

[0067] The teaching evaluation short text sentiment classification method based on distributed approximate homomorphic encryption in this embodiment combines the distributed approximate homomorphic encryption (BHE) algorithm with the ALBERT-Mixplus fusion model to achieve feature extraction and sentiment classification of teaching evaluation short texts in a fully encrypted environment.

[0068] The teaching review short text sentiment classification method of the present invention proposes a distributed approximate homomorphic encryption (BHE) algorithm for the first time. The algorithm constructs a local-public cloud distributed fully homomorphic encryption (FHE) computing structure and introduces user security side devices to perform FHE encryption calculations. In order to adapt to the text sentiment classification based on the pre-trained model, this algorithm adopts a "simplified calculation" method to approximate non-polynomial functions (such as GELU, Softmax, LayerNorm) that cannot support fully homomorphic encryption. This method ensures the privacy security of the sentiment classification model in a public cloud computing environment, and is particularly suitable for scenarios with high data privacy requirements such as colleges and universities.

[0069] In the teaching evaluation short text sentiment classification method of the present invention, an ALBERT-Mixplus fusion model that fully combines the characteristics of the teaching evaluation short text is also proposed, which mainly includes two core parts: the meta-information extraction layer and the fusion feature extraction layer. The meta-information extraction layer adopts a lightweight pre-trained ALBERT model to enhance the model's anti-noise ability by learning a large amount of general semantic knowledge. The fusion feature extraction layer is composed of a bidirectional gated recurrent unit (BiGRU) and a multi-scale convolutional neural network (MCNN-MCO), which can fully capture the high-dimensional semantic representation of the teaching evaluation text, and through the precise calculation of the semantic association between the context and the keyword weight, the model prediction result is closer to the real semantics.

[0070] See also Figure 2As shown, the overall architecture of the teaching review short text sentiment classification method based on distributed approximate homomorphic encryption of the present invention consists of two parts: distributed approximate homomorphic encryption (BHE) algorithm and ALBERT-Mixplus fusion model. In an embodiment of the present invention, the distributed approximate homomorphic encryption (BHE) algorithm is based on the CKKS (Homomorphic Encryption Scheme for Approximate Arithmetic) encryption scheme, and is called by the SEAL (Simple Encrypted Arithmetic Library) encryption algorithm library. Through the distributed approximate homomorphic encryption proposed by this method, data transmission, calculation and prediction can be realized in an encrypted environment, which provides a strong guarantee for privacy protection when the sentiment classification model is deployed in a public cloud. The distributed approximate homomorphic encryption BHE algorithm provides an efficient and secure solution path for the distributed application scenarios of AI+ business systems in colleges and universities based on the combination of local data storage and cloud computing.

[0071] In an embodiment of the present invention, the calculation of the ALBERT-Mixplus fusion model is performed in the cloud to perform classification prediction on the encrypted short text data; wherein the calculation of the ALBERT-Mixplus fusion model includes standard fine-tuning of the pre-trained model and LN-feature extraction; the specific process is:

[0072] The pre-trained ALBERT-Mixplus fusion model is fine-tuned in standard way, and the non-polynomial function in the pre-trained model is replaced with the ReLU activation function and the Appro-Softmax function. The pre-trained ALBERT-Mixplus fusion model is optimized by using the labeled teaching evaluation short text data, and the model parameters are optimized by the Original LN calculation module and the mean square error (MSE), thus obtaining an ALBERT-Mixplus fusion model that fully supports homomorphic encryption operations.

[0073] See also Figure 3 As shown in Figure 2, given an original ALBERT-Mixplus model without FHE approximation, labeled M, a fully HE-supported ALBERT-Mixplus model is generated by fine-tuning the teaching evaluation data. In the standard fine-tuning process, the pre-trained model M was fine-tuned, and the non-polynomial functions (GeLu, Softmax) in the pre-trained model were replaced with ReLU and Appro-Softmax models. Then, the ALBERT-Mixplus fusion model was fine-tuned using labeled teaching evaluation sentiment short text data. After completion, Appro-LN was added while maintaining Original LN. Since the original LN contains complex nonlinear operations, the role of Appro-LN is to approximate it as a linear operation, and the parameters are optimized by comparing the mean square error MSE between Original LN and Appro-LN. After the Original LN tunes the parameters, the Original LN is deleted to obtain a model that fully supports homomorphic encryption operations.

[0074] Among them, the approximate workflow of the entire distributed approximately homomorphic encryption (BHE) algorithm is:

[0075] Data: labeled teaching evaluation data D;

[0076] Input: pre-trained ALBERT-Mixplus fusion model M, Appro-Softmax: S.

[0077] 1. Replace GELU and Softmax functions with S and ReLU activation functions;

[0078] 2. While not done do;

[0079] 3. Sample batches (xi,yi) from dataset D;

[0080] 4. Use (xi,yi) and S to optimize Model;

[0081] End

[0082] 5. Added Appro-LN calculation module;

[0083] 6. While not done do:

[0084] Sample batches (xi,yi) from dataset D;

[0085] Freeze Beyond All parameters;

[0086] Calculate k-th Original LN calculation module output O k ,

[0087] Calculate the loss value:

[0088] Using mean squared error MSE loss and Γ = Σkl k renew

[0089] End

[0090] Cancel the Original LN calculation module

[0091] Return Get a model that fully supports homomorphic encryption operations

[0092] In the ALBERT-Mixplus fusion model, the number of Encoder layers of the ALBERT model is set to 6, and the multi-scale convolutional neural network (MCNN-MCO) uses convolution kernels of different sizes for convolution operations, combining three convolutional layers with convolution kernel sizes of 2, 3, and 4 respectively, and further extracts key sentiment features in combination with its own attention mechanism (MCNN-SA) after the convolutional layer output.

[0093] In the ALBERT-Mixplus fusion model, a 6-layer ALBERT Encoder layer is set, and three sets of self-attention calculations and three sets of convolution calculations are used, with convolution kernel sizes of 2, 3, and 4. Under the premise of meeting the FHE encryption requirements, ReLU is selected as the activation function, and the Max calculation is placed on the user side. At the same time, the approximate Appro-Softmax function supported by BHE is used as the classifier.

[0094] The ALBERT-Mixplus fusion model uses a dual-layer structure of a meta-information extraction layer and a fusion feature extraction layer to effectively improve the ability to extract features from teaching review texts. By reasonably allocating contextual semantic associations and keyword weights, the model can more accurately reflect the true semantics of teaching reviews.

[0095] Compared with the existing sentiment classification methods for short teaching and review texts, the ALBERT-Mixplus fusion model has significant improvements in multiple performance indicators and is fully compatible with the distributed approximate homomorphic encryption (BHE) algorithm.

[0096] The present invention is based on the distributed approximate homomorphic encryption method for sentiment classification of teaching review short texts, which makes an important contribution to the field of text sentiment classification based on homomorphic encryption. By proposing the distributed approximate homomorphic encryption and ALBERT-Mixplus fusion model, this method not only solves the problem of data privacy protection, but also improves the performance of the model in sentiment classification tasks. This method provides a safe and efficient solution for colleges and universities and other scenarios that require a high degree of data privacy protection, and has broad application prospects.

[0097] In an embodiment of the present invention, in order to meet the problem of non-polynomial function calculation in the fusion sentiment classification model based on the BHE-FHE system, in the ALBERT-Mixplus fusion sentiment classification model structure, ReLU is selected as the activation function, and the pooling operation is discarded in the MCNN, and the Max() calculation is placed on the user side. At the same time, the approximate Appro-Softmax function supported by BHE is used as the classifier. The final result of the sentiment classification is calculated by the distributed approximate homomorphic encryption (BHE) algorithm and completed in an encrypted state.

[0098] Among them, ReLU is selected as the activation function. When the MAX comparison operation in the ReLU function is performed, a local-public cloud distributed fully homomorphic encryption (FHE) computing architecture is constructed, wherein the ReLU fully homomorphic includes the following steps: the cloud side passes the ciphertext data input of the encrypted short text data to the local side for local Max() calculation; once the connection is received, the user device on the local side calls the local Max function to obtain the result, and uses the private key to decrypt the ciphertext into plaintext data on the local side. After the plaintext data after the Max() operation on the local side is encrypted with the public key, the re-encrypted result is returned to the cloud side.

[0099] In this embodiment, the fully homomorphic workflow with ReLU as the activation function is:

[0100] 1. ReLU max ()→U / / Decompose ReLU calculation, U represents the local (user) side device;

[0101] 2.C(E)→U(E). / / C represents the cloud device, which sends the encrypted data from the cloud device to the local (user) side device;

[0102] 3. Represents plaintext data, and uses the private key to decrypt the ciphertext into plaintext on the local (user) side device;

[0103] 4. Represents the plaintext data after Max() operation on the local (user) side;

[0104] 5. The plaintext data after the Max() operation is encrypted with the public key and sent back to the cloud to complete the subsequent ReLU() calculation.

[0105] In an embodiment of the present invention, the second non-polynomial function approximated by the BHE approximate homomorphic algorithm is Softmax. The Softmax function converts a set of values ​​into a probability distribution. The approximate Appro-Softmax function is implemented using an estimation network of addition and multiplication. The calculation formula of the approximate Appro-Softmax function is:

[0106]

[0107] In the formula, x i represents the i-th element in the input vector, x j represents the jth element of the input vector, ∑ j It means summing the scores of all categories; ReLU works by splitting the Max operation to the user side, and a three-layer linear neural network T is used to approximate the inverse operation.

[0108] To better estimate Softmax, the present invention randomly generates input tensors with values ​​between [-3,3] and uses the Softmax score as the mean square error (MSE) target. Then, the present invention optimizes T for 100k steps with a learning rate of 1e-3 until the MSE loss drops to 1e-6.

[0109] Similar to the above non-polynomial function (GeLu, Softmax) replacement, LN (LayerNorm) operation is not directly supported in FHE. In order to achieve FHE (fully homomorphic encryption) approximation, BHE adopts a piecewise polynomial fitting method. The main goal of LN is to normalize the activation of each input sample so that its mean is 0 and its variance is 1. The distributed approximate homomorphic encryption (BHE) algorithm uses piecewise polynomial fitting to approximate LayerNorm. The calculation formula of LayerNorm is:

[0110]

[0111] In the formula, x is the input variable, μ is the mean, and σ is the 2 is the variance, ξ is a constant, and the LayerNorm approximation is used to ensure that the model is compatible with FHE; the approximate version of LayerNorm is expressed as:

[0112]

[0113] Among them, a iare the polynomial coefficients obtained through training, x is the input vector, and Appro-LN is used as the LN-Extraction module to learn valuable semantic information from LN Layers. By simplifying and replacing LayerNorm, the distributed approximate homomorphic encryption (BHE) algorithm implements the LayerNorm operation under the homomorphic encryption framework.

[0114] In an embodiment of the present invention, the ALBERT-Mixplus fusion model performs feature extraction and sentiment classification, including the following steps:

[0115] The local (user) side constructs a distributed word vector for the text sentence;

[0116] Encrypt the word vector using the local (user) encryption public key;

[0117] The encrypted word vectors are transmitted to the cloud, where the BHE-FHE model is used to calculate the sentiment classification task.

[0118] The activation function of the local (user) side calculation model, including the ReLU activation function and Max() calculation;

[0119] After receiving the ciphertext data returned by the local (user) side, the cloud continues to perform the sentiment classification task of the model;

[0120] Finally, the local (user) side uses the encrypted private key to decrypt the sentiment classification results.

[0121] Among them, the local (user) side encrypts the distributed calculation results of the word vector and transmits them to the cloud. The cloud uses a distributed approximate homomorphic encryption algorithm to perform classification task calculations, and the activation function is calculated on the local (user) side.

[0122] In this embodiment, the workflow of the ALBERT-Mixplus fusion model based on the BHE approximate homomorphic algorithm in the sentiment classification task is as follows:

[0123] Input: text statement, encrypted private key, encrypted public key K public ,BHE-FHE model M.

[0124] 1. The local (user) side constructs a distributed word vector P from the text sentence q → q ;

[0125] 2. Local (user) side public key encryption word vector

[0126] 3. Perform sentiment classification tasks of the encrypted model on the cloud side:

[0127] 4. Activation function of the local (user) side computing model: C a =ReLU(c i );

[0128] 5. After the cloud side receives the ciphertext data returned by the activation function on the local (user) side, it continues the model sentiment classification task:

[0129] 6. The local (user) side uses the encrypted private key to decrypt the sentiment classification results:

[0130] In the ALBERT-Mixplus fusion model of this embodiment, semantic feature extraction is performed through the meta-information extraction layer and the hybrid feature extraction layer. In order to reduce the overhead of the BHE algorithm, the lightweight pre-trained ALBERT model is adopted in the meta-information extraction layer. By combining the specific context of the target word, the vector representation is dynamically adjusted to improve the semantic expression ability of the word. The hybrid feature extraction layer is composed of BiGRU+MCNN, which is used to capture the local features and high-dimensional global sentiment semantic information of the teaching evaluation text at different scales, enhance the sentiment feature extraction ability of the model, and finally complete the sentiment classification task through the classifier after reducing the dimensionality of the mixed text feature vector.

[0131] In order to significantly reduce the number of model parameters and computing resource requirements while maintaining the high performance of the fusion model, the present invention adopts the ALBERT model in the meta-information extraction layer of the ALBERT-Mixplus fusion model. ALBERT shares parameters between different Encoder layers instead of using different parameters for each layer. The meta-information extraction layer architecture is as follows: Figure 4 , Figure 5 and Figure 6 As shown, where X = [x1, x2…x n ] represents the input vector fused with position encoding, TR is the encoder module in Transformer, and it should be noted that in the multi-head attention part of each encoder, the present invention constructs the fusion feature according to the following process:

[0132] (1) Map the Query, Key, and Value matrices through h different linear transformations;

[0133] (2) The BHE-approximated Softmax classifier is used to calculate different Attention representations through Formula 6;

[0134]

[0135] (3) Concatenate different Attention feature expressions Z^, Z^=[Z0, Z1, ... Z8]. In this model, the depth of self-attention is set to 8;

[0136] (4) The concatenated matrix and a weight matrix W O Multiply them together, and finally get the information representation Z containing all self-attention heads, which is input into the position feed-forward network FFN.

[0137] In an embodiment of the present invention, the fusion feature extraction layer of the ALBERT-Mixplus fusion model is composed of two models, Bidirectional Gated Recurrent Unit (BiGRU) and Multiscale Convolutional Neural Network (MCNN), connected in series. The feature fusion vector representation output by ALBERT is passed through BiGRU to capture semantic context and global information, and then the local semantic features of different granularities are extracted through the multi-scale convolution kernel of MCNN-MCO to fully obtain the high-dimensional semantic representation of the teaching review text, and MCNN-SA identifies the key emotional features that have a greater impact on the model classification results.

[0138] In order to improve the index of the ALBERT-Mixplus fusion model on the classification time of encrypted data, the present invention adopts the BiGRU model to replace the traditional BiLSTM three-gate structure (forget gate, input gate, output gate). BiGRU is a recurrent neural network, an improved variant of RNN and BiLSTM, and is composed of two relatively independent GRU units, one forward processing data according to the time series and the other reverse processing data according to the time series. Through these two bidirectional structures, BiGRU can capture forward and backward information at the same time. The GRU unit includes Update Gate and Reset Gate. Update Gate controls how much of the previous state is included in the current state, and Reset Gate controls the influence of the previous state on the current state.

[0139] Among them, the calculation formula of GRU is:

[0140] Z t =σ(W z ·[h t-1 ,x t ])

[0141] r t =σ(W r ·[[h t-1 ,x t ]])

[0142]

[0143]

[0144] Among them, σ represents the sigmoid activation function, Z t It is Update Gate, t It is Reset Gate, h t Represents the current state. represents the candidate state, W r Represents the weight matrix of the reset gate, W z Represents the weight matrix of the update gate.

[0145] After the BiGRU layer, we get a new feature vector representation: H = n >. The newly obtained vector representation H is input into the multi-scale convolution kernel to extract the local sentiment features of the teaching review text at multiple scales. For the MCNN module structure, see Figure 7 As shown, MCNN includes multi-scale convolution operation (MCNN-MCO) and self-attention mechanism (MCNN-SA). It should be noted that in order to meet the needs of FHE encryption, the fusion model abandons the pooling operation in the present invention. MCNN-MCO uses convolution kernels of different sizes to slide from top to bottom on the text vector matrix to extract local text features. The calculation process is:

[0146]

[0147] C=(c1,c2,…,c n-m+1 )

[0148] in, represents the convolution operation, W represents the parameter weight matrix, b represents the bias value, H i:i+m-1 It means that the BiGRU module outputs the text feature vector of the i-th to i+m-1-th rows; f is the activation function. This paper adopts ReLU as the activation function. The size of the convolution kernel filter is set to (2, 3, 4). After the convolution operation, we get the feature representations C2, C3, and C4.

[0149] In the embodiment of the present invention, MCNN-SA aims to enhance the key feature extraction capability of the vector after the convolution operation, highlight the key emotional features, and calculate the convolution layer output feature C i The corresponding attention score β i , the overall attention output V is obtained by weighted summation. The specific calculation process is:

[0150]

[0151] ​

[0152]

[0153] Among them, ReLU is a nonlinear activation function, W z and B z is the weight parameter matrix, and finally the attention features of each convolution channel are fused into the feature representation vectors S1, S2, and S3. The attention feature fusion vector of multiple convolution channels is concatenated and expressed as:

[0154] Z=Concat(S1,S2,S3)

[0155] The sentiment classification probability P is calculated by approximating the Appro-Softmax function (Appro-Softmax) m :

[0156] Pm=(Approximate)Soft max(W m Z+b s ).

[0157] The method for sentiment classification of teaching review short texts based on distributed approximate homomorphic encryption of the present invention can realize the transmission, calculation and prediction of text data in a fully encrypted environment by using the distributed approximate homomorphic encryption (BHE) algorithm. It not only ensures the privacy protection of data when it is deployed in the public cloud, but also enables the sentiment classification model to be safely applied in scenarios such as colleges and universities that have high requirements for data privacy. Especially in the scenario where local data storage is combined with cloud computing, the method of the present invention provides an efficient and secure solution for the distributed application of AI+ business systems.

[0158] In the present invention, the ALBERT MixPlus text sentiment classification model fully combines the features of short teaching review texts, and uses the double-layer structure of the meta-information extraction layer and the fusion feature extraction layer to effectively improve the model's feature extraction ability for teaching review texts. By increasing the ratio of contextual semantic associations and keyword weights in the fusion feature vector, the model can more accurately predict the true teaching review semantics. At the same time, the ALBERT MixPlus model is fully compatible with the distributed approximate homomorphic encryption (BHE) algorithm, ensuring that high classification performance can be maintained in an encrypted environment.

[0159] Compared with the traditional sentiment classification method of teaching review short text, the method of the present invention has been improved in multiple performance indicators. By introducing the combined structure of BiGRU and MCNN, the model can better capture global semantic information and local sentiment features, improving the accuracy and robustness of sentiment classification. In addition, by abandoning the pooling operation, the method of the present invention better adapts to the special needs of the fully homomorphic encryption environment, so that sentiment classification can also be efficiently performed on encrypted data.

[0160] In summary, the sentiment classification method for teaching review short texts based on distributed approximate homomorphic encryption provides a comprehensive solution for efficient and secure sentiment classification by ensuring data privacy, improving classification performance and expanding adaptability. It has significant application value and broad promotion prospects. It is not only suitable for AI+ business systems in colleges and universities, but can also be extended to other fields with strict requirements on data privacy and security, such as medical data analysis, financial risk control, etc. The powerful adaptability and extensibility of the present invention enable this method to perform well in a variety of complex application scenarios, providing safe and efficient sentiment classification services.

[0161] It should be understood that, although described in a certain order, these steps are not necessarily performed in sequence in the above order. Unless there is clear explanation in this article, the execution of these steps does not have strict order restriction, and these steps can be performed in other orders. Moreover, a part of the steps of the present embodiment may include a plurality of steps or a plurality of stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a part of the steps or stages in other steps or other steps.

[0162] In one embodiment, a computer device is also provided in an embodiment of the present invention, comprising at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor performs the steps of the teaching and review short text sentiment classification method based on distributed approximate homomorphic encryption.

[0163] In one embodiment, the present invention also provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the steps of the teaching and review short text sentiment classification method based on distributed approximate homomorphic encryption.

[0164] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be realized by instructing the relevant hardware through a computer program represented by computer instructions, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory.

[0165] Non-volatile memory may include read-only memory, magnetic tape, floppy disk, flash memory or optical storage, etc. Volatile memory may include random access memory or external cache memory. As an illustration and not limitation, RAM may be in various forms, such as static random access memory or dynamic random access memory, etc.

[0166] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for sentiment classification of teaching review short texts based on distributed approximate homomorphic encryption, characterized in that: The method comprises the following steps: Use distributed approximate homomorphic encryption to encrypt the input teaching evaluation short text data; The encrypted short text data is input into the ALBERT-Mixplus fusion model deployed in the cloud for feature extraction and sentiment classification, wherein the ALBERT-Mixplus fusion model includes: The meta-information extraction layer uses a lightweight pre-trained ALBERT model to extract features from the input text through a multi-head self-attention mechanism and learn general semantic knowledge; The fusion feature extraction layer further extracts the features output by the ALBERT model through bidirectional gated recurrent units, multi-scale convolutional neural networks, and self-attention mechanisms; The splicing output layer splices the features extracted by the fusion feature extraction layer, and calculates the probability of sentiment classification through the approximate Appro-Softmax function; Decrypt the encrypted sentiment classification results on the local user side and output the final classification results; In the ALBERT-Mixplus fusion model structure, ReLU is selected as the activation function, and the pooling operation is discarded in the multi-scale convolutional neural network. The Max() calculation is placed on the user side. At the same time, the approximate Appro-Softmax function supported by distributed approximate homomorphic encryption is used as the classifier. The final result of sentiment classification is calculated by the distributed approximate homomorphic encryption algorithm and completed in an encrypted state. When ReLU is selected as the activation function and the MAX comparison operation in the ReLU function is performed, a local-public cloud distributed fully homomorphic encryption computing architecture is constructed, wherein the ReLU fully homomorphic includes the following steps: the cloud side passes the ciphertext data input of the encrypted short text data to the local side for local Max() calculation; once the connection is received, the user device on the local side calls the local Max function to obtain the result, and uses the private key to decrypt the ciphertext into plaintext data on the local side. After the plaintext data after the Max() operation on the local side is encrypted with the public key, the re-encrypted result is returned to the cloud side; The approximate Appro-Softmax function is implemented using an estimation network of addition and multiplication. The calculation formula of the approximate Appro-Softmax function is: In the formula, x i represents the i-th element in the input vector, x j represents the jth element of the input vector, ∑ j Indicates the sum of the scores of all categories; ReLU works by splitting the Max operation to the user side, and a three-layer linear neural network T is used to approximate the reciprocal operation; Distributed approximate homomorphic encryption uses piecewise polynomial fitting to approximate LayerNorm. The calculation formula of LayerNorm is: In the formula, x is the input variable, μ is the mean, and σ is the 2 is the variance, ξ is a constant, and the LayerNorm approximation is used to ensure that the model is compatible with FHE; the approximate version of LayerNorm is expressed as: Among them, a i are the polynomial coefficients obtained through training, x is the input vector, and Appro-LN is used as the LN-Extraction module to learn valuable semantic information from LN Layers. By simplifying and replacing LayerNorm, distributed approximate homomorphic encryption implements LayerNorm operation under the homomorphic encryption framework.

2. The method for sentiment classification of teaching review short texts based on distributed approximate homomorphic encryption as claimed in claim 1, characterized in that: Distributed approximately homomorphic encryption is based on the CKKS encryption scheme and is called by the SEAL encryption algorithm library.

3. The method for sentiment classification of teaching review short texts based on distributed approximate homomorphic encryption as claimed in claim 2, characterized in that: In the teaching evaluation short text sentiment classification method based on distributed approximate homomorphic encryption, the calculation of the ALBERT-Mixplus fusion model is performed on the cloud, and the encrypted short text data is classified and predicted; wherein, the calculation of the ALBERT-Mixplus fusion model includes standard fine-tuning of the pre-trained model and LN-feature extraction; the pre-trained ALBERT-Mixplus fusion model is subjected to standard fine-tuning, and the non-polynomial function in the pre-trained model is replaced with the ReLU activation function and the Appro-Softmax function; the pre-trained ALBERT-Mixplus fusion model is optimized by the labeled teaching evaluation short text data, and the model parameters are optimized by the Original LN calculation module and the mean square error, so as to obtain an ALBERT-Mixplus fusion model that fully supports homomorphic encryption operations.

4. The method for sentiment classification of teaching review short texts based on distributed approximate homomorphic encryption as claimed in claim 3, characterized in that: In the ALBERT-Mixplus fusion model, the number of Encoder layers of the ALBERT model is set to 6, and the multi-scale convolutional neural network uses convolution kernels of different sizes for convolution operations. It combines three convolution layers with convolution kernel sizes of 2, 3, and 4 respectively, and extracts key sentiment features based on its own attention mechanism after the convolution layer output.

5. The method for sentiment classification of teaching review short texts based on distributed approximate homomorphic encryption as claimed in claim 1, characterized in that: The ALBERT-Mixplus fusion model performs feature extraction and sentiment classification, including the following steps: The local side constructs distributed word vectors for text sentences; Encrypt the word vector using the local encryption public key; The encrypted word vectors are transmitted to the cloud, where the BHE-FHE model is used to calculate the sentiment classification task. The local side calculates the activation function of the model, including the ReLU activation function and Max() calculation; After receiving the ciphertext data returned by the local side, the cloud continues to perform the sentiment classification task of the model; Finally, the local side uses the encrypted private key to decrypt the sentiment classification results.

6. The method for sentiment classification of teaching review short texts based on distributed approximate homomorphic encryption as claimed in claim 5, characterized in that: The local side encrypts the distributed computing results of the word vectors and transmits them to the cloud. The cloud uses a distributed approximate homomorphic encryption algorithm to perform classification task calculations, and the activation function is calculated on the local side.

Images