Text recognition method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a text recognition method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out the hash operation of word segmentation data of a to-be-recognized text, and obtaining a first hash value corresponding to the word segmentation data; judging whether the first hash value is successfully matched with a second hash value stored in a database or not; if matching between the first hash value and the second hash value fails, processing the word segmentation data by using a word embedding model and an attention mechanism to obtain sensitive similarity corresponding to the word segmentation data; determining whether the word segmentation data is sensitive data or not according to the sensitive similarity; if the word segmentation data is sensitive data, a distillation model and a conditional random field model are used for classifying the sensitive data, and a classification result of the sensitive data is obtained, so that the problems that potential sensitive information cannot be recognized at present, so that more missed judgment and misjudgment phenomena exist, and the accuracy rate is low are solved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a text recognition method, device, electronic device and storage medium. Background Art

[0002] With the advent of the big data era, a large amount of text data will be generated. This text data involves sensitive information such as personal privacy, financial information, medical records, etc. Efficient and accurate identification of sensitive information has become one of the core tasks of information security technology.

[0003] In the related art, the sensitive information identification method relies on rule matching or a preset list of sensitive words to realize the identification of sensitive information. The above method can only process some common privacy information, but when faced with diverse and complex semantic scenarios, it is often unable to effectively identify potential sensitive information, resulting in many missed judgments, misjudgments and low accuracy. Summary of the invention

[0004] The technical problem to be solved by the embodiments of the present application is to provide a text recognition method, device, electronic device and storage medium to solve the current problem that potential sensitive information cannot be identified, resulting in many missed judgments, misjudgments and low accuracy.

[0005] In a first aspect, an embodiment of the present application provides a text recognition method, comprising:

[0006] Performing a hash operation on the word segmentation data of the text to be recognized to obtain a first hash value corresponding to the word segmentation data;

[0007] Determining whether the first Hash value successfully matches the second Hash value stored in the database;

[0008] If the first hash value fails to match the second hash value, the word segmentation data is processed using a word embedding model and an attention mechanism to obtain a sensitive similarity corresponding to the word segmentation data;

[0009] Determining whether the word segmentation data is sensitive data according to the sensitive similarity;

[0010] If the word segmentation data is sensitive data, the distillation model and the conditional random field model are used to classify the sensitive data to obtain a classification result of the sensitive data.

[0011] Optionally, the using a word embedding model and an attention mechanism to process the word segmentation data to obtain a sensitive similarity corresponding to the word segmentation data includes:

[0012] Input the word segmentation data into the word embedding model to obtain a word embedding vector corresponding to the word segmentation data;

[0013] Using the attention mechanism to process the word segmentation data, obtaining an attention weight corresponding to the word segmentation data;

[0014] Obtaining a weighted embedding vector corresponding to the word segmentation data according to the word embedding vector and the attention weight;

[0015] The weighted embedding vector is determined as the sensitive similarity corresponding to the word segmentation data.

[0016] Optionally, determining whether the segmented data is sensitive data according to the sensitive similarity includes:

[0017] Calculating the Euclidean distance between the weighted embedding vector and the embedding vector corresponding to the sensitive word stored in the database;

[0018] Determine whether the word segmentation data is sensitive data according to the Euclidean distance.

[0019] Optionally, determining whether the word segmentation data is sensitive data according to the Euclidean distance includes:

[0020] If the Euclidean distance is less than a first sensitivity threshold, determining that the segmented data is sensitive data;

[0021] If the Euclidean distance is between the first sensitivity threshold and the second sensitivity threshold, the segmented word data is determined to be a suspected sensitive word, and the suspected sensitive word is stored in a database of labels to be classified, wherein the first sensitivity threshold is less than the second sensitivity threshold;

[0022] If the Euclidean distance is greater than the second sensitivity threshold, the word segmentation data is determined to be non-sensitive data.

[0023] Optionally, the method further comprises: pre-training the word embedding model;

[0024] The pre-training of the word embedding model comprises:

[0025] Build a vocabulary based on words in the text data;

[0026] Moving a sliding window of a set length on the text data, and counting the co-occurrence frequency of words in the sliding window;

[0027] Constructing a word frequency co-occurrence matrix according to the co-occurrence frequencies;

[0028] The word embedding model is trained based on the vocabulary and the word frequency co-occurrence matrix.

[0029] Optionally, the using the distillation model and the conditional random field model to classify the sensitive data to obtain the classification result of the sensitive data includes:

[0030] Inputting the sensitive data into the distillation model to obtain a context feature vector of the sensitive data;

[0031] Inputting the context feature vector into the conditional random field model to obtain a part-of-speech tag for the sensitive data;

[0032] A classification result of the sensitive data is determined according to the part-of-speech tag.

[0033] Optionally, the distillation model includes: a teacher model and a student model;

[0034] The method further comprises:

[0035] training the distillation model;

[0036] The training of the distillation model comprises:

[0037] Inputting the sample data into the teacher model for model training to obtain a multi-level attention weighted vector corresponding to the sample data;

[0038] Performing knowledge distillation on the multi-level attention weighted vector to obtain a distilled multi-level attention weighted vector;

[0039] The distilled multi-level attention weighted vector is input into the student model to make the student model converge.

[0040] In a second aspect, a text recognition device includes:

[0041] A calculation module, used for performing a hash operation on the word segmentation data of the text to be recognized to obtain a first hash value corresponding to the word segmentation data;

[0042] A judging module, used to judge whether the first Hash value matches the second Hash value stored in the database successfully;

[0043] A processing module, configured to process the word segmentation data using a word embedding model and an attention mechanism to obtain a sensitive similarity corresponding to the word segmentation data if the first hash value fails to match the second hash value;

[0044] A determination module, used to determine whether the segmented data is sensitive data according to the sensitive similarity;

[0045] A classification module is used to classify the sensitive data using a distillation model and a conditional random field model if the segmented data is sensitive data, so as to obtain a classification result of the sensitive data.

[0046] According to a third aspect, an electronic device includes:

[0047] A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the text recognition method is implemented when the processor executes the computer program.

[0048] In a fourth aspect, a computer-readable storage medium stores a computer program that implements the text recognition method when executed by a processor.

[0049] Compared with the prior art, the embodiments of the present application have the following advantages:

[0050] In the embodiment of the present application, by performing hash operations on the word segmentation data of the text to be recognized, known sensitive data can be quickly screened out, significantly improving the processing efficiency of the system, avoiding complex calculations on all data, reducing computing overhead, and improving processing speed. As for potential sensitive data, the word segmentation data is processed using a word embedding model and an attention mechanism to obtain the sensitive similarity corresponding to the word segmentation data; based on the sensitive similarity, it is determined whether the word segmentation data is sensitive data; if the word segmentation data is sensitive data, the sensitive data is classified using a distillation model and a conditional random field model to obtain the classification result of the sensitive data, thereby improving the accuracy of sensitive information recognition and enhancing the ability to expand new sensitive word types, thereby achieving accurate recognition of potential sensitive data and avoiding the occurrence of missed judgments and misjudgments.

[0051] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A flowchart of a text recognition method provided in an embodiment of the present application;

[0053] Figure 2 A flowchart of a text recognition method provided in an embodiment of the present application;

[0054] Figure 3 A flowchart of the steps of text recognition provided in an embodiment of the present application;

[0055] Figure 4 A schematic diagram of the structure of the teacher model provided in the embodiment of the present application;

[0056] Figure 5 A schematic diagram of the structure of a student model provided in an embodiment of the present application;

[0057] Figure 6 A schematic diagram of the structure of the distillation model and the conditional random field model provided in the embodiments of the present application;

[0058] Figure 7 A structural schematic diagram of a text recognition device provided in an embodiment of the present application;

[0059] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0061] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0062] Reference Figure 1 , shows a flowchart of a text recognition method provided by an embodiment of the present application, and the text recognition method may include the following steps:

[0063] Step 101: performing a hash operation on the word segmentation data of the text to be recognized to obtain a first hash value corresponding to the word segmentation data.

[0064] In practical applications, before performing a hash operation on the word segmentation data of the text to be recognized, the text to be recognized needs to be preprocessed to obtain a plurality of word segmentation data.

[0065] Preprocessing the file to be recognized to obtain multiple segmentation data includes: performing segmentation and regularization on the text to be recognized to obtain multiple segmentation data, that is, using a dictionary-based two-way maximum matching algorithm to segment the text to be recognized, and retaining special words in combination with part-of-speech tagging, and removing stop words, punctuation marks, etc. in the segmentation data based on regularization rules to obtain multiple segmentation data.

[0066] A hash operation is performed on the multiple word segmentation data to calculate a first hash value corresponding to each word segmentation data.

[0067] Among them, hash operation, also known as hash algorithm, is an algorithm that converts an input of any length (usually called "message" or "data") into an output of a fixed length (i.e., hash value or hash code) through a hash function. The hash conversion process is irreversible, that is, the original input data cannot be restored from the hash value. The hash algorithm is widely used in data integrity verification, information security, digital signatures, fast data search and other fields.

[0068] Step 102: Determine whether the first Hash value matches the second Hash value stored in the database successfully. If the first Hash value fails to match the second Hash value, execute step 103; if the first Hash value matches the second Hash value successfully, execute step 106.

[0069] Determine whether the first hash value matches the second hash value stored in the database. If the first hash value fails to match the second hash value, use the word embedding model and attention mechanism to process the word segmentation data. If the first hash value matches the second hash value successfully, determine that the word segmentation data is sensitive data. By using the hash algorithm, known sensitive data can be quickly screened out, which significantly improves the processing efficiency of the system, avoids complex calculations on all data, reduces computing overhead, and improves processing speed.

[0070] Step 103: Use the word embedding model and the attention mechanism to process the word segmentation data to obtain the sensitive similarity corresponding to the word segmentation data.

[0071] The word embedding model is an important means of language model and representation learning technology in natural language processing. The word embedding model embeds a high-dimensional space of all word numbers in a dimension into a continuous vector space with a much lower dimension, and each word or phrase is mapped to a vector in the real number field. This technology enables machines to understand and process human language by capturing the semantic relationship between words.

[0072] The attention mechanism is an important technology in deep learning. Its core idea is to enable the model to selectively focus on different parts of the input sequence and assign different weights to these parts to highlight the information that is more critical to the task. This mechanism imitates the way humans allocate attention when processing information, that is, when faced with a large amount of information, humans will selectively focus on important parts and ignore other irrelevant information.

[0073] After the first hash value fails to match the second hash value, the present application uses a word embedding model and an attention mechanism to process the word segmentation data to obtain the sensitive similarity corresponding to the word segmentation data. The word embedding model and the attention mechanism can be used to accurately identify potential sensitive data.

[0074] Step 104: Determine whether the segmented data is sensitive data according to the sensitivity similarity.

[0075] Step 105: If the segmented data is sensitive data, the distillation model and the conditional random field model are used to classify the sensitive data to obtain a classification result of the sensitive data.

[0076] If the word segmentation data is sensitive data, the distillation model and the conditional random field model are used to classify the sensitive data to obtain the classification result of the sensitive data. If the word segmentation data is non-sensitive data, that is, the word segmentation data is ordinary data, no processing is performed.

[0077] Step 106: Determine that the segmented data is sensitive data.

[0078] In the embodiment of the present application, by performing hash operations on the word segmentation data of the text to be identified, known sensitive data can be quickly screened out, which significantly improves the processing efficiency of the system, avoids complex calculations on all data, reduces computing overhead, and improves processing speed. As for potential sensitive data, the word segmentation data is processed using a word embedding model and an attention mechanism to obtain the sensitive similarity corresponding to the word segmentation data; based on the sensitive similarity, it is determined whether the word segmentation data is sensitive data; if the word segmentation data is sensitive data, the sensitive data is classified using a distillation model and a conditional random field model to obtain the classification result of the sensitive data, thereby improving the accuracy of sensitive information identification and enhancing the ability to expand new sensitive word types, thereby achieving accurate identification of potential sensitive data and avoiding the occurrence of missed judgments and misjudgments.

[0079] Reference Figure 2 , shows a flowchart of a text recognition method provided by an embodiment of the present application, and the text recognition method may include the following steps:

[0080] Step 201: performing a hash operation on the word segmentation data of the text to be recognized to obtain a first hash value corresponding to the word segmentation data.

[0081] Step 202: Determine whether the first Hash value matches the second Hash value stored in the database successfully. If the first Hash value fails to match the second Hash value, execute step 203; if the first Hash value matches the second Hash value successfully, execute step 209.

[0082] Step 203: Input the word segmentation data into the word embedding model to obtain a word embedding vector corresponding to the word segmentation data.

[0083] Among them, the word embedding model of the present application can be GloVe (Global Vectors for Word Representation, a word embedding model based on global word frequency statistics), or other models, which is not specifically limited in the present application.

[0084] In practical applications, before the word segmentation data is input into the word embedding model to obtain the word embedding vector corresponding to the word segmentation data, the word embedding model needs to be trained in advance. The training of the word embedding model includes the following steps:

[0085] 1. Build a vocabulary based on the words in the text data.

[0086] In practical applications, text data is segmented and stop words are removed, that is, common meaningless words are removed to obtain multiple words, and then repeated words in the multiple words are removed during the statistical process to obtain multiple processed words.

[0087] Among them, the frequency filtering strategy is generally used to remove duplicate words from multiple words, ignoring words with extremely low occurrence times to avoid the influence of rare words on subsequent processing.

[0088] The processed multiple words are sorted in alphabetical order or by the frequency of occurrence of the words to construct a unique vocabulary, which is used for the subsequent construction and training of the word embedding model to ensure that the vocabulary is aligned with the words in the known sensitive word list to avoid missing important sensitive words.

[0089] Use the pre-trained word embedding model to build a word embedding vector for each word in the vocabulary, ensuring that each word can be mapped to a vector space of a certain fixed dimension.

[0090] The constructed word embedding vector is standardized to ensure that the constructed word embedding vector has a uniform scale, and the standardized word embedding vector is saved as a binary file for subsequent use.

[0091] 2. Move a sliding window of a set length on the text data, and count the co-occurrence frequencies of words in the sliding window.

[0092] In practical applications, the length of the sliding window can be predefined. Specifically, the length of the sliding window can be set according to the vocabulary distribution. For example, the length of the sliding window is set within the range of 3-5 words. The sliding window is used to count the co-occurrence relationship of words in text data to ensure that the short-distance dependency relationship between words is captured.

[0093] Perform co-occurrence statistics on the words in each sliding window, record the number of times each word co-occurs with its neighboring words, and generate the word co-occurrence matrix E c ,The co-occurrence matrix is ​​a symmetric matrix, where the rows and columns represent each word in the vocabulary, and the elements in the matrix represent the frequency of ,word co-occurrence.

[0094] 3. Construct a word frequency co-occurrence matrix based on the co-occurrence frequencies.

[0095] 4. Train the word embedding model based on the vocabulary and the word frequency co-occurrence matrix.

[0096] Define the word embedding model (Glove) objective function J, through which the relationship between the word embedding vector and the word frequency matrix corresponding to the vocabulary is made as close as possible.

[0097]

[0098] Among them, X ij is the co-occurrence frequency of word i and word j in the word frequency matrix, and V is the length of the vocabulary.

[0099] w i and w j are the word embedding vectors of word i and word j respectively.

[0100] b i and b j are the bias terms for word i and word j respectively.

[0101] f(X ij ) is a weight function used to adjust the weights of word pairs with different frequencies.

[0102] Use the gradient descent optimization algorithm to minimize the objective function J and get the word embedding vector w of the vocabulary i and w j , the trained word vectors are kept in binary text format for quick loading and use in subsequent processing.

[0103] In the word embedding model (Glove), the reconstruction error of the word frequency matrix can be obtained by P ij and P i To express.

[0104]

[0105] Among them, F(w i ,w j ) is the reconstruction error of the word frequency matrix.

[0106] P ij is the co-occurrence probability of word i and word j, P i is the marginal probability of word i.

[0107] By minimizing the reconstruction error of word vectors and word frequency matrices, word embedding models are able to capture the semantic relationship between words and thus perform well in various natural language processing tasks.

[0108] Step 204: Use the attention mechanism to process the word segmentation data to obtain the attention weight corresponding to the word segmentation data.

[0109] The attention mechanism is used to calculate the attention weight corresponding to each word segmentation data, and the attention weight is calculated by the following formula.

[0110] A=softmax(W a Z+b a )

[0111] Among them, A is the attention weight, Z is the word embedding vector corresponding to each word segmentation data after conversion, and W a is the weight matrix, b a Bias vector, softmax is the activation function.

[0112] Step 205: Obtain a weighted embedding vector corresponding to the word segmentation data according to the word embedding vector and the attention weight.

[0113] By applying the attention weights to the word embedding vector, we can get the weighted embedding vector corresponding to the word segmentation data, which is achieved by the following formula.

[0114] weighted_Z=A·Z

[0115] weighted_Z is the weighted embedding vector, A is the attention weight, and Z is the corresponding word embedding vector after each word segmentation data is converted.

[0116] Step 206: Determine the weighted embedding vector as the sensitive similarity corresponding to the word segmentation data.

[0117] Step 207: Determine whether the segmented data is sensitive data according to the sensitivity similarity.

[0118] In practical applications, step 207 includes the following sub-steps:

[0119] Sub-step 2071: Calculate the Euclidean distance between the weighted embedding vector and the embedding vector corresponding to the sensitive word stored in the database.

[0120] The Euclidean distance between the weighted embedding vector and the embedding vector corresponding to the sensitive word stored in the database is calculated by the following formula:

[0121]

[0122] Among them, d is the Euclidean distance, x i is the word embedding vector of word i, y i They are the corresponding word embedding vectors in the word i database

[0123] Sub-step 2072: Determine whether the segmented data is sensitive data according to the Euclidean distance.

[0124] Before determining whether the word segmentation data is sensitive data according to the Euclidean distance, a first sensitivity threshold α1 and a second sensitivity threshold α2 are set, where the first sensitivity threshold is less than the second sensitivity threshold (α1 < α2). If the Euclidean distance is less than the first sensitivity threshold (d < α1), it is determined that the word segmentation data is sensitive data.

[0125] If the Euclidean distance is between the first sensitivity threshold and the second sensitivity threshold (α1 < d < α2), it is determined that the word segmentation data is a suspected sensitive word, and the suspected sensitive word is stored in the database of labels to be classified for subsequent update of new sensitive types.

[0126] If the Euclidean distance is greater than the second sensitivity threshold (d > α2), it is determined that the word segmentation data is non-sensitive data, that is, the word segmentation data is an ordinary word.

[0127] Step 208: If the word segmentation data is sensitive data, use the distillation model and the conditional random field model to classify the sensitive data to obtain the classification result of the sensitive data.

[0128] The distillation model includes: a teacher model and a student model. The teacher model and the student model are set as Bert (Bidirectional Encoder Representations from Transformer, pre-trained model)-BiLSTM (Bidirectional Long Short-Term Memory, bidirectional long short-term memory network)-Triplet Attention (attention mechanism). Then, the teacher model is trained, and the parameters of the teacher model are passed into the student model for training.

[0129] Step 209: Determine that the word segmentation data is sensitive data.

[0130] In the embodiment of the present application, by setting double sensitivity thresholds, sensitive words and suspected sensitive words can be clearly distinguished. The sensitive words that are not fully determined are classified into the database of labels to be classified for subsequent processing, improving the accuracy of sensitive information recognition and enhancing the system's ability to expand new sensitive word types.

[0131] In the embodiment of the present application, by introducing a mechanism for storing suspected sensitive words, the uncertain sensitive words are classified into the database of labels to be classified, and new sensitive word types are dynamically expanded during subsequent updates, enhancing the system's self-adaptability and enabling it to meet the changing privacy information requirements.

[0132] Refer to Figure 3 , which shows a flowchart of the steps of a text recognition method provided by the embodiment of the present application. The text recognition method may include the following steps:

[0133] Step 301: performing a hash operation on the word segmentation data of the text to be recognized to obtain a first hash value corresponding to the word segmentation data.

[0134] Step 302: Determine whether the first Hash value matches the second Hash value stored in the database successfully. If the first Hash value fails to match the second Hash value, execute step 303; if the first Hash value matches the second Hash value successfully, execute step 308.

[0135] Step 303: Use the word embedding model and the attention mechanism to process the word segmentation data to obtain the sensitive similarity corresponding to the word segmentation data.

[0136] Step 304: Determine whether the segmented data is sensitive data according to the sensitivity similarity.

[0137] If the word segmentation data is sensitive data, the distillation model and the conditional random field model are used to classify the sensitive data to obtain the classification result of the sensitive data. If the word segmentation data is non-sensitive data, that is, the word segmentation data is ordinary data, no processing is performed.

[0138] Step 305: Input the sensitive data into the distillation model to obtain a context feature vector of the sensitive data.

[0139] The distillation model includes: a teacher model and a student model, and the teacher model and the student model are set to Bert-BiLSTM-Triplet Attention; the teacher model is trained, and the teacher model parameters are passed into the student model, and sensitive words are classified in combination with the conditional random field model.

[0140] Before inputting sensitive data into the distillation model, the distillation model needs to be trained. Training the distillation model includes:

[0141] 1. Input the sample data into the teacher model for model training to obtain a multi-level attention weighted vector corresponding to the sample data.

[0142] 2. Perform knowledge distillation on the multi-level attention weighted vector to obtain a distilled multi-level attention weighted vector.

[0143] Knowledge distillation includes: intermediate layer distillation, output layer distillation and weight adjustment. The layer-by-layer distillation process is carried out through the intermediate layer and the output layer, and the distillation temperature is adjusted to ensure the convergence effect of the student model to obtain the multi-level attention weighted vector after distillation.

[0144] The teacher model is set to Bert-BiLSTM-Triplet Attention. Figure 4 As shown in the figure, Bert is a 12-layer Transformer, BiLSTM (Bidirectional Long Short-Term Memory) is a multi-layer bidirectional LSTM structure, and Triplet Attention is a three-branch attention mechanism.

[0145] Teacher model: Usually a complex, large model with high accuracy and generalization ability. The teacher model can be a deep neural network.

[0146] The sample data is input into the teacher model for model training to obtain a multi-level attention weighted vector corresponding to the sample data.

[0147] The first branch of the teacher model is BiLSTM combined with Triplet Attention, that is, the bidirectional LSTM module combined with Triplet Attention. The first branch extracts the context dependency. BiLSTM consists of two LSTM layers: a forward LSTM layer and a reverse LSTM layer. The forward LSTM layer processes data from the beginning to the end of the sequence, while the reverse LSTM layer processes data from the end to the beginning of the sequence. The outputs of the two LSTM layers will be spliced ​​together to form the final output.

[0148] The specific calculation of the bidirectional LSTM module is as follows:

[0149]

[0150] in, Forward transmission sequence, is the forward hidden state, is the reverse transmission sequence, is the reverse hidden state

[0151] Bidirectional LSTM module output layer O t The calculation formula is:

[0152] O t =tanh(W h h t +b o );

[0153] Among them, W h is the weight matrix of the output layer, h t is the hidden state at time step t, b o is the bias vector of the output layer, and tanh is the hyperbolic tangent activation function, which is used to compress the output value into the range of (-1,1).

[0154] The first attention mechanism module in the first branch uses the word attention mechanism. The word attention score of this branch is calculated as follows:

[0155] u i =tanh(W w h i +b w ),

[0156] Among them, u i is the attention score at time step i, W w is the weight matrix, h i is the hidden state at time step t, b w is the bias vector and tanh is the hyperbolic tangent activation function.

[0157] The calculation formula of attention weight A is as follows:

[0158]

[0159] Among them, A is the attention weight, u i is the attention score at time step i, u w is the query vector, T is the length of the input sequence, exp is the exponential function, is the sum of the attention scores over all time steps.

[0160] The weighted embedding vector is obtained by multiplying the attention weights with the hidden state of the input sequence and summing them.

[0161] The calculation formula of the weighted embedding vector Attn1 is:

[0162]

[0163] Among them, Attn1 is the weighted embedding vector, A is the attention weight, and h i is the hidden state at time step t.

[0164] The second branch of the teacher model is the word attention mechanism, which is used to extract the weight of words in sentences, perform feature decomposition (Z-Pool) on the data, obtain the feature matrix, and then perform convolution operation to capture word features. The specific calculation is as follows:

[0165] Z-Pool(W)=[MaxPool 0d (W),AvgPool 0d (W)]

[0166] Among them, MaxPool 0d (W) is the maximum pooling result of the input feature map W, AvgPool 0d(W) is the average pooling result of the input feature map W, and [.,.] means concatenating the two results together.

[0167] A=softmax(W a Z+b a )

[0168] Among them, A is the attention weight, Z is the word embedding vector corresponding to each word segmentation data after conversion, and W a is the weight matrix, b a Bias vector, softmax is the activation function.

[0169] Attn2=A·Sigmoid(Z)

[0170] Among them, Attn2 is the weighted embedding vector, A is the attention weight, Z is the input feature map, and Sigmoid is the Sigmoid activation function.

[0171] The third branch of the teacher model is the sentence attention mechanism, which is used to extract the weight of the sentence in the article, decompose the data into features, obtain the feature matrix, and then perform convolution operations to capture the sentence features. The specific calculation is as follows:

[0172] Z-Pool(H)=[MaxPool 0d (H),AvgPool 0d (H)]

[0173] Among them, MaxPool 0d (H) is the maximum pooling result of the input feature map H, AvgPool 0d (H) is the average pooling result of the input feature map H, and [.,.] means concatenating the two results together.

[0174] A=softmax(W a Z+b a )

[0175] Among them, A is the attention weight, Z is the word embedding vector corresponding to each word segmentation data after conversion, and W a is the weight matrix, b a Bias vector, softmax is the activation function.

[0176] Attn3=A·Sigmoid(Z)

[0177] Among them, Attn3 is the weighted embedding vector, A is the attention weight, Z is the input feature map, and Sigmoid is the Sigmoid activation function.

[0178] The attention weights of the three branches of the teacher model are integrated to obtain the final output of the teacher model as multi-level attention information, which can more comprehensively capture the characteristics of sensitive words and obtain a weighted vector combining word and sentence attention.

[0179] Attn=A(σ(Attn1)+σ(Attn2)+σ(Attn3))

[0180] Among them, σ is the Sigmoid activation function, and Attn is the sum of the weighted embedding vectors of the three branches.

[0181] 3. Input the distilled multi-level attention weighted vector into the student model for model training to make the student model converge.

[0182] The student model is set to a simplified version of Bert-BiLSTM-Triplet Attention. Figure 5 As shown in the figure, Bert is a 6-layer Transformer, BiLSTM is half the number of layers in the BiLSTM structure in the teacher model, and TripletAttention is simplified to a single branch in the three-branch attention mechanism.

[0183] Student model: Usually a simpler, smaller model that aims to mimic the behavior of the teacher model. The student model can be a shallow neural network, a linear model, or other lightweight model.

[0184] Step 306: Input the context feature vector into the conditional random field model to obtain the part-of-speech tag of the sensitive data.

[0185] Step 307: Determine the classification result of the sensitive data according to the part-of-speech tag.

[0186] The contextual features of the obtained sensitive data are input into the CRF (conditional random field) model to obtain the dependency relationship between the sensitive data, further annotate the part-of-speech tags of each sensitive data, identify the boundaries of sensitive phrases composed of multiple words, and obtain the type of sensitive data.

[0187] The CRF model defines a conditional probability distribution P(Y|X), where X is the input sequence x={x1,x2,…,x n}, x is the context feature of sensitive data, Y is the output sequence (label sequence) y = {y1, y2, ..., y n}, y is the part-of-speech tag. The goal of the CRF model is to find the tag sequence Y that maximizes the conditional probability P(Y|X), which is evaluated by maximum likelihood estimation.

[0188]

[0189] Among them, Score(X,Y) is the score of the input sequence X and the output sequence Y.

[0190] Z(X) is a normalization factor that ensures the probabilities sum to 1.

[0191] The log-likelihood function L(P(y|x)) can be expressed as:

[0192] L(P(y|x))=∑ i log(P(y|x));

[0193] Substituting the definition of conditional probability P(y|x) into the log-likelihood function, we obtain:

[0194]

[0195] Further expansion:

[0196] L(P(y|x))=∑ i (log(exp(Score(X,Y)))-log(Z(X)))

[0197] Simplified:

[0198] L(P(y|x))=∑ i (Score(X,Y)-log(Z(X)))

[0199] Among them, Score(X,Y) is the score of the input sequence X and the output sequence Y.

[0200] log(Z(X)) is the logarithm of the normalization factor.

[0201] Step 308: Determine that the segmented data is sensitive data.

[0202] In order for those skilled in the art to better understand the use of the distillation model and the conditional random field model to classify the sensitive data, obtain the classification results of the sensitive data, and Figure 6 Take this as an example to explain the implementation process in detail.

[0203] The unlabeled sample data is preprocessed to remove the noise of the sample data to obtain the sample data, the sample data is input into the teacher model for training to obtain the multi-level attention weighted vector corresponding to the sample data, the multi-level attention weighted vector is subjected to knowledge distillation, and the distilled multi-level attention weighted vector is input into the student model to make the student model converge. In this way, when the word segmentation data is sensitive data, the student model is used to process the sensitive data to obtain the context feature vector of the sensitive data, the context feature vector is input into the conditional random field model to obtain the dependency relationship of the sensitive data and the part-of-speech tag of the sensitive data, and the classification result of the sensitive data is determined according to the part-of-speech tag.

[0204] In the embodiment of the present application, Bert-BiLSTM-Triplet Attention is used as the teacher model, and the layer-by-layer knowledge distillation technology is used to pass complex model parameters to the simplified student model, so that the system achieves efficient sensitive data classification. While maintaining high classification accuracy, the student model significantly reduces the computational complexity and achieves the effect of lightweight processing.

[0205] In the embodiment of the present application, by adjusting the distillation temperature, the distillation model can better balance the accuracy and convergence speed during the training process, ensure the rapid convergence of the student model, achieve efficient reasoning performance and real-time processing capabilities, and is suitable for sensitive data identification needs in large-scale data scenarios.

[0206] Reference Figure 7 , shows a schematic diagram of the structure of a text recognition device provided in an embodiment of the present application, which specifically includes the following modules:

[0207] The calculation module 701 is used to perform a hash operation on the word segmentation data of the text to be recognized to obtain a first hash value corresponding to the word segmentation data.

[0208] The judgment module 702 is used to judge whether the first Hash value matches the second Hash value stored in the database successfully.

[0209] The processing module 703 is used to process the word segmentation data using a word embedding model and an attention mechanism to obtain a sensitive similarity corresponding to the word segmentation data if the first hash value fails to match the second hash value.

[0210] The determination module 704 is used to determine whether the segmented data is sensitive data according to the sensitive similarity.

[0211] The classification module 705 is used to classify the sensitive data using a distillation model and a conditional random field model to obtain a classification result of the sensitive data if the segmented data is sensitive data.

[0212] Optionally, the processing module includes:

[0213] A word embedding vector unit, used to input the word segmentation data into the word embedding model to obtain a word embedding vector corresponding to the word segmentation data;

[0214] An attention weight unit, used to process the word segmentation data using the attention mechanism to obtain an attention weight corresponding to the word segmentation data;

[0215] A weighted embedding vector unit, used to obtain a weighted embedding vector corresponding to the word segmentation data according to the word embedding vector and the attention weight;

[0216] A sensitive similarity unit is used to determine the weighted embedding vector as the sensitive similarity corresponding to the word segmentation data.

[0217] Optionally, the determining module includes:

[0218] A calculation unit, used to calculate the Euclidean distance between the weighted embedding vector and the embedding vector corresponding to the sensitive word stored in the database;

[0219] A determination unit is used to determine whether the segmented data is sensitive data according to the Euclidean distance.

[0220] Optionally, the determining unit is specifically configured to determine that the segmented data is sensitive data if the Euclidean distance is less than a first sensitivity threshold;

[0221] If the Euclidean distance is between the first sensitivity threshold and the second sensitivity threshold, the segmented word data is determined to be a suspected sensitive word, and the suspected sensitive word is stored in a database of labels to be classified, wherein the first sensitivity threshold is less than the second sensitivity threshold;

[0222] If the Euclidean distance is greater than the second sensitivity threshold, the word segmentation data is determined to be non-sensitive data.

[0223] Optionally, the device further comprises: a first training module, used for pre-training the word embedding model;

[0224] The first training module includes:

[0225] Lexical units, used to build a vocabulary based on words in text data;

[0226] A statistical unit, used for moving a sliding window of a set length on the text data to count the co-occurrence frequency of words in the sliding window;

[0227] A construction unit, used to construct a word frequency co-occurrence matrix according to the co-occurrence frequency;

[0228] A training unit is used to train the word embedding model according to the vocabulary and the word frequency co-occurrence matrix.

[0229] Optionally, the device further comprises:

[0230] A sensitive module is used to determine that the segmented data is sensitive data if the first Hash value matches the second Hash value successfully.

[0231] Optionally, the classification module includes:

[0232] A distillation unit, used for inputting the sensitive data into the distillation model to obtain a context feature vector of the sensitive data;

[0233] A conditional random field unit, used for inputting the context feature vector into the conditional random field model to obtain a part-of-speech tag for the sensitive data;

[0234] A classification unit is used to determine a classification result of the sensitive data according to the part-of-speech tag.

[0235] Optionally, the distillation model includes: a teacher model and a student model;

[0236] The device also includes: a second training module, used to train the distillation model;

[0237] The second training module includes:

[0238] A teacher training unit, used for inputting sample data into the teacher model for model training to obtain a multi-level attention weighted vector corresponding to the sample data;

[0239] A knowledge distillation unit, used to perform knowledge distillation on the multi-level attention weighted vector to obtain a distilled multi-level attention weighted vector;

[0240] The student training unit inputs the distilled multi-level attention weighted vector into the student model to make the student model converge.

[0241] The embodiment of the present application provides a method for quickly screening out known sensitive data by performing a hash operation on the word segmentation data of the text to be identified, significantly improving the processing efficiency of the system, avoiding complex calculations on all data, reducing computing overhead, and improving processing speed. As for potential sensitive data, the word segmentation data is processed using a word embedding model and an attention mechanism to obtain the sensitive similarity corresponding to the word segmentation data; based on the sensitive similarity, it is determined whether the word segmentation data is sensitive data; if the word segmentation data is sensitive data, the sensitive data is classified using a distillation model and a conditional random field model to obtain the classification result of the sensitive data, thereby improving the accuracy of sensitive information identification and enhancing the system's ability to expand new sensitive word types, thereby achieving accurate identification of potential sensitive data and avoiding the occurrence of missed judgments and misjudgments.

[0242] An embodiment of the present application further provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the text recognition method when executing the computer program.

[0243] Figure 8 FIG. 8 is a schematic diagram showing the structure of an electronic device 800 according to an embodiment of the present invention. Figure 8 As shown, the electronic device 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 802 or computer program instructions loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 can also be stored. The CPU 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0244] Multiple components in the electronic device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, a microphone, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0245] The various processes and processing described above may be performed by the processing unit 801. For example, the method of any of the above embodiments may be implemented as a computer software program, which is tangibly contained in a computer-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the CPU 801, one or more actions in the method described above may be performed.

[0246] In addition, an embodiment of the present application further provides a computer-readable storage medium, which stores a computer program that implements the text recognition method when executed by a processor.

[0247] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0248] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, devices, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0249] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminals (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0250] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0251] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal so that a series of operating steps are executed on the computer or other programmable terminal to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable terminal. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0252] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0253] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or terminal including the elements.

[0254] The above is a detailed introduction to a text recognition method, a text recognition device, an electronic device and a computer-readable storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for a person skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A text recognition method, characterized in that: include: Performing a hash operation on the word segmentation data of the text to be recognized to obtain a first hash value corresponding to the word segmentation data; Determining whether the first Hash value successfully matches the second Hash value stored in the database; If the first hash value fails to match the second hash value, the word segmentation data is processed using a word embedding model and an attention mechanism to obtain a sensitive similarity corresponding to the word segmentation data; Determining whether the word segmentation data is sensitive data according to the sensitive similarity; If the word segmentation data is sensitive data, the distillation model and the conditional random field model are used to classify the sensitive data to obtain a classification result of the sensitive data.

2. The method according to claim 1, characterized in that The word embedding model and the attention mechanism are used to process the word segmentation data to obtain the sensitive similarity corresponding to the word segmentation data, which includes: Input the word segmentation data into the word embedding model to obtain a word embedding vector corresponding to the word segmentation data; Using the attention mechanism to process the word segmentation data, obtaining an attention weight corresponding to the word segmentation data; Obtaining a weighted embedding vector corresponding to the word segmentation data according to the word embedding vector and the attention weight; The weighted embedding vector is determined as the sensitive similarity corresponding to the word segmentation data.

3. The method according to claim 2, characterized in that Determining whether the segmented data is sensitive data according to the sensitive similarity includes: Calculating the Euclidean distance between the weighted embedding vector and the embedding vector corresponding to the sensitive word stored in the database; Determine whether the word segmentation data is sensitive data according to the Euclidean distance.

4. The method according to claim 3, characterized in that Determining whether the word segmentation data is sensitive data according to the Euclidean distance includes: If the Euclidean distance is less than a first sensitivity threshold, determining that the segmented data is sensitive data; If the Euclidean distance is between the first sensitivity threshold and the second sensitivity threshold, the segmented word data is determined to be a suspected sensitive word, and the suspected sensitive word is stored in a database of labels to be classified, wherein the first sensitivity threshold is less than the second sensitivity threshold; If the Euclidean distance is greater than the second sensitivity threshold, the word segmentation data is determined to be non-sensitive data.

5. The method according to claim 1, characterized in that The method further comprises: pre-training the word embedding model; The pre-training of the word embedding model comprises: Build a vocabulary based on words in the text data; Moving a sliding window of a set length on the text data, and counting the co-occurrence frequency of words in the sliding window; Constructing a word frequency co-occurrence matrix according to the co-occurrence frequencies; The word embedding model is trained based on the vocabulary and the word frequency co-occurrence matrix.

6. The method according to claim 1, characterized in that The using the distillation model and the conditional random field model to classify the sensitive data to obtain the classification result of the sensitive data includes: Inputting the sensitive data into the distillation model to obtain a context feature vector of the sensitive data; Inputting the context feature vector into the conditional random field model to obtain a part-of-speech tag for the sensitive data; A classification result of the sensitive data is determined according to the part-of-speech tag.

7. The method according to any one of claims 1 to 6, characterized in that: The distillation model includes: a teacher model and a student model; The method further comprises: training the distillation model; The training of the distillation model comprises: Inputting the sample data into the teacher model for model training to obtain a multi-level attention weighted vector corresponding to the sample data; Performing knowledge distillation on the multi-level attention weighted vector to obtain a distilled multi-level attention weighted vector; The distilled multi-level attention weighted vector is input into the student model to make the student model converge.

8. A text recognition device, characterized in that: include: A calculation module, used for performing a hash operation on the word segmentation data of the text to be recognized to obtain a first hash value corresponding to the word segmentation data; A judging module, used to judge whether the first Hash value matches the second Hash value stored in the database successfully; A processing module, configured to process the word segmentation data using a word embedding model and an attention mechanism to obtain a sensitive similarity corresponding to the word segmentation data if the first hash value fails to match the second hash value; A determination module, used to determine whether the segmented data is sensitive data according to the sensitive similarity; A classification module is used to classify the sensitive data using a distillation model and a conditional random field model if the segmented data is sensitive data, so as to obtain a classification result of the sensitive data.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the text recognition method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: When the computer-readable storage medium stores a computer program that implements the text recognition method according to any one of claims 1 to 7 when executed by a processor.

Images