A variant text recognition method, device and readable storage medium

Abstract

Embodiments of the present application provide a variant text recognition method, device and readable storage medium, comprising: identifying a suspected sensitive word in an original text, replacing the suspected sensitive word with a mask identifier to obtain a masked text, and converting each element in the masked text into a vector to obtain a masked text vector sequence; replacing each Chinese character in the original text with a corresponding pinyin to obtain a pinyin text, and converting each element in the pinyin text into a vector to obtain a pinyin text vector sequence; inputting the masked text vector sequence and the pinyin text vector sequence into a pre-trained variant text recognition model for text representation processing and domain knowledge enhancement processing to obtain a predicted text; comparing the original text and the predicted text, and determining an element different from the predicted text in the original text as a variant text.

Description

Technical Field

[0001] This invention relates to the field of text recognition technology, and more particularly to a variant text recognition method, apparatus, and readable storage medium. Background Technology

[0002] Variant text recognition is a text processing method for identifying variant and sensitive information in social networks. After variant text is published on social networks, the platform uses heuristics and language model methods to recall and handle it, ensuring the security of online content. Heuristic methods match variant text using a confusion dictionary and identify it according to rules, offering advantages such as ease of use and good interpretability. Language model methods use models that learn text representations from large-scale corpora to identify variant text, offering advantages such as strong fitting ability and strong generalization ability.

[0003] Currently, variant text recognition typically employs a confusion dictionary and a pre-trained language model. The constructed confusion dictionary is used to match candidate words, and variant text is identified based on lexical and grammatical analysis. When similar words are encountered, the confusion dictionary needs to be updated. The language model performs end-to-end variant text recognition through masked language modeling tasks or substitution marker detection tasks. When the variant text belongs to a small sample domain, the pre-trained sample set needs to be adjusted, and the text representation updated.

[0004] Existing variant text recognition technologies are mostly based on general domains. However, for small-sample domains where learnable knowledge is difficult to obtain and the sample size is small, the knowledge types and distributions differ significantly from those in general domains. Therefore, existing technologies have weak recognition capabilities in small-sample domains. Heuristic methods require experts to extract effective patterns to update confusion dictionaries and lexical and grammatical rules, which limits the scope of consideration and results in poor learning ability and portability. Language model-based recognition methods lack sufficient sample size in small-sample domains. When using pre-training methods, the distribution of pre-trained samples differs significantly from the domain sample distribution, resulting in insufficient representation ability, weak generalization ability, and inability to recall variant texts. To address the shortcomings of these existing technologies, this invention proposes a variant text recognition method based on domain knowledge enhancement and a language model. This method does not require expert rules and exhibits excellent fitting and generalization capabilities in small-sample domains.

[0005] In the process of developing this invention, the applicant discovered at least the following problems in the prior art:

[0006] It is difficult to obtain learnable knowledge, and it has weak recognition ability, poor portability, weak generalization ability, insufficient representation ability, and inability to effectively recall variant texts in small sample domains. Summary of the Invention

[0007] This invention provides a variant text recognition method, apparatus, and readable storage medium, which solves the problems in the prior art of difficulty in obtaining learnable knowledge, weak recognition ability in small sample domains, poor portability, weak generalization ability, insufficient representation ability, and inability to effectively recall variant texts.

[0008] To achieve the above objectives, in one aspect, embodiments of the present invention provide a variant text recognition method, comprising:

[0009] Identify suspected sensitive words in the original text, replace the suspected sensitive words with mask identifiers to obtain masked text, and convert each element in the masked text into a vector to obtain a masked text vector sequence;

[0010] Each Chinese character in the original text is replaced with its corresponding pinyin to obtain pinyin text, and each element of the pinyin text is converted into a vector to obtain a pinyin text vector sequence;

[0011] The masked text vector sequence and the pinyin text vector sequence are input into a pre-trained variant text recognition model for text representation processing and domain knowledge enhancement processing to obtain the predicted text.

[0012] The original text and the predicted text are compared, and elements in the original text that are different from the predicted text are identified as variant texts.

[0013] On the other hand, embodiments of the present invention provide a variant text recognition device, comprising:

[0014] A masking unit is used to identify suspected sensitive words in the original text, replace the suspected sensitive words with masking identifiers to obtain masked text, and convert each element in the masked text into a vector to obtain a masked text vector sequence.

[0015] The Pinyin unit is used to replace each Chinese character in the original text with the corresponding Pinyin to obtain Pinyin text, and to convert each element of the Pinyin text into a vector to obtain a Pinyin text vector sequence;

[0016] The prediction unit is used to input the masked text vector sequence and the pinyin text vector sequence into a pre-trained variant text recognition model for text representation processing and domain knowledge enhancement processing to obtain the predicted text;

[0017] A comparison unit is used to compare the original text and the predicted text, and to identify elements in the original text that are different from the predicted text as variant text.

[0018] On the other hand, embodiments of the present invention provide a readable storage medium storing program code for implementing any of the methods described above.

[0019] The above technical solution has the following beneficial effects: It converts the original text into pinyin text and further into a pinyin text vector sequence. The pinyin text generalizes and blurs the original text to a certain extent, and the pinyin text vector sequence carries information from the original text. It identifies suspected sensitive words in the original text, replaces them with mask identifiers, and quantizes them to obtain a mask text vector sequence. The pinyin text vector sequence and the mask text vector sequence are combined and processed by a variant text recognition model to predict the word at the corresponding mask identifier position, obtaining the predicted text. The variant text is determined by comparing the predicted text with the original text. Utilizing the generalization and blurring characteristics of pinyin text increases the vocabulary range that can be covered by prediction and improves the accuracy of variant text prediction. Furthermore, a variant text recognition model combining RoBERTa layers and GAT layers is established. The attention mechanism of the graph attention neural network in the GAT layer improves the prediction effect of RoBERTa semantic analysis in determining the word at the mask identifier position. Furthermore, in the process of combining the RoBERTa layer and the GAT layer, a feature fusion mechanism is used to fuse the text semantic representation vector sequence output by the RoBERTa layer with the input pinyin text vector sequence of the GAT layer. This allows the input information of the GAT layer to simultaneously contain information from the semantic analysis and prediction of the RoBERTa layer, as well as information from the original text. This enables the GAT layer to make further predictions using more comprehensive information. The prediction results of the RoBERTa layer and the GAT layer are further fused, strengthening the accuracy of semantic prediction results through the attention mechanism. The various models in this embodiment of the invention, combined as a whole, significantly improve the accuracy of variant text recognition. Furthermore, through two-stage pre-training, small-sample domain representations are learned on the basis of general representations. Therefore, without the need for expert-led heuristic rules, it possesses excellent learning ability and portability. By encoding domain knowledge through GAT and using the feature fusion mechanism, the distribution difference between pre-training samples and domain samples is reduced. Simultaneously encoding domain knowledge and general knowledge, the model of this invention has excellent representation ability and good fitting and generalization ability in small-sample domains. The technical solution of this invention solves the problem of insufficient ability of existing technologies to recognize variant texts in small sample domains, and this technology has the ability to generalize to different domains. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart of a variant text recognition method according to one embodiment of the present invention;

[0022] Figure 2 This is a schematic diagram illustrating the process of converting raw text into a masked text vector sequence and a pinyin text vector sequence, as described in one embodiment of the present invention.

[0023] Figure 3 This is a schematic diagram of the network structure of a variant text recognition model according to one embodiment of the present invention;

[0024] Figure 4 This is a schematic diagram of the graph attention neural network construction process of the GAT layer in one embodiment of the present invention;

[0025] Figure 5 This is a structural diagram of a variant text recognition device according to one embodiment of the present invention;

[0026] Figure 6 This is another flowchart of a variant text recognition method according to one embodiment of the present invention;

[0027] Figure 7 This is a schematic diagram illustrating the interaction between small-sample domain knowledge, the GAT layer, and the RoBERTa layer in one embodiment of the present invention.

[0028] Figure 8 This is a schematic diagram illustrating feature fusion of the output results of the GAT layer and the RoBERTa layer in one embodiment of the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] On the one hand, such as Figure 1 As shown, an embodiment of the present invention provides a variant text recognition method, including:

[0031] Step S10: Identify suspected sensitive words in the original text, replace the suspected sensitive words with mask identifiers to obtain masked text, and convert each element in the masked text into a vector to obtain a masked text vector sequence;

[0032] Step S20: Replace each Chinese character in the original text with the corresponding pinyin to obtain pinyin text, and convert each element in the pinyin text into a vector to obtain a pinyin text vector sequence;

[0033] Step S30: Input the masked text vector sequence and the pinyin text vector sequence into the pre-trained variant text recognition model for text representation processing and domain knowledge enhancement processing to obtain the predicted text;

[0034] Step S40: Compare the original text and the predicted text, and identify the elements in the original text that are different from the predicted text as variant texts.

[0035] The following is an illustration using a specific example, such as... Figure 2 As shown, for example, if the sensitive word is designated as "mood", and the original text is "Today the weather is nice and haha", through step S10, the text "new weather" is identified and detected as a suspected sensitive word in the original text. "New weather" is then replaced with a mask identifier to obtain the masked text "Today MASKMASK is nice and haha". Each character, word, corresponding pinyin, and punctuation mark in the masked text is considered an element of the masked text. Figure 2In the RoBERTa vocabulary, the vectors corresponding to MASK, characters, and the pinyin of characters are stored in a single row record. In some specific embodiments, the index values of the rows where MASK, characters, and the pinyin of characters are located are used as the corresponding vectors for MASK, characters, and the pinyin of characters. When replacing each element in the masked text "Today MASK MASK is very good haha" with the index of the element in the RoBERTa vocabulary, we get "79119211031032523196215061506". Here, "791" is the index corresponding to "今", "1921" is the index corresponding to "天", and "103" is the index corresponding to "MASK". In some embodiments, we can further obtain the vector corresponding to the element from the RoBERTa vocabulary through the corresponding index or directly use the index to replace the vector corresponding to the element, resulting in a masked text vector sequence. The masked text vector sequence is composed of the vectors corresponding to each element in the masked text arranged in the order of the element in the masked text. Through step S20, each Chinese character in the original text "Today is newly sunny and very good haha" is replaced with the corresponding pinyin to obtain the pinyin text "jin tian xin qing hen hao ha ha". Each pinyin corresponding to a character in the pinyin text is an element of the pinyin text. For example, "jin" is an element, and "tian" is also an element. Then, we query the RoBERTa vocabulary again and replace the elements in the pinyin text with the obtained indexes in the order of the elements in the pinyin text to get "2131621252212482146521329214252118621186". In some embodiments, we further obtain the vector corresponding to the element from the RoBERTa vocabulary through the corresponding index or directly use the index to replace the vector corresponding to the element, resulting in a pinyin text vector sequence. Through step S30, the obtained masked text vector sequence and pinyin text vector sequence are input into a pre-trained variant text recognition model to obtain a predicted text. The pinyin text vector sequence records the original information of the original text, and the masked text vector sequence contains the information of the suspected sensitive word. The variant text recognition model analyzes by combining the original information and the information of the suspected sensitive word to determine the predicted word of the word at the position masked by MASK, thus obtaining the predicted text. For example, the obtained predicted text is "Today I'm in a good mood haha". In step S40, by comparing the different words in the original text and the predicted text, for example, in this example, "新晴" in the original text is different from "心情" in the predicted text, so it is considered that "新晴" in the original text is a variant text. Figure 2 The RoBERTa vocabulary used herein is not a limitation on the embodiments of the present invention. There are various methods in the art to convert Chinese characters and pinyin into indexes or vectors. The RoBERTa vocabulary is just one of the conversion tools based on the text vector vocabulary.

[0036] Among them, the text vector vocabulary is a table that is pre-trained based on a general corpus and maps characters in the general corpus to their corresponding indices and / or vectors; the text vector vocabulary can also map the pinyin corresponding to characters in the general corpus to their corresponding indices and / or vectors.

[0037] The embodiments of this invention have the following technical effects: The original text is converted into Pinyin text and further into a Pinyin text vector sequence. The Pinyin text performs a certain degree of generalization and fuzzification on the original text, and the Pinyin text vector sequence carries information from the original text. Suspected sensitive words in the original text are identified and replaced with mask identifiers. A mask text vector sequence is obtained through quantization. The Pinyin text vector sequence and the mask text vector sequence are combined and processed by a variant text recognition model to predict the word at the position corresponding to the mask identifier, thus obtaining the predicted text. The variant text is determined by comparing the predicted text with the original text. By utilizing the generalization and fuzziness characteristics of Pinyin text, the range of words that can be covered by prediction is increased, improving the accuracy of variant text prediction.

[0038] Further, the step of identifying suspected sensitive words in the original text, replacing the suspected sensitive words with mask identifiers to obtain masked text, and converting each element in the masked text into a vector to obtain a masked text vector sequence includes:

[0039] Step S11: Search for sensitive words recorded in a preset set of sensitive words in the original text, and use the found sensitive words as the suspected sensitive words;

[0040] Step S12: For each Chinese character in the original text, the Chinese character is used as a similar character to search for the corresponding original character in a preset similar word list. The pinyin of the original character is used to replace the Chinese character in the original text to obtain similar pinyin text from the original text. In addition, the pinyin of the similar pinyin text is searched for that is the same as the pinyin of the sensitive words recorded in the sensitive word set, and the Chinese character in the original text corresponding to the found pinyin is used as the suspected sensitive word.

[0041] Step S13: Replace the suspected sensitive words in the original text with mask identifiers to obtain the masked text;

[0042] Step S14: Convert each element in the masked text into a corresponding vector according to a preset text vector vocabulary to obtain the masked text vector sequence;

[0043] The similar word list is used to record the correspondence between original characters and similar characters; each original character corresponds to at least one similar character.

[0044] In some embodiments, a sensitive word set is pre-established, which includes at least one sensitive word. For specific projects, sensitive words of interest are added to the sensitive word set. For example, in some online chats or articles, variant words are used to replace sensitive words through homophones, similar characters, or similar sounds. The technical solution of this invention aims to identify the replaced sensitive words. Adding sensitive words to the sensitive word set, through the embodiments of this invention, can determine whether there are cases where variant words are used to replace sensitive words of interest in the original text, and identify the variant words. A similar word list is pre-established, which includes homophones, homonyms, near-homophones, near-homophones, words with the same radical and number of strokes, synonyms, similar words, and / or popular online alternatives to sensitive words from the sensitive word set. Suspected sensitive words are identified through two methods, S11 and S12. The order of execution of steps S11 and S12 is not limited; step S12 can be executed first, followed by step S11. In step S11, sensitive words recorded in the sensitive word set are searched in the original text, and the found sensitive words are taken as suspected sensitive words. Even if a sensitive word appears directly in the original text, it needs to be replaced with a mask identifier, and the variant text recognition model of this embodiment is used for prediction to determine whether it is still a sensitive word after prediction. In order to further expand the comprehensiveness of variant text recognition, using pinyin can cover a wider range of vocabulary. In step S12, words recorded in the similar word list are searched in the original text, and the found words are converted into pinyin to obtain similar pinyin text. In the similar pinyin text, pinyin that is the same as the pinyin corresponding to the sensitive words in the sensitive word list is searched, and the elements in the original text corresponding to the found pinyin are also taken as suspected sensitive words. The suspected sensitive words found in steps S11 and S12 may overlap or not overlap. In step S13, the suspected sensitive words found in steps S11 and S12 are replaced with mask identifiers to obtain masked text. In step S14, each element in the masked text is converted into a vector according to a preset text vector word list to obtain the masked text vector sequence.

[0045] The embodiments of the present invention have the following technical effects: First, the original text is directly checked for the presence of sensitive words, and then a similar word list is used to expand the coverage of suspected sensitive words, thereby maximizing the discovery of variant texts in the original text.

[0046] Furthermore, such as Figure 3 As shown, the variant text recognition model includes: a RoBERTa layer, a GAT layer, and an output layer;

[0047] The step of inputting the masked text vector sequence and the pinyin text vector sequence into a pre-trained variant text recognition model for text representation processing and domain knowledge enhancement processing to obtain predicted text includes:

[0048] Step S31: Input the masked text vector sequence into the trained RoBERTa layer for text representation processing to obtain a text semantic representation vector sequence;

[0049] Step S32: Perform initial feature fusion processing on the text semantic representation vector sequence and the pinyin text vector sequence to obtain the first fused vector sequence;

[0050] Step S33: Input the first fused vector sequence into the trained GAT layer for domain knowledge enhancement processing to obtain an attention feature vector sequence;

[0051] Step S34: Perform secondary feature fusion processing on the attention feature vector sequence and the text semantic representation vector sequence to obtain a second fused vector sequence;

[0052] Step S35: Input the second fused vector sequence into the trained output layer for probability prediction processing to obtain the predicted text.

[0053] In some embodiments, the output layer includes, but is not limited to, a fully connected neural network layer; the activation function used in the output layer includes, but is not limited to, Softmax().

[0054] Primary feature fusion processing is one of many feature fusion processing methods, specifically residual fusion; secondary feature fusion processing is one of many feature fusion processing methods, specifically residual fusion.

[0055] Among them, such as Figure 4 As shown, the GAT layer is a graph attention neural network layer. For each pinyin in the text vector vocabulary, the vector corresponding to the pinyin is used as the root node, and similar characters of the pinyin are used as the neighbor nodes of the root node to construct the GAT layer. The text vector vocabulary is pre-generated based on a general corpus, establishing a mapping between characters in the general corpus and their corresponding indices and / or vectors. The text vector vocabulary can also map the pinyin corresponding to characters in the general corpus to their corresponding indices and / or vectors. The text vector vocabulary includes: the mapping relationship between characters in the general corpus and their corresponding indices and / or vectors, the mapping relationship between pinyin corresponding to characters in the general corpus and their corresponding indices and / or vectors, and the mapping relationship between specific strings and their corresponding indices and / or vectors. The specific strings include mask identifiers. For each pinyin in the text vector vocabulary, based on small sample neighborhood knowledge, similar characters, homophones, homonyms, near-homophones, and / or near-homophones are pre-defined as similar characters of the pinyin.

[0056] The embodiments of this invention have the following technical effects: A variant text recognition model combining RoBERTa and GAT layers is established. The attention mechanism of the graph attention neural network in the GAT layer improves the prediction effect of words whose mask identifier positions are determined by RoBERTa semantic analysis. Furthermore, during the combination of the RoBERTa and GAT layers, a feature fusion mechanism is used to fuse the text semantic representation vector sequence output by the RoBERTa layer with the input pinyin text vector sequence of the GAT layer. This allows the input information of the GAT layer to simultaneously contain information predicted by the semantic analysis of the RoBERTa layer and information from the original text, enabling the GAT layer to use more comprehensive information for further prediction. The prediction results of the RoBERTa layer and the GAT layer are further fused, strengthening the accuracy of semantic prediction results through the attention mechanism. The combination of the various layers of the model in this embodiment significantly improves the accuracy of variant text recognition. By encoding domain knowledge in GAT and using the feature fusion mechanism, the distribution difference between pre-training samples and domain samples is reduced. Simultaneously encoding domain knowledge and general knowledge, the variant text recognition model of this embodiment has excellent representation capabilities and good fitting and generalization capabilities in small sample domains. This technology solves the problem of insufficient ability of existing technologies to recognize variant texts in small sample domains, and has the ability to generalize to different domains.

[0057] Further, the initial feature fusion processing of the text semantic representation vector sequence and the pinyin text vector sequence to obtain the first fused vector sequence includes:

[0058] Step S321: The vectors at each sensitive index position in the text semantic representation vector sequence are accumulated and added to the vectors at the same sensitive index position in the pinyin text vector sequence to obtain the first fused vector sequence; wherein, the sensitive index position is the index position of the suspected sensitive word;

[0059] The step of performing secondary feature fusion processing on the attention feature vector sequence and the text semantic representation vector sequence to obtain the second fused vector sequence includes:

[0060] Step S341: Replace the vector at the same sensitive index position in the text semantic representation vector sequence with the vector at each sensitive index position in the attention feature vector sequence to obtain the second fusion vector sequence.

[0061] In some instances, such as Figure 3 As shown, for example, if the original text is "Today is sunny and nice haha", the sensitive word is "mood", and the suspected sensitive word is "sunny", then the masked text is "Today is MASKMASK and nice haha", and the corresponding masked text vector sequence is E. 今 E天 E [mask] E [mask] E 很 E 好 E 哈 E 哈 , the text semantic representation vector sequence obtained after prediction by the trained RoBERTa layer is R 今 R 天 R [mask1] R [mask2] R 很 R 好 R 哈 R 哈 , after prediction by the trained RoBERTa layer, the two Es in the masked text vector sequence [mask] will have the same or different semantics, corresponding to R [mask1] and R [mask2] . The pinyin text corresponding to the original text "今天新晴很好哈哈" is "jintianxinqinghenhao haha", and the pinyin text vector sequence corresponding to the pinyin text is E jin E tian E xin E qing E hen E hao E ha E ha , in this example, the sensitive index positions in the text semantic representation vector sequence correspond to R [mask1] and R [mask2] , and the sensitive index positions in the pinyin text vector sequence correspond to E xin and E qing , adding R [mask1] to E xin and normalizing to obtain P xin , adding R [mask2] to E qing and normalizing to obtain P qing The first fusion vector sequence E jin E tian P xin P qing E hen E hao E ha E ha , the first fusion vector sequence E jin E tian P xin P qing E hen E hao E ha E ha After analysis by the trained GAT layer, the attention feature vector sequence G jinG tian G xin G qing G hen G hao G ha G ha Using the vector G in the attention feature vector sequence at the said sensitive index position xin and G qing Replace the R corresponding to the sensitive index position in the text semantic representation vector sequence [mask1] and R [mask2] The second fusion vector sequence R is obtained. 今 R 天 G xin G qing R 很 R 好 R 哈 R 哈 .

[0062] Furthermore, the method includes:

[0063] Independently train the untrained RoBERTa layer to obtain the trained RoBERTa layer;

[0064] The trained GAT layer and the trained output layer are obtained by training the GAT layer and the output layer based on the trained RoBERTa layer.

[0065] In some embodiments, through two-stage pre-training, few-sample domain representations are learned on top of general representations. Therefore, without the need for expert-led heuristic rules, it possesses excellent learning and portability capabilities. By using GAT to encode domain knowledge and a feature fusion mechanism, the distributional differences between pre-training samples and domain samples are reduced. Simultaneously encoding both domain and general knowledge, the model of this invention exhibits excellent representational capabilities and demonstrates good fitting and generalization abilities in few-sample domains. This invention solves the problem of insufficient ability of existing technologies to recognize variant texts in few-sample domains, and it also possesses generalization capabilities across different domains.

[0066] Furthermore, the independently trained RoBERTa layer to be trained is obtained by obtaining the trained RoBERTa layer, including:

[0067] For each general training text in the general corpus, a mask identifier is used to randomly mask the general training text to obtain a masked general training text. The masked general training text is then converted into a masked general training text vector sequence by querying a text vector vocabulary. The text vector vocabulary is generated based on the general corpus and includes each Chinese character in the general corpus and its corresponding vector, the pinyin corresponding to each Chinese character in the general corpus and its corresponding vector, and a special identifier and its corresponding vector. The special identifier includes the mask identifier.

[0068] Each mask's universal training text vector sequence is input into the RoBERTa layer to be trained, and training is performed by minimizing the loss function of the RoBERTa layer. The precision of the RoBERTa layer is iterated to the first target precision in order to determine the parameters of the RoBERTa layer and obtain the trained RoBERTa layer and the text semantic representation vector corresponding to each Chinese character and special identifier in the text vector vocabulary recorded in the trained RoBERTa layer.

[0069] The loss function in the RoBERTa layer n is the number of mask labels after random masking of the general training text; T i f is the i-th character in the general training text that is replaced with a mask identifier; M (T i After randomly masking the general training text, T i In the context, θ is the estimated parameters of the RoBERTa layer.

[0070] In some embodiments, during training, a RoBERTa layer is first trained using a general corpus with a random mask to obtain the trained RoBERTa layer. In some specific embodiments, the random masking can be performed as follows: for each general training text in the general corpus, a first percentage of elements are randomly selected as replacement elements; a second percentage of elements are randomly selected and replaced with mask identifiers; a third percentage of the remaining replacement elements are randomly replaced with other random elements; and the other elements remain unchanged. Preferably, the first percentage is 15%; the second percentage is 80%; and the third percentage is 50%. In some embodiments, the first target precision is 0.75.

[0071] Furthermore, the GAT layer to be trained is pre-constructed based on the text vector vocabulary and the similarity vocabulary;

[0072] The method for constructing the GAT layer to be trained includes:

[0073] For each pinyin in the text vector vocabulary, the root node is defined by the pinyin in the text vector vocabulary, and the neighbor nodes corresponding to the original characters in the similarity vocabulary that have similar pinyin to the pinyin in the text vector vocabulary are used to construct the GAT layer to be trained. The feature vector of each root node is randomly initialized. The feature vector of each neighbor node is the text semantic representation vector corresponding to the same Chinese character as the neighbor node, which is recorded in the trained RoBERTa layer.

[0074] The similar word list is used to record the correspondence between original characters and similar characters; each original character corresponds to at least one similar character.

[0075] Further, the process of training the GAT layer and the output layer to be trained based on the trained RoBERTa layer to obtain the trained GAT layer and the trained output layer includes:

[0076] For each domain training text in the domain training corpus, the characters marked as sensitive words in the domain training text are replaced with mask identifiers to obtain the masked domain training text corresponding to the domain training text. The masked domain training text is then vectorized to obtain the masked domain training text vector sequence corresponding to the domain training text. The domain training text is then converted to Pinyin to obtain the domain training Pinyin text corresponding to the domain training text. The domain training Pinyin text is then vectorized to obtain the domain training Pinyin text vector sequence corresponding to the domain training text.

[0077] The masked domain training text vector sequence corresponding to the domain training text is input into the trained RoBERTa layer to obtain the domain training text semantic representation vector sequence corresponding to the domain training text.

[0078] The semantic representation vector sequence of the domain training text corresponding to the domain training text and the domain training pinyin text vector sequence corresponding to the domain training text are subjected to initial feature fusion processing to obtain the first fused training vector sequence corresponding to the domain training text.

[0079] The first fusion training vector sequence corresponding to the domain training text is input into the GAT layer to be trained for domain knowledge enhancement processing to obtain the training attention feature vector sequence corresponding to the domain training text.

[0080] The second fused training vector sequence corresponding to the domain training text is obtained by performing a secondary feature fusion process on the training attention feature vector sequence corresponding to the domain training text and the semantic representation vector sequence corresponding to the domain training text.

[0081] The second fused training vector sequence corresponding to the domain training text is input into the output layer to be trained, and the loss function of the output layer is minimized. The accuracy of the output layer is iterated to the second target accuracy in order to determine the parameters of the output layer and obtain the trained output layer and the trained GAT layer.

[0082] Among them, the loss function in the output layer In this context, m represents the mask corresponding to the domain training text, and the number of mask identifiers in the domain training text. G is the sum of the feature vectors of all neighboring nodes of the root node corresponding to the i-th pinyin in the domain training text, where C is the number of all neighboring nodes of the root node corresponding to the i-th pinyin in the GAT layer. i T is the i-th vector in the sequence of training attention feature vectors corresponding to the domain training text; i The i-th vector in the semantic representation vector sequence of the domain training text corresponding to the domain training text; the domain training corpus is obtained by annotating the sensitive words in each text of the domain corpus to which the sensitive words belong.

[0083] In some embodiments, the second target precision is 0.75; the domain training corpus is obtained by labeling sensitive words in the text of the domain to which the sensitive words belong, according to preset rules or by manual annotation. The masked domain training text is vectorized to obtain the corresponding masked domain training text vector sequence. Specifically, the index and / or vector corresponding to each element in the masked domain training text can be queried according to a text vector vocabulary to vectorize the masked domain training text into a masked domain training text vector sequence. The domain training pinyin text is vectorized to obtain the corresponding domain training pinyin text vector sequence. Specifically, the index and / or vector corresponding to each element in the domain training pinyin text can be queried according to a text vector vocabulary to vectorize the domain training pinyin text into a domain training pinyin text vector sequence. The initial feature fusion processing is one of many feature fusion processing methods, specifically residual fusion; the secondary feature fusion processing is one of many feature fusion processing methods, specifically residual fusion; in some instances, such as... Figure 3 As shown, for example, if the domain training text is "Today is sunny and nice haha", the sensitive word is "mood", and the suspected sensitive word is "sunny", then the masked domain training text is "Today is MASKMASK and nice haha", and the corresponding masked domain training text vector sequence is E. 今 E 天 E [mask] E [mask] E 很 E好 E 哈 E 哈 , the sequence of semantic representation vectors of the domain training text obtained after prediction by the trained RoBERTa layer is R 今 R 天 R [mask1] R [mask2] R 很 R 好 R 哈 R 哈 , after prediction by the trained RoBERTa layer, the two Es in the masked domain training text vector sequence [mask] will have the same or different semantics, corresponding to R [mask1] and R [mask2] . The domain training pinyin text corresponding to the domain training text "Today's weather is sunny and great, haha" is "jintianxinqinghenhao haha", and the sequence of vectors of the domain training pinyin text corresponding to the domain training pinyin text is E jin E tian E xin E qing E hen E hao E ha E ha , in this example, the sensitive index positions in the sequence of semantic representation vectors of the domain training text correspond to R [mask1] and R [mask2] , and the sensitive index positions in the sequence of vectors of the domain training pinyin text correspond to E xin and E qing , adding R [mask1] to E xin and normalizing to obtain P xin , adding R [mask2] to E qing and normalizing to obtain P qing The first fusion training vector sequence E jin E tian P xin P qing E hen E hao E ha E ha , the first fusion training vector sequence E jin E tia n P xin P qing E hen E hao E ha E ha After being processed by the untrained GAT layer for domain knowledge enhancement, the training attention feature vector sequence G corresponding to the domain training text is obtainedjin G tian G xin G qing G hen G hao G ha G ha Using the vector G at the sensitive index position in the training attention feature vector sequence xin and G qing Replace the sensitive index positions corresponding to the R values ​​in the sequence of semantic representation vectors of the training text in the domain. [mask1] and R [mask2] The second fused training vector sequence R is obtained. 今 R 天 G xin G qing R 很 R 好 R 哈 R 哈 The second fused training vector sequence corresponding to the domain training text is input into the output layer to be trained, and the loss function of the output layer is minimized. The accuracy of the output layer is iterated to the second target accuracy in order to determine the parameters of the output layer and obtain the trained output layer and the trained GAT layer.

[0084] The embodiments of this invention have the following technical effects: The original text is converted into Pinyin text and further into a Pinyin text vector sequence. The Pinyin text generalizes and blurs the original text to a certain extent, and the Pinyin text vector sequence carries information from the original text. Suspected sensitive words in the original text are identified and replaced with mask identifiers. A mask text vector sequence is obtained through quantization. The Pinyin text vector sequence and the mask text vector sequence are combined and processed by a variant text recognition model to predict the word at the corresponding position of the mask identifier, resulting in predicted text. The variant text is determined by comparing the predicted text with the original text. The generalization and blurring characteristics of Pinyin text are utilized to increase the vocabulary range that can be covered by prediction and improve the accuracy of variant text prediction. Furthermore, a variant text recognition model combining RoBERTa layers and GAT layers is established. The attention mechanism of the graph attention neural network in the GAT layer is used to improve the prediction effect of RoBERTa semantic analysis in determining the word at the mask identifier position. Furthermore, in the process of combining the RoBERTa layer and the GAT layer, a feature fusion mechanism is used to fuse the text semantic representation vector sequence output by the RoBERTa layer with the input pinyin text vector sequence of the GAT layer. This allows the input information of the GAT layer to simultaneously contain information from the semantic analysis and prediction of the RoBERTa layer, as well as information from the original text. This enables the GAT layer to make further predictions using more comprehensive information. The prediction results of the RoBERTa layer and the GAT layer are further fused, strengthening the accuracy of semantic prediction results through the attention mechanism. The various models in this embodiment of the invention, combined as a whole, significantly improve the accuracy of variant text recognition. Furthermore, through two-stage pre-training, small-sample domain representations are learned on the basis of general representations. Therefore, without the need for expert-led heuristic rules, it possesses excellent learning ability and portability. By encoding domain knowledge through GAT and using the feature fusion mechanism, the distribution difference between pre-training samples and domain samples is reduced. Simultaneously encoding domain knowledge and general knowledge, the model of this invention has excellent representation ability and good fitting and generalization ability in small-sample domains. The technical solution of this invention solves the problem of insufficient ability of existing technologies to recognize variant texts in small sample domains, and this technology has the ability to generalize to different domains.

[0085] On the other hand, such as Figure 5 As shown, an embodiment of the present invention provides a variant text recognition device, comprising:

[0086] The masking unit 100 is used to identify suspected sensitive words in the original text, replace the suspected sensitive words with mask identifiers to obtain masked text, and convert each element in the masked text into a vector to obtain a masked text vector sequence.

[0087] The Pinyin unit 200 is used to replace each Chinese character in the original text with the corresponding Pinyin to obtain Pinyin text, and to convert each element of the Pinyin text into a vector to obtain a Pinyin text vector sequence;

[0088] Prediction unit 300 is used to input the masked text vector sequence and the pinyin text vector sequence into a pre-trained variant text recognition model for text representation processing and domain knowledge enhancement processing to obtain predicted text;

[0089] The comparison unit 400 is used to compare the original text and the predicted text, and to identify elements in the original text that are different from the predicted text as variant text.

[0090] Further, the masking unit 100 includes:

[0091] The sensitive word precise matching module is used to find sensitive words recorded in a preset sensitive word set in the original text, and to use the found sensitive words as the suspected sensitive words;

[0092] The sensitive word similarity matching module is used to, for each Chinese character in the original text, search for the original character corresponding to the similar character in a preset similar word table, replace the Chinese character in the original text with the pinyin of the original character to obtain similar pinyin text from the original text; and, in the similar pinyin text, search for pinyin that is the same as the pinyin of the sensitive words recorded in the sensitive word set, and use the Chinese character in the original text corresponding to the found pinyin as the suspected sensitive word;

[0093] The prediction mask module is used to replace the suspected sensitive words in the original text with mask identifiers to obtain the masked text;

[0094] The masked text vector acquisition module is used to convert each element in the masked text into a corresponding vector according to a preset text vector vocabulary to obtain the masked text vector sequence;

[0095] The similar word list is used to record the correspondence between original characters and similar characters; each original character corresponds to at least one similar character.

[0096] Furthermore, the variant text recognition model includes: a RoBERTa layer, a GAT layer, and an output layer;

[0097] The prediction unit 300 includes:

[0098] The text semantic representation prediction module is used to input the masked text vector sequence into the trained RoBERTa layer for text representation processing to obtain a text semantic representation vector sequence.

[0099] The primary fusion prediction module is used to perform initial feature fusion processing on the text semantic representation vector sequence and the pinyin text vector sequence to obtain the first fusion vector sequence.

[0100] The attention feature prediction module is used to input the first fused vector sequence into the trained GAT layer for domain knowledge enhancement processing to obtain the attention feature vector sequence.

[0101] The secondary fusion prediction module is used to perform secondary feature fusion processing on the attention feature vector sequence and the text semantic representation vector sequence to obtain a second fusion vector sequence.

[0102] The probability prediction module is used to input the second fused vector sequence into the trained output layer for probability prediction processing to obtain the predicted text.

[0103] Furthermore, the primary fusion prediction module is specifically used for:

[0104] The vectors at each sensitive index position in the text semantic representation vector sequence are accumulated and added to the vectors at the same sensitive index position in the pinyin text vector sequence to obtain the first fused vector sequence; wherein, the sensitive index position is the index position of the suspected sensitive word;

[0105] The secondary fusion prediction module is specifically used for:

[0106] The vector at each sensitive index position in the attention feature vector sequence is used to replace the vector at the same sensitive index position in the text semantic representation vector sequence to obtain the second fusion vector sequence.

[0107] Further, the device includes:

[0108] The first training unit is used to independently train the RoBERTa layer to be trained and obtain the trained RoBERTa layer.

[0109] The second-stage training unit is used to train the GAT layer and the output layer to be trained based on the trained RoBERTa layer, thus obtaining the trained GAT layer and the trained output layer.

[0110] Furthermore, the first-stage training unit includes:

[0111] A general training text vector sequence acquisition module is used to randomly mask each general training text in the general corpus using a mask identifier to obtain a masked general training text, and then convert the masked general training text into a masked general training text vector sequence by querying a text vector vocabulary; wherein, the text vector vocabulary is generated based on the general corpus and includes each Chinese character in the general corpus and its corresponding vector, the pinyin corresponding to each Chinese character in the general corpus and its corresponding vector, and a special identifier and its corresponding vector; the special identifier includes the mask identifier;

[0112] The first-stage training module is used to input the general training text vector sequence for each mask into the RoBERTa layer to be trained, and to train it by minimizing the loss function of the RoBERTa layer. The precision of the RoBERTa layer is iterated to the first target precision in order to determine the parameters of the RoBERTa layer and obtain the trained RoBERTa layer and the text semantic representation vector corresponding to each Chinese character and special identifier in the text vector vocabulary recorded in the trained RoBERTa layer.

[0113] The loss function in the RoBERTa layer In this context, n represents the number of mask identifiers after random masking of the general training text; T i f is the i-th character in the general training text that is replaced with a mask identifier; M (T i After randomly masking the general training text, T i In the context, θ is the estimated parameters of the RoBERTa layer.

[0114] Furthermore, the device also includes: a GAT layer construction unit;

[0115] The GAT layer construction unit is used to pre-construct the book search GAT layer based on the text vector vocabulary and the similar vocabulary;

[0116] The GAT layer construction unit is specifically configured as follows: for each pinyin in the text vector vocabulary, the root node is the pinyin in the text vector vocabulary, and the neighbor nodes corresponding to the original characters in the similarity vocabulary that have similar pinyin to the pinyin in the text vector vocabulary are the neighbor nodes corresponding to the root node to construct the GAT layer to be trained; wherein, the feature vector of each root node is randomly initialized; the feature vector of each neighbor node is the text semantic representation vector corresponding to the same Chinese character corresponding to the neighbor node recorded in the trained RoBERTa layer;

[0117] The similar word list is used to record the correspondence between original characters and similar characters; each original character corresponds to at least one similar character.

[0118] Furthermore, the second-stage training unit includes:

[0119] The training mask and pinyin vector sequence acquisition module is used to, for each domain training text in the domain training corpus, replace the characters marked as sensitive words in the domain training text with mask identifiers to obtain the masked domain training text corresponding to the domain training text; vectorize the masked domain training text to obtain the masked domain training text vector sequence corresponding to the domain training text; convert the domain training text to pinyin to obtain the domain training pinyin text corresponding to the domain training text; and vectorize the domain training pinyin text to obtain the domain training pinyin text vector sequence corresponding to the domain training text.

[0120] The domain training text semantic acquisition module is used to input the masked domain training text vector sequence corresponding to the domain training text into the trained RoBERTa layer to obtain the domain training text semantic representation vector sequence corresponding to the domain training text.

[0121] The first fusion training module is used to perform initial feature fusion processing on the semantic representation vector sequence of the domain training text corresponding to the domain training text and the domain training pinyin text vector sequence corresponding to the domain training text to obtain the first fusion training vector sequence corresponding to the domain training text.

[0122] The attention training module is used to input the first fusion training vector sequence corresponding to the domain training text into the GAT layer to be trained for domain knowledge enhancement processing, so as to obtain the training attention feature vector sequence corresponding to the domain training text.

[0123] The second fusion training module is used to perform secondary feature fusion processing on the training attention feature vector sequence corresponding to the domain training text and the semantic representation vector sequence of the domain training text corresponding to the domain training text to obtain the second fusion training vector sequence corresponding to the domain training text.

[0124] The output layer training module is used to input the second fused training vector sequence corresponding to the domain training text into the output layer to be trained for training, and to minimize the loss function of the output layer. The accuracy of the output layer is iterated to the second target accuracy in order to determine the parameters of the output layer and obtain the trained output layer and the trained GAT layer.

[0125] Among them, the loss function in the output layer In this context, m represents the mask corresponding to the domain training text, and the number of mask identifiers in the domain training text. G is the sum of the feature vectors of all neighboring nodes of the root node corresponding to the i-th pinyin in the domain training text, where C is the number of all neighboring nodes of the root node corresponding to the i-th pinyin in the GAT layer. i T is the i-th vector in the sequence of training attention feature vectors corresponding to the domain training text; i The i-th vector in the semantic representation vector sequence of the domain training text corresponding to the domain training text; the domain training corpus is obtained by annotating the sensitive words in each text of the domain corpus to which the sensitive words belong.

[0126] The embodiments of the present invention are product-class embodiments that correspond one-to-one with the aforementioned variant text recognition method embodiments. The embodiments of the present invention can be understood based on the aforementioned variant text recognition method embodiments, and will not be repeated here.

[0127] The embodiments of this invention have the following technical effects: The original text is converted into Pinyin text and further into a Pinyin text vector sequence. The Pinyin text generalizes and blurs the original text to a certain extent, and the Pinyin text vector sequence carries information from the original text. Suspected sensitive words in the original text are identified and replaced with mask identifiers. A mask text vector sequence is obtained through quantization. The Pinyin text vector sequence and the mask text vector sequence are combined and processed by a variant text recognition model to predict the word at the corresponding position of the mask identifier, resulting in predicted text. The variant text is determined by comparing the predicted text with the original text. The generalization and blurring characteristics of Pinyin text are utilized to increase the vocabulary range that can be covered by prediction and improve the accuracy of variant text prediction. Furthermore, a variant text recognition model combining RoBERTa layers and GAT layers is established. The attention mechanism of the graph attention neural network in the GAT layer is used to improve the prediction effect of RoBERTa semantic analysis in determining the word at the mask identifier position. Furthermore, in the process of combining the RoBERTa layer and the GAT layer, a feature fusion mechanism is used to fuse the text semantic representation vector sequence output by the RoBERTa layer with the input pinyin text vector sequence of the GAT layer. This allows the input information of the GAT layer to simultaneously contain information from the semantic analysis and prediction of the RoBERTa layer, as well as information from the original text. This enables the GAT layer to make further predictions using more comprehensive information. The prediction results of the RoBERTa layer and the GAT layer are further fused, strengthening the accuracy of semantic prediction results through the attention mechanism. The various models in this embodiment of the invention, combined as a whole, significantly improve the accuracy of variant text recognition. Furthermore, through two-stage pre-training, small-sample domain representations are learned on the basis of general representations. Therefore, without the need for expert-led heuristic rules, it possesses excellent learning ability and portability. By encoding domain knowledge through GAT and using the feature fusion mechanism, the distribution difference between pre-training samples and domain samples is reduced. Simultaneously encoding domain knowledge and general knowledge, the model of this invention has excellent representation ability and good fitting and generalization ability in small-sample domains. The technical solution of this invention solves the problem of insufficient ability of existing technologies to recognize variant texts in small sample domains, and this technology has the ability to generalize to different domains.

[0128] On the other hand, embodiments of the present invention provide a readable storage medium storing program code for implementing any of the methods described above.

[0129] The technical solutions of the present invention will be described in detail below with reference to specific application examples. For technical details not described in the implementation process, please refer to the relevant descriptions above.

[0130] Explanation of terms used in the embodiments of this invention:

[0131] General Corpus: Text containing domain knowledge from multiple domains; Variant Text: Alternative text used to hide real entities or events; Confusion Dictionary: A dictionary storing real text and its variant texts; Pre-trained Language Model: A general language representation model that performs self-supervised task learning using a large-scale corpus before learning a specific natural language task; Small-Shot Domain: A domain where learnable knowledge is difficult to obtain and the sample size is small; Domain Knowledge: Relevant knowledge and concepts with characteristics of small-shot domains; Detection Text: Suspected variant text constructed based on domain knowledge and task objectives; Language Model: A machine learning method that learns text representations by modeling the probability distribution of sentences; Graph Neural Network: A machine learning method that uses neural networks to learn graph structure representations; Domain-Specific Corpus: Sample corpus containing both small-shot domains and similar sample domains; RoBERTa: An algorithmic structure for learning language models, whose learning task is to mask text tags and then restore the tags; GAT: A graph neural network built based on an attention mechanism.

[0132] The following is based on Figure 6 The embodiments of the present invention will be explained as follows:

[0133] The embodiments of the present invention have a two-stage pre-training process: the first stage is RoBERTa layer training; the second stage is GAT layer training, which includes a domain knowledge enhancement method.

[0134] All corpus samples are in text format. The training corpus is divided into two parts: a general corpus and a domain-specific corpus annotated by rules or manually; the test corpus is a small sample of domain-specific corpus that needs to be predicted.

[0135] Step S601, Corpus Preprocessing: A RoBERTa vocabulary is pre-built based on the general corpus; characters in the general corpus are converted to pinyin, and the pinyin is added to the RoBERTa vocabulary; the pinyin corresponding to Chinese characters in the RoBERTa vocabulary is added to the RoBERTa vocabulary; the RoBERTa vocabulary includes characters and their corresponding indices and / or vectors; the RoBERTa vocabulary also includes the pinyin of the characters and their corresponding indices and / or vectors.

[0136] Step S602, Data preprocessing: Replace suspected sensitive words in the original text in prediction mode or the text to be detected (i.e., suspected sensitive words) in the corpus in training mode with special character MASK markers; replace the original text in prediction mode or the corpus in training mode with pinyin;

[0137] Step S603, obtain character and pinyin features: convert the original text or the corpus in the training mode that has been replaced with MASK tags (i.e. mask identifiers) into corresponding indices according to the RoBERTa vocabulary to form the input of the variant text recognition model; convert the pinyin of the original text or the corpus in the training mode into corresponding indices according to the RoBERTa vocabulary to form the input of the variant text recognition model;

[0138] Step S604: Determine whether the current process is training or prediction; if it is training, proceed to step S605; if it is prediction, proceed to step S609: perform model inference in step S.

[0139] Step S605, Pre-training the language model RoBERTa: The first stage of pre-training uses the RoBERTa layer for text representation learning. The loss function of the RoBERTa layer is... Where n is the sentence length, T i For the predicted label, After the MASK strategy, T i In the context of θ, the estimated model parameters are used. The model's accuracy is iterated to above 0.75 using a general corpus selected from the training data (higher accuracy is better, but adjustments are needed based on factors such as data, resources, and time), before proceeding to the second stage of model pre-training.

[0140] Step S606, pre-build GAT layer graph attention neural network: Based on small sample domain knowledge, domain similar characters, homophones, homophones with different tones, near-homophones with the same tone, and near-homophones with different tones are considered as similar characters; using all the pinyin in the RoBERTa vocabulary as root nodes (i.e., pinyin nodes), and using the similar characters of the pinyin as neighbor nodes (i.e., character nodes), a similar speech graph is constructed. The similar speech graph is specifically constructed in the form of a GAT layer graph attention neural network. The pinyin node features are randomly initialized, and the character node features are the character vectors pre-trained by RoBERTa. The node features are encoded using GAT.

[0141] Step S607, GAT layer (Graph Attention Neural Network) training: The training task is to use the acquired GMASK features (equivalent to G... xin and G qing Predict the original characters of the mask, with the loss function being... Where m is the number of MASK tags. In the GAT graph, the pinyin encoding E i All neighbor characteristics, C is the number of neighbor nodes, G i H is the label predicted after GMASK. i T represents the labeled features after GAT encoding. iThe token features encoded by RoBERTa after the first-stage pre-training, and θ are the estimated model parameters. The model is trained using the corpora in the relevant field in the training corpus, and the model can be saved after the accuracy is iterated to more than 0.75; as Figure 7 The figure shows the interaction between few-shot domain knowledge, GAT, and RoBERTa.

[0142] Step S608, GMASK feature fusion: As Figure 8 shown, the features with structured information are obtained after GAT training and learning, and they are residually connected with the MASK features encoded by RoBERTa (equivalent to R [mask1] and R [mask2] ), replacing the features corresponding to the original MASK in RoBERTa (equivalent to using the vectors G xin and G qing at the sensitive index positions in the attention feature vector sequence to replace the R [mask1] and R [mask2] corresponding to the sensitive index positions in the text semantic representation vector sequence); The second-stage pre-training is based on the first-stage RoBERTa model pre-training and conducts domain knowledge enhanced language model training.

[0143] The trained variant text recognition model is obtained through steps S605 to S608.

[0144] Step S609: Model inference: Use the model saved in step S608 to predict the test text corpus. After the input text is processed by steps 602 and 603, the variant text recognition model is used for prediction. The characters in the output text that are inconsistent with the input are the recognized variant text. Xinqing is the variant text of mood; The variant text recognition exits.

[0145] The embodiments of the present invention have the following technical effects: Through two-stage pre-training, the present invention learns few-shot domain representations based on general representations. Therefore, without expert-led heuristic rules, it can have good learning ability and portability; By encoding domain knowledge through GAT and the GMASK feature fusion mechanism, the distribution difference between pre-training samples and domain samples is reduced, and both domain knowledge and general knowledge are encoded. Therefore, the model of the present invention has good representation ability and good fitting and generalization ability in few-shot domains. The present invention solves the problem of insufficient ability to recognize variant texts in the prior art in few-shot domains, and this technology has the generalization ability for different domains.

[0146] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process may be rearranged without departing from the scope of this disclosure. The appended method claims provide elements of various steps in an exemplary order and are not intended to limit the scope to the specific order or hierarchy described.

[0147] In the above detailed description, various features are combined together in a single embodiment to simplify this disclosure. This approach to disclosure should not be construed as reflecting an intention that embodiments of the claimed subject matter require more features than are explicitly stated in each claim. Rather, as reflected in the appended claims, the invention is presented with fewer features than all of the features of the single disclosed embodiment. Therefore, the appended claims are hereby explicitly incorporated into the detailed description, wherein each claim stands alone as a preferred embodiment of the invention.

[0148] The disclosed embodiments have been described above to enable any person skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments without departing from the spirit and scope of this disclosure. Therefore, this disclosure is not limited to the embodiments given herein, but is consistent with the broadest scope of the principles and novel features disclosed in this application.

[0149] The foregoing description includes examples of one or more embodiments. It is certainly impossible to describe all possible combinations of components or methods in order to describe the above embodiments, but those skilled in the art will recognize that further combinations and arrangements of the various embodiments are possible. Therefore, the embodiments described herein are intended to cover all such changes, modifications, and variations that fall within the scope of the appended claims. Furthermore, the term "comprising" as used in the specification or claims is interpreted in a manner similar to the term "including," just as "including:" is interpreted as a conjunction in the claims. Additionally, the use of any term "or" in the specification of the claims is intended to mean "non-exclusive or."

[0150] Those skilled in the art will also understand that the various illustrative logical blocks, units, and steps listed in the embodiments of the present invention can be implemented by electronic hardware, computer software, or a combination of both. To clearly demonstrate the interchangeability of hardware and software, the functions of the various illustrative components, units, and steps described above have been generally described. Whether such functionality is implemented through hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can implement the described functions using various methods for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of the present invention.

[0151] The various illustrative logic blocks or units described in the embodiments of this invention can be implemented or operate the described functions using a general-purpose processor, digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The general-purpose processor can be a microprocessor; alternatively, it can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented using a combination of computing devices, such as a digital signal processor and a microprocessor, multiple microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.

[0152] The steps of the methods or algorithms described in the embodiments of this invention can be directly embedded in hardware, a software module executed by a processor, or a combination of both. The software module can be stored in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from and write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and storage medium can be housed in an ASIC, which can be housed in a user terminal. Optionally, the processor and storage medium can also be housed in different components of the user terminal.

[0153] In one or more exemplary designs, the functions described in the embodiments of the present invention can be implemented in hardware, software, firmware, or any combination of these three. If implemented in software, these functions can be stored on a computer-readable medium or transmitted on a computer-readable medium in the form of one or more instructions or code. Computer-readable media include computer storage media and communication media that facilitate the transfer of computer programs from one place to another. Storage media can be any available media that can be accessed by a general-purpose or special-purpose computer. For example, such computer-readable media can include, but is not limited to, RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store program code in the form of instructions or data structures and other forms that can be read by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. Furthermore, any connection can be suitably defined as a computer-readable medium, for example, if the software is transmitted from a website, server or other remote resource via a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wirelessly, such as infrared, wireless and microwave, it is also included in the defined computer-readable medium. The disks and discs mentioned include compressed disks, laser discs, optical discs, DVDs, floppy disks, and Blu-ray discs. Disks typically copy data magnetically, while disks typically copy data optically using lasers. Combinations of the above can also be contained in computer-readable media.

[0154] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method of recognizing a variant text, characterized by, The method comprises the following steps: identifying a suspected sensitive word in an original text and replacing the suspected sensitive word with a mask identifier to obtain a mask text, and converting each element in the mask text into a vector to obtain a mask text vector sequence; replacing each Chinese character in the original text with a corresponding pinyin to obtain a pinyin text, and converting each element in the pinyin text into a vector to obtain a pinyin text vector sequence; inputting the mask text vector sequence and the pinyin text vector sequence into a pre-trained variant text recognition model for text representation processing and domain knowledge enhancement processing to obtain a predicted text; comparing the original text and the predicted text, and determining an element different from the predicted text in the original text as a variant text; the variant text recognition model comprises a RoBERTa layer, a GAT layer and an output layer; the method of inputting the mask text vector sequence and the pinyin text vector sequence into the pre-trained variant text recognition model for text representation processing and domain knowledge enhancement processing to obtain a predicted text comprises: inputting the mask text vector sequence into the trained RoBERTa layer for text representation processing to obtain a text semantic representation vector sequence; performing initial feature fusion processing on the text semantic representation vector sequence and the pinyin text vector sequence to obtain a first fusion vector sequence; inputting the first fusion vector sequence into the trained GAT layer for domain knowledge enhancement processing to obtain an attention feature vector sequence; performing secondary feature fusion processing on the attention feature vector sequence and the text semantic representation vector sequence to obtain a second fusion vector sequence; inputting the second fusion vector sequence into the trained output layer for probability prediction processing to obtain the predicted text.

2. The variant text recognition method of claim 1, wherein, the method of identifying a suspected sensitive word in an original text and replacing the suspected sensitive word with a mask identifier to obtain a mask text, and converting each element in the mask text into a vector to obtain a mask text vector sequence comprises: finding a sensitive word recorded in a preset sensitive word set in the original text, and taking the found sensitive word as the suspected sensitive word; for each Chinese character in the original text, finding a similar word corresponding to the original word in a preset similar word table using the similar word as the original word, replacing the Chinese character in the original text with the pinyin of the original word to obtain a similar pinyin text from the original text, and finding a pinyin in the similar pinyin text that is the same as the pinyin of the sensitive word recorded in the sensitive word set, and taking the Chinese character in the original text corresponding to the found pinyin as the suspected sensitive word; replacing the suspected sensitive word in the original text with a mask identifier to obtain the mask text; converting each element in the mask text into a corresponding vector according to a preset text vector table to obtain the mask text vector sequence; wherein the similar word table is used to record the corresponding relationship between the original word and the similar word; each original word corresponds to at least one similar word.

3. The variant text recognition method of claim 1, wherein, The first feature fusion processing of the text semantic representation vector sequence and the pinyin text vector sequence comprises: Vectors at each sensitive index position in the text semantic representation vector sequence are added to vectors at the same sensitive index position in the pinyin text vector sequence to obtain a first fusion vector sequence; the sensitive index position is an index position where the suspected sensitive word is located; The second feature fusion processing of the attention feature vector sequence and the text semantic representation vector sequence comprises: Vectors at each sensitive index position in the attention feature vector sequence are used to replace vectors at the same sensitive index position in the text semantic representation vector sequence to obtain a second fusion vector sequence.

4. The variant text recognition method of claim 1, wherein, The method comprises: training the RoBERTa layer to obtain a trained RoBERTa layer; training the GAT layer and the output layer based on the trained RoBERTa layer to obtain a trained GAT layer and a trained output layer.

5. The variant text recognition method of claim 4, wherein, The method comprises: For each general training text in the general corpus, a mask identifier is used to randomly mask the general training text to obtain a masked general training text, and the masked general training text is converted into a masked general training text vector sequence by querying a text vector word table; the text vector word table is generated according to the general corpus and contains each Chinese character in the general corpus and the corresponding vector, the pinyin corresponding to each Chinese character in the general corpus and the corresponding vector, and a special identifier and the corresponding vector; the special identifier includes a mask identifier; inputting each mask general training text vector sequence to the RoBERTa layer to be trained to minimize the loss function of the RoBERTa layer iterating the accuracy of the RoBERTa layer to a first target accuracy to determine the RoBERTa layer after the parameters of the RoBERTa layer are trained, and a text semantic representation vector corresponding to each Chinese character and special identifier in the text vector vocabulary recorded in the RoBERTa layer after the parameters of the RoBERTa layer are trained. In the loss function of the RoBERTa layer , n is the number of mask identifiers after random masking of the general training text; T i is the i-th word replaced by the mask identifier in the general training text; f M (T i ) is the context of T i after random masking of the general training text, and θ is the estimated parameter of the RoBERTa layer.

6. The variant text recognition method of claim 5, wherein, The GAT layer is pre-constructed according to the text vector word table and a similar word table; The method for constructing the GAT layer comprises: For each pinyin in the text vector word table, the pinyin in the text vector word table is used as a corresponding root node, and similar words in a similar word table corresponding to original words having similar pinyins with the pinyin in the text vector word table are used as neighbor nodes corresponding to the root node to construct the GAT layer; wherein the feature vector of each root node is randomly initialized; the feature vector of each neighbor node is a text semantic representation vector corresponding to the same Chinese character recorded in the trained RoBERTa layer corresponding to the neighbor node; The similar word table is used to record the corresponding relationship between original words and similar words; each original word corresponds to at least one similar word.

7. The variant text recognition method of claim 6, wherein, The method comprises: For each field training text in the field training corpus of the field, replace the word marked as a sensitive word in the field training text with a mask identifier to obtain a mask field training text corresponding to the field training text, and vectorize the mask field training text to obtain a mask field training text vector sequence corresponding to the field training text; convert the field training text into pinyin to obtain a field training pinyin text corresponding to the field training text, and vectorize the field training pinyin text to obtain a field training pinyin text vector sequence corresponding to the field training text; input the mask field training text vector sequence corresponding to the field training text into the trained RoBERTa layer to obtain a field training text semantic representation vector sequence corresponding to the field training text; perform primary feature fusion processing on the field training text semantic representation vector sequence corresponding to the field training text and the field training pinyin text vector sequence corresponding to the field training text to obtain a first fusion training vector sequence corresponding to the field training text; input the first fusion training vector sequence corresponding to the field training text into the GAT layer to be trained for field knowledge enhancement processing to obtain a training attention feature vector sequence corresponding to the field training text; perform secondary feature fusion processing on the training attention feature vector sequence corresponding to the field training text and the field training text semantic representation vector sequence corresponding to the field training text to obtain a second fusion training vector sequence corresponding to the field training text; inputting the second fusion training vector sequence corresponding to the field training text to the output layer to be trained to train the output layer by minimizing the loss function of the output layer iterating the accuracy of the output layer to the second target accuracy to determine the trained output layer and the trained GAT layer. wherein, in the loss function of the output layer m is the number of mask identifiers in the mask domain training text corresponding to the domain training text, is the sum of the feature vectors of all neighbor nodes of the root node corresponding to the i th pinyin in the GAT layer in the domain training pinyin text corresponding to the domain training text, C is the number of all neighbor nodes of the root node corresponding to the i th pinyin in the GAT layer in the domain training pinyin text corresponding to the domain training text, G i is the i th vector in the sequence of training attention feature vectors corresponding to the domain training text; T i is the i th vector in the sequence of domain training text semantic representation vectors corresponding to the domain training text; the domain training corpus is obtained by marking the sensitive words in each text in the domain corpus to which the sensitive words belong.

8. A variant text recognition apparatus, characterized by comprising: a mask unit configured to identify a suspected sensitive word in an original text, replace the suspected sensitive word with a mask identifier to obtain a mask text, and convert each element in the mask text into a vector to obtain a mask text vector sequence; a pinyin unit configured to replace each Chinese character in the original text with a corresponding pinyin to obtain a pinyin text, and convert each element in the pinyin text into a vector to obtain a pinyin text vector sequence; a prediction unit configured to input the mask text vector sequence and the pinyin text vector sequence into a pre-trained variant text recognition model for text representation processing and field knowledge enhancement processing to obtain a predicted text; a comparison unit configured to compare the original text and the predicted text, and determine elements in the original text that are different from the predicted text as variant texts; the variant text recognition model comprises a RoBERTa layer, a GAT layer, and an output layer; the prediction unit 300 comprises: a text semantic representation prediction module configured to input the mask text vector sequence into a trained RoBERTa layer for text representation processing to obtain a text semantic representation vector sequence; a primary fusion prediction module configured to perform primary feature fusion processing on the text semantic representation vector sequence and the pinyin text vector sequence to obtain a first fusion vector sequence; an attention feature prediction module configured to input the first fusion vector sequence into the trained GAT layer for field knowledge enhancement processing to obtain an attention feature vector sequence; a secondary fusion prediction module, configured to perform secondary feature fusion processing on the attention feature vector sequence and the text semantic representation vector sequence to obtain a second fusion vector sequence; a probability prediction module, configured to input the second fusion vector sequence into a trained output layer to perform probability prediction processing to obtain the predicted text.

9. A readable storage medium, characterized by, The storage has program codes corresponding to the method of any one of claims 1-7.

Images