Text processing method and apparatus

Abstract

This application discloses a text processing method and apparatus. The method includes: acquiring the target text to be processed, and converting N target text statements contained in the target text into statement vectors to obtain N statement vectors; processing each statement vector using a target model to obtain topic probability distribution information corresponding to each statement vector; determining the probability that the target text statement corresponding to each statement vector belongs to a target type statement based on the topic probability distribution information and the target topic probability distribution information, wherein the target topic probability distribution information is used to characterize the probability that each of M topics belongs to the target type topic, and the target type statement is a statement containing text information that is prohibited from being disclosed. This application solves the technical problem of low recognition efficiency in the prior art when identifying prohibited data in text.

Description

Technical Field

[0001] This application relates to the fields of text processing and information security, and more specifically, to a text processing method and apparatus. Background Technology

[0002] In the field of information security, it may be necessary to identify sensitive information in transmitted text information. Sensitive information includes, but is not limited to, user privacy data that is prohibited from being disclosed, confidential information, and so on.

[0003] In existing technologies, sensitive information in text can typically be identified using business-customized methods such as domain keywords and feature libraries. However, this approach requires a large number of domain experts to manually process and collect keywords or feature fingerprints, resulting in high costs for sensitive information identification. Although existing technologies can also extract feature libraries using machine learning or deep learning methods, and then identify sensitive information by feature matching or determining whether feature information is contained, this approach has significant limitations. It heavily relies on the comprehensiveness of the feature information library and is often applicable to vertical or limited domains, with mediocre performance in general scenarios. Furthermore, deep learning methods often require a large amount of labeled training data for supervised training, which takes a long time to prepare and incurs significant human costs.

[0004] Therefore, existing technologies suffer from low recognition efficiency when identifying prohibited data in text. Currently, no effective solution has been proposed to address this problem. Summary of the Invention

[0005] This application provides a text processing method and apparatus to at least solve the technical problem of low recognition efficiency in the prior art when recognizing data that is prohibited from being disclosed in text.

[0006] According to one aspect of this application, a text processing method is provided, comprising: acquiring a target text to be processed, and converting N target text statements contained in the target text into statement vectors to obtain N statement vectors, wherein N is a positive integer greater than or equal to 1; processing each statement vector through a target model to obtain topic probability distribution information corresponding to each statement vector, wherein the topic probability distribution information corresponding to each statement vector represents the probability that the target text statement corresponding to the statement vector belongs to each of M topics, the target model is a mathematical model for predicting the probability that a text statement belongs to a topic, and M is a positive integer greater than or equal to 1; determining the probability that the target text statement corresponding to each statement vector belongs to a target type statement based on the topic probability distribution information corresponding to each statement vector and the target topic probability distribution information, wherein the target topic probability distribution information is used to represent the probability that each of the M topics belongs to a target type topic, the target type statement is a statement containing text information that is prohibited from being disclosed, and the target type topic is a topic involving the target type statement.

[0007] Furthermore, the text processing method also includes: performing preprocessing operations on the target text, wherein the preprocessing operations are used to divide the text content of the target text into K text sentences based on the punctuation marks in the target text, wherein the K text sentences include at least N target text sentences, and K is a positive integer greater than or equal to 1; filtering out the first text sentence among the K text sentences whose number of characters is less than a preset number of characters to obtain the remaining N target text sentences; and vectorizing the N target text sentences through a text representation model to obtain N sentence vectors, wherein the text representation model is a language model that converts text sentences into semantic vectors.

[0008] Further, the target model is obtained through the following process: A first training set is acquired, consisting of multiple user data sets, each user data set being a text statement, and the multiple user data sets including at least one target type statement; the user data in the first training set is vectorized to obtain a first vector set corresponding to the first training set, wherein each vector in the first vector set corresponds to a user data set in the first training set; topic clustering is performed on the first vector set to obtain M topics corresponding to the first vector set and the number of vectors corresponding to each of the M topics; the target model is generated based on the M topics corresponding to the first vector set and the number of vectors corresponding to each of the M topics.

[0009] Furthermore, the text processing method also includes: calculating the probability distribution of the M topics corresponding to the first vector set and the number of vectors corresponding to each of the M topics by fitting a probability distribution model to obtain the topic probability distribution information corresponding to the first vector set. The topic probability distribution information corresponding to the first vector set is used to characterize the probability that each text statement in the first training set belongs to each of the M topics. The probability distribution model is a mathematical model constructed based on a probability fitting distribution algorithm. The probability distribution model that has completed the fitting probability distribution calculation is used as the target model.

[0010] Furthermore, the text processing method also includes: after using the probability distribution model that has completed the fitting probability distribution calculation as the target model, obtaining a second training set, wherein the second training set consists of multiple public data, each public data being a text statement; vectorizing the public data in the second training set to obtain a second vector set corresponding to the second training set, wherein each vector in the second vector set corresponds to a public data in the second training set; and processing each vector in the second vector set based on the target model to obtain topic probability distribution information corresponding to the second vector set, wherein the topic probability distribution information corresponding to the second vector set is used to characterize the probability corresponding to each text statement in the second training set belonging to each of the M topics.

[0011] Furthermore, the topic probability distribution information corresponding to the first vector set is represented by an M-dimensional first probability vector, and the topic probability distribution information corresponding to the second vector set is represented by an M-dimensional second probability vector. After obtaining the topic probability distribution information corresponding to the second vector set, the text processing method further includes: calculating the reciprocal of the second probability vector to obtain a third probability vector; performing a dot product calculation on the first probability vector and the third probability vector to obtain a fourth probability vector, wherein the fourth probability vector represents the target topic probability distribution information.

[0012] Furthermore, the topic probability distribution information corresponding to each statement vector is represented by the fifth probability vector. The text processing method also includes: performing a dot product calculation on the fifth probability vector and the fourth probability vector to obtain the calculation result; and using the calculation result as the probability that the target text statement corresponding to each statement vector belongs to the target type statement.

[0013] Furthermore, the text processing method also includes: calculating the vector distance between every two vectors in the first vector set; determining V clusters corresponding to the first vector set based on the vector distance, wherein each cluster consists of at least two vectors from the first vector set, the vector distance between at least two vectors in each cluster is less than a first preset distance, each cluster corresponds to a topic, and V is a positive integer greater than M; determining M first clusters from the V clusters based on the number of vectors contained in each cluster, wherein the number of vectors contained in each first cluster is greater than the number of vectors contained in other clusters, and the other clusters are any clusters in the V clusters other than the M first clusters; Each of the M first-class clusters undergoes vector filtering to obtain M second-class clusters. The vector filtering process filters out the first vector within each first-class cluster. The first vector is the vector in the first-class cluster whose distance to the centroid of the first-class cluster is greater than a second preset distance. The centroid of the cluster is the vector located at the center of all vectors corresponding to the first-class cluster. The topic corresponding to each second-class cluster is then used as a topic corresponding to the first vector set, resulting in M ​​topics corresponding to the first vector set. The number of vectors contained in each second-class cluster is used as the number of vectors corresponding to the topic in the M topics, thus obtaining the number of vectors corresponding to each topic in the M topics.

[0014] Furthermore, the text representation model is obtained through the following process: Multiple training texts are acquired, and three consecutive text sentences in each training text are taken as a training subset. Each training text contains Y text sentences, where Y is an integer greater than or equal to 3. The three text sentences in the training subset are converted into semantic vectors, resulting in a first semantic vector, a second semantic vector, and a third semantic vector corresponding to the training subset. The first semantic vector is the vector corresponding to the first text sentence in the training subset, the second semantic vector is the vector corresponding to the second text sentence, and the third semantic vector is the vector corresponding to the third text sentence. The first text sentence precedes the second text sentence, and the second text sentence precedes the third text sentence. Vector fitting is performed on the first and third semantic vectors to obtain a fourth semantic vector. The mean squared error loss between the fourth and second semantic vectors is calculated. The preset language model is iteratively trained based on the mean squared error loss, the fourth semantic vector, and the second semantic vector to obtain the text representation model.

[0015] According to another aspect of this application, a text processing apparatus is also provided, comprising: an acquisition module, configured to acquire target text to be processed, and convert N target text statements contained in the target text into statement vectors to obtain N statement vectors, wherein N is a positive integer greater than or equal to 1; a vector processing module, configured to process each statement vector through a target model to obtain topic probability distribution information corresponding to each statement vector, wherein the topic probability distribution information corresponding to each statement vector represents the probability that the target text statement corresponding to the statement vector belongs to each of M topics, the target model is a mathematical model for predicting the probability that a text statement belongs to a topic, and M is a positive integer greater than or equal to 1; and a determination module, configured to determine the probability that the target text statement corresponding to each statement vector belongs to a target type statement based on the topic probability distribution information corresponding to each statement vector and the target topic probability distribution information, wherein the target topic probability distribution information represents the probability that each of the M topics belongs to a target type topic, the target type statement is a statement containing text information that is prohibited from being disclosed, and the target type topic is a topic involving the target type statement.

[0016] In this application, a method is adopted to vectorize text statements and then calculate the topic probability distribution of the text statements based on the statement vectors. First, the target text to be processed is obtained, and the N target text statements contained in the target text are converted into statement vectors, resulting in N statement vectors, where N is a positive integer greater than or equal to 1. Then, each statement vector is processed by a target model to obtain the topic probability distribution information corresponding to each statement vector. The topic probability distribution information corresponding to each statement vector represents the probability that the target text statement corresponding to the statement vector belongs to each of the M topics. The target model is a mathematical model used to predict the probability that a text statement belongs to a topic, where M is a positive integer greater than or equal to 1. Finally, based on the topic probability distribution information corresponding to each statement vector and the target topic probability distribution information, the probability that the target text statement corresponding to each statement vector belongs to the target type statement is determined. The target topic probability distribution information is used to represent the probability that each of the M topics belongs to the target type topic. The target type statement is a statement containing text information that is prohibited from being disclosed, and the target type topic is a topic involving the target type statement.

[0017] As can be seen from the above, this application vectorizes text statements and processes the corresponding statement vectors through a mathematical model to calculate the topic probability distribution of each text statement. Combining the probability that each of the M topics may involve sensitive information (i.e., information that is prohibited from being disclosed), the sensitive information in the text can be automatically identified through mathematical model calculation. This not only eliminates the need for a large amount of manual cost in keyword collection but also eliminates the need to consider the completeness of the feature information database. Therefore, it can save a lot of time in identifying sensitive information and improve the efficiency of identifying sensitive information in the text.

[0018] Therefore, the technical solution of this application achieves the goal of identifying sensitive information in text without constructing a feature information database of sensitive information, thereby reducing the technical effect of sensitive information identification cost and identification time, and solving the technical problem of low identification efficiency in the prior art when identifying data that is prohibited from being disclosed in text. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0020] Figure 1 This is a flowchart of an optional text processing method according to an embodiment of this application;

[0021] Figure 2 This is a flowchart of an optional text vectorization process according to an embodiment of this application;

[0022] Figure 3 This is a flowchart illustrating the training process of an optional text representation model according to an embodiment of this application.

[0023] Figure 4 This is a schematic diagram of the structure of an optional text representation model according to an embodiment of this application;

[0024] Figure 5 This is a schematic diagram of an optional text processing method according to an embodiment of this application;

[0025] Figure 6 This is a flowchart of an optional method for determining the probability distribution information of a target topic according to an embodiment of this application;

[0026] Figure 7 This is a flowchart illustrating a topic clustering operation on a first vector set according to an embodiment of this application;

[0027] Figure 8 This is a flowchart of another optional target text processing according to an embodiment of this application;

[0028] Figure 9 This is a schematic diagram of an optional text processing device provided according to an embodiment of this application. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] It should be noted that all information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this application are information and data authorized by the user or fully authorized by all parties. For example, if there is an interface between this system and the relevant user or organization, before obtaining the relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain the relevant information after receiving the consent information from the aforementioned user or organization.

[0032] The present application will be further described below with reference to various embodiments. Example 1

[0033] According to an embodiment of this application, an embodiment of a text processing method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0034] To facilitate a clearer understanding of the solutions described in the embodiments of this application, the following provides a brief explanation of any technical terms that may be involved.

[0035] Text representation: A technique for transforming raw text data into data that is easier for machine learning applications. Common representation techniques include TF-IDF, WORD2VEC, and BERT. Among them, TF-IDF (term frequency–inverse document frequency) is a commonly used weighted technique for text information retrieval and mining, which is actually TF * IDF, where TF is the term frequency and IDF is the inverse document frequency. Word2vec (word embeddings) is used to convert words in natural language into dense vectors that computers can understand. BERT (Bidirectional Encoder Representation from Transformers) is a pre-trained large-scale language representation model, often used for text data representation and other language text processing tasks.

[0036] Topic analysis: Topic analysis is a text analysis technique. In topic modeling, a topic represents a concept or direction, which can be divided into different granular levels and represented as a series of related words or content, which are the conditional probabilities of these words or content.

[0037] Topic clustering: Based on the topic characteristics of text data, texts with higher topic similarity are aggregated into the same cluster to complete the clustering fitting of the topic distribution of text data.

[0038] Gaussian Distribution: Also known as the normal distribution, the Gaussian distribution is a distribution where the random variable X follows a mathematical expectation of μ and a variance of σ. 2 The probability density function of a Gaussian distribution is normally distributed, denoted as X ~ N(μ, σ). The expected value μ determines its location, and its standard deviation σ determines the amplitude of the distribution.

[0039] Figure 1 This is a flowchart of an optional text processing method according to an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps:

[0040] Step S101: Obtain the target text to be processed, and convert the N target text statements contained in the target text into statement vectors to obtain N statement vectors. Wherein, N is a positive integer greater than or equal to 1.

[0041] Optionally, in this application, a text processing system serves as the executing entity of the text processing method. Specifically, the target text to be processed is text that needs to be reviewed, and the review content is to check whether there is sensitive information in the target text that is prohibited from being disclosed, such as user privacy data, company confidential data, etc.

[0042] Text processing systems can be deployed on servers, firewalls, or terminal devices such as computers and mobile phones. Taking a text processing system deployed on a firewall as an example, the firewall receives target text sent by the server and calls the text processing system to review whether the target text contains sensitive information that should not be disclosed. The text processing system first splits the content of the target text into N target text statements, and then converts each target text statement into a statement vector, thus obtaining N statement vectors.

[0043] Step S102: Process each statement vector through the target model to obtain the topic probability distribution information corresponding to each statement vector.

[0044] In step S102, the topic probability distribution information corresponding to each statement vector represents the probability that the target text statement corresponding to the statement vector belongs to each of the M topics. The target model is a mathematical model used to predict the probability that a text statement belongs to a topic, where M is a positive integer.

[0045] Optionally, the target model in this application is a mathematical model constructed based on a probability distribution algorithm. Unlike neural network models, the target model in this application does not require a large amount of labeling work on training data during construction, thereby saving labor costs and shortening the identification time of sensitive information.

[0046] Furthermore, the target model in this application is used to predict the probability that the target text statement corresponding to the input statement vector belongs to a topic. For example, assuming M is 50, when the target model processes statement vector A, the output topic probability distribution information corresponding to statement vector A represents the probability that statement vector A belongs to each of the 50 topics. For example, the probability that the text statement corresponding to statement vector A belongs to topic 1 is P1, the probability that the text statement corresponding to statement vector A belongs to topic 2 is P2, the probability that the text statement corresponding to statement vector A belongs to topic 3 is P3, and so on.

[0047] Step S103: Determine the probability that the target text statement corresponding to each statement vector belongs to the target type statement based on the topic probability distribution information and the target topic probability distribution information corresponding to each statement vector.

[0048] In step S103, the target topic probability distribution information is used to characterize the probability that each topic among the M topics belongs to the target type topic. The target type statement is a statement containing text information that is prohibited from being published, and the target type topic is a topic involving the target type statement.

[0049] Optionally, the target topic probability distribution information is used to characterize the probability that each topic among the M topics belongs to the target type topic. For example, when M is 50, the target topic probability distribution information is used to characterize the probability that each topic among the 50 topics belongs to the target type topic. For example, the probability that topic 1 belongs to the target type topic is P1.1, the probability that topic 2 belongs to the target type topic is P2.2, and the probability that topic 3 belongs to the target type topic is P3.3.

[0050] Furthermore, in this application, the target topic probability distribution information and the topic probability distribution information corresponding to each statement vector are both M-dimensional probability vectors. For example, the target topic probability distribution information is represented by vector H1, and the topic probability distribution information corresponding to statement vector A is represented by vector H-A1. Then, the probability that the target text statement corresponding to statement vector A belongs to the target type statement is the product of vector H1 and vector H-A1.

[0051] Based on the above steps S101 to S103, this application employs a method of vectorizing text statements and then calculating the topic probability distribution of the text statements based on the statement vectors. First, the target text to be processed is obtained, and the N target text statements contained in the target text are converted into statement vectors, resulting in N statement vectors, where N is a positive integer. Then, each statement vector is processed by a target model to obtain the topic probability distribution information corresponding to each statement vector. The topic probability distribution information corresponding to each statement vector represents the probability that the target text statement corresponding to that statement vector belongs to each of the M topics. The target model is a mathematical model used to predict the probability that a text statement belongs to a topic, where M is a positive integer greater than or equal to 1. Finally, based on the topic probability distribution information corresponding to each statement vector and the target topic probability distribution information, the probability that the target text statement corresponding to each statement vector belongs to a target type statement is determined. The target topic probability distribution information represents the probability that each of the M topics belongs to a target type topic. A target type statement is a statement containing text information that is prohibited from being disclosed, and a target type topic is a topic involving a target type statement.

[0052] As can be seen from the above, this application vectorizes text statements and processes the corresponding statement vectors through a mathematical model to calculate the topic probability distribution of each text statement. Combining the probability that each of the M topics may involve sensitive information (i.e., information that is prohibited from being disclosed), the sensitive information in the text can be automatically identified through mathematical model calculation. This not only eliminates the need for a large amount of manual cost in keyword collection but also eliminates the need to consider the completeness of the feature information database. Therefore, it can save a lot of time in identifying sensitive information and improve the efficiency of identifying sensitive information in the text.

[0053] Therefore, the technical solution of this application achieves the goal of identifying sensitive information in text without constructing a feature information database of sensitive information, thereby reducing the technical effect of sensitive information identification cost and identification time, and solving the technical problem of low identification efficiency in the prior art when identifying data that is prohibited from being disclosed in text.

[0054] In one alternative embodiment, Figure 2 A flowchart illustrating an optional text vectorization process according to an embodiment of this application is shown, such as... Figure 2 As shown, it includes the following steps:

[0055] Step S201: Perform preprocessing operations on the target text.

[0056] In step S201, the preprocessing operation is used to divide the text content of the target text into K text statements based on the punctuation marks in the target text, wherein the K text statements include at least N target text statements, and K is a positive integer greater than or equal to 1.

[0057] Optionally, the text processing system can divide the text content of the target text into multiple text sentences based on the punctuation marks in the target text. For example, using common punctuation marks such as commas and periods as delimiters, the text content of the target text can be segmented into sentences to obtain K text sentences corresponding to the target text.

[0058] It is important to note that this application uses text statements as the minimum transformation object for vector transformation. Compared to using characters or words as vector transformation objects, text statements have richer semantic information and are more complete in expression, thus they can better characterize whether the text contains sensitive information. For example, a single number cannot characterize a user's private data, but multiple numbers strung together may characterize a user's mobile phone number or ID card number.

[0059] Step S202: Filter the first text statement among the K text statements whose number of characters is less than the preset number of characters to obtain the remaining N target text statements.

[0060] Optionally, to reduce the number of text statements processed and prevent irrelevant text statements from participating in the subsequent sensitive information identification process, the text processing system can filter out the first text statement among the K text statements whose character count is less than a preset character count, thereby obtaining the remaining N target text statements. The preset character count can be customized, for example, set to 2. This application does not impose any special limitations on the setting of the preset character count.

[0061] It should be noted that since statements with too few characters contain less semantic information, these statements can be directly filtered out without participating in the subsequent vector conversion process, thereby improving the efficiency of the system's computing resources.

[0062] Step S203: Vectorize the N target text statements using a text representation model to obtain N statement vectors. The text representation model is a language model that converts text statements into semantic vectors.

[0063] Optionally, the text representation model in this application can be a pre-trained large-scale language representation model, such as the BERT model. Additionally, the text representation model can convert text sentences into sentence vectors (also known as semantic vectors).

[0064] In one alternative embodiment, Figure 3 A flowchart illustrating the training process of an optional text representation model according to an embodiment of this application is shown, such as... Figure 3 As shown, it includes the following steps:

[0065] Step S301: Obtain multiple training texts and take three consecutive text statements in each training text as a training subset.

[0066] In step S301, each training text contains Y text statements, where Y is an integer greater than or equal to 3.

[0067] Optionally, assuming there is a training text D, by segmenting the training text D into sentences, 5 text sentences (corresponding to Y) can be obtained in sequence, for example, sentence 1-1, sentence 1-2, sentence 1-3, sentence 1-4, and sentence 1-5.

[0068] Furthermore, after obtaining 5 text statements, the text processing system can use every three consecutive text statements as a training subset. For example, statements 1-1, 1-2, and 1-3 can be used as training subset C1, statements 1-2, 1-3, and 1-4 can be used as training subset C2, and statements 1-3, 1-4, and 1-5 can be used as training subset C3.

[0069] Step S302: Convert the three text statements in the training subset into semantic vectors respectively to obtain the first semantic vector, the second semantic vector, and the third semantic vector corresponding to the training subset.

[0070] In step S302, the first semantic vector is the vector corresponding to the first text statement in the training subset, the second semantic vector is the vector corresponding to the second text statement in the training subset, and the third semantic vector is the vector corresponding to the third text statement in the training subset. The first text statement is located before the second text statement, and the second text statement is located before the third text statement.

[0071] Optionally, taking the above training subset C1 as an example, the text processing system performs vector transformation on sentences 1-1, 1-2 and 1-3 in the training subset C1 respectively to obtain the first semantic vector corresponding to sentence 1-1, the second speech vector corresponding to sentence 1-2 and the third semantic vector corresponding to sentence 1-3.

[0072] Step S303: Perform vector fitting on the first semantic vector and the third semantic vector to obtain the fourth semantic vector.

[0073] Step S304: Calculate the mean squared error loss between the fourth semantic vector and the second semantic vector.

[0074] Step S305: Based on the mean squared error loss value, the fourth semantic vector, and the second semantic vector, the preset language model is iteratively trained to obtain the text representation model.

[0075] Optionally, such as Figure 4 As shown, Figure 4 This is a schematic diagram of an optional text representation model according to an embodiment of this application. By applying a sliding window processing method to the training text, multiple training subsets can be constructed. For example, a training subset consists of three text sentences (text1, text2, text3), where the three text sentences are consecutive segments within the same paragraph. Then, these three text sentences (text1, text2, text3) are fed into the BERT model, which outputs three semantic vectors. These are then linearly transformed through a fully connected Dense layer, outputting a multi-dimensional sentence vector embedding. By fitting the first and last sentence vectors of text1 and text3, the representation vector corresponding to text2 (corresponding to the fourth semantic vector) is obtained. Then, based on the semantic vector and representation vector corresponding to text2, the BERT model is trained and optimized using the mean squared error loss (MSE) as the loss function. After fine-tuning, an effective BERT representation model (i.e., a text representation model) can be obtained.

[0076] In one optional embodiment, to generate the target model, the text processing system first acquires a first training set, which consists of multiple user data sets, each user data set being a text statement, and including at least one target type statement among the multiple user data sets. Then, the text processing system vectorizes the user data in the first training set to obtain a first vector set corresponding to the first training set, where each vector in the first vector set corresponds to one user data set in the first training set. Subsequently, the text processing system performs topic clustering on the first vector set to obtain M topics corresponding to the first vector set and the number of vectors corresponding to each of the M topics. Finally, the text processing system generates the target model based on the M topics corresponding to the first vector set and the number of vectors corresponding to each of the M topics.

[0077] Optionally, Figure 5 This is a schematic diagram of an optional text processing method according to an embodiment of this application, such as... Figure 5 As shown, Figure 5 The user data in the dataset constitutes the first training set. The text representation model can then be used to vectorize the user data in the first training set, thereby obtaining the first vector set corresponding to the first training set (corresponding to...). Figure 5 The text processing system first obtains a user vector set, and then performs topic clustering on the user vector set to obtain M topics corresponding to the user vector set, such as topic 1, topic 2, etc. Each topic corresponds to at least one vector.

[0078] Furthermore, the text processing system can generate a target model based on the M topics corresponding to the first vector set and the number of vectors corresponding to each of the M topics. Specifically, the text processing system uses a probability distribution model to fit the probability distribution of the M topics corresponding to the first vector set and the number of vectors corresponding to each of the M topics to obtain the topic probability distribution information corresponding to the first vector set. This topic probability distribution information is used to characterize the probability that each text statement in the first training set belongs to each of the M topics. The probability distribution model is a mathematical model constructed based on a probability fitting distribution algorithm. Finally, the text processing system uses the probability distribution model that has completed the fitting probability distribution calculation as the target model.

[0079] Optionally, such as Figure 5 As shown, the probability distribution model can be a mathematical model constructed based on a Gaussian fitting distribution algorithm. The text processing system can use this probability distribution model to calculate the probability distribution of the M topics corresponding to the user vector set and the number of vectors corresponding to each of the M topics using Gaussian fitting, thereby obtaining the topic probability distribution information corresponding to the user vector set (i.e., Figure 5The topic density distribution of user data in the training set. The topic probability distribution information corresponding to the user vector set is used to characterize the probability that each text statement in the first training set belongs to each of the M topics. For example, if the first training set contains user data V1 and user data V2, then the topic probability distribution information corresponding to the user vector set characterizes the probability that user data V1 belongs to each topic and the probability that user data V2 belongs to each topic.

[0080] In addition, Figure 5 In the middle, the probability distribution model for completing the Gaussian fitting probability distribution calculation is as follows: Figure 5 The Gaussian model in this application is the target model.

[0081] In one optional embodiment, after using the probability distribution model that has completed the fitting probability distribution calculation as the target model, the text processing system obtains a second training set, which consists of multiple public data points, each of which is a text statement. Then, the text processing system vectorizes the public data in the second training set to obtain a second vector set corresponding to the second training set, where each vector in the second vector set corresponds to a public data point in the second training set. Finally, the text processing system processes each vector in the second vector set based on the target model to obtain topic probability distribution information corresponding to the second vector set, where the topic probability distribution information corresponding to the second vector set is used to characterize the probability that each text statement in the second training set belongs to each of the M topics.

[0082] Optionally, such as Figure 5 As shown, Figure 5 The publicly available data in the dataset forms the second training set. The text processing system can use a text representation model to vectorize the publicly available data in the second training set, thereby obtaining a public vector set (corresponding to the second vector set). Subsequently, the text processing system calls a Gaussian model to process each vector in the public vector set, obtaining the topic probability distribution information corresponding to the public vector set. Figure 5 The topic density distribution of the public data in the dataset. It should be noted that the topic probability distribution information corresponding to the public vector set is used to characterize the probability that each public data belongs to each of the M topics. For example, if the second training set contains public data W1 and public data W2, then the topic probability distribution information corresponding to the public vector set characterizes the probability that public data W1 belongs to each topic, and the probability that public data W2 belongs to each topic.

[0083] In one optional embodiment, the topic probability distribution information corresponding to the first vector set is represented by an M-dimensional first probability vector, and the topic probability distribution information corresponding to the second vector set is represented by an M-dimensional second probability vector. After obtaining the topic probability distribution information corresponding to the second vector set, the text processing system can calculate the reciprocal of the second probability vector to obtain a third probability vector, and then perform a dot product calculation on the first probability vector and the third probability vector to obtain a fourth probability vector, wherein the fourth probability vector represents the target topic probability distribution information.

[0084] Optionally, such as Figure 5 As shown, after obtaining the topic density distribution of user data (corresponding to the topic probability distribution information of the first vector set) and the topic density distribution of public data (corresponding to the topic probability distribution information of the second vector set), the text processing system can also perform a differential comparison between the topic density distribution of user data and the topic density distribution of public data. The specific process of differential comparison is as follows: assuming that the first probability vector representing the topic density distribution of user data is P(pos) and the second probability vector representing the topic density distribution of public data is P(neg), the text processing system calculates the reciprocal of P(neg) to obtain the third probability component 1 / P(neg). Finally, since both the first probability vector P(pos) and the third probability component 1 / P(neg) are M-dimensional probability vectors, the text processing system can perform a dot product calculation on P(pos) and 1 / P(neg) to obtain the fourth probability vector P = P(pos) * (1 / P(neg)), where the fourth probability vector P represents the target topic probability distribution information.

[0085] It should be noted that if a text statement in the first training set has fewer semantically similar text statements that can be matched in the second training set, the likelihood of that text statement containing sensitive information is higher, and vice versa. This is because the second training set consists entirely of publicly available data. With a sufficient amount of publicly available data in the second training set, if a text statement cannot be matched with any semantically similar publicly available data, or if the number of matched publicly available data is extremely small, it indicates that the text statement may have never appeared in publicly available data, or rarely appears in publicly available data. Therefore, the likelihood of that text statement containing prohibited information is relatively high.

[0086] Based on the above theories, such as Figure 6As shown, this application utilizes the Gaussian model obtained during the topic definition fitting process to perform inference and prediction on vectors in both the user vector set and the public vector set. Then, it obtains the probability distribution corresponding to each vector belonging to M topics, and accumulates these probability distributions to obtain the user data topic density distribution and the public data topic density distribution. Specifically, this application can use the user data topic density distribution as the positive sample density distribution and the public data topic density distribution as the negative sample density distribution. By comparing the differences between the positive and negative sample topic probability density distributions, it can obtain the target topic probability distribution information P, which measures the probability that each topic belongs to a sensitive topic.

[0087] In one optional embodiment, the topic probability distribution information corresponding to each statement vector can be represented by a fifth probability vector. Based on this, the text processing system can perform a dot product calculation on the fifth probability vector and the fourth probability vector to obtain the calculation result, and then use the calculation result as the probability that the target text statement corresponding to each statement vector belongs to the target type statement.

[0088] Optionally, since the topic probability distribution information corresponding to each statement vector represents the probability that the target text statement corresponding to the statement vector belongs to each of the M topics, the fifth probability vector is also an M-dimensional probability vector. Therefore, assuming that the fifth probability vector corresponding to the i-th statement vector in the N statement vectors is P(i), the probability that the target text statement corresponding to the i-th statement vector belongs to the target type statement is P(i)*P.

[0089] like Figure 5 As shown, for the target text (i.e. Figure 5 Each target text statement corresponding to the new test text in the dataset is also converted into a statement vector using a text representation model, and then processed by the target model (i.e., ...). Figure 5 The Gaussian model in the model is used to perform inference tests on the statement vectors to obtain the topic probability distribution information corresponding to the statement vectors. Finally, the probability that the target text statement corresponding to the statement vector belongs to the target type statement is calculated by combining the target topic probability distribution information. Figure 5 The sensitivity probability of the target text statement corresponding to the statement vector shown.

[0090] In one alternative embodiment, the process of performing topic clustering on the first vector set is described below, such as... Figure 7 As shown, it includes the following steps:

[0091] Step S701: Calculate the vector distance between every two vectors in the first vector set.

[0092] Step S702: Determine the V clusters corresponding to the first vector set based on the vector distance.

[0093] In step S702, each cluster consists of at least two vectors from the first vector set, the vector distance between at least two vectors in each cluster is less than a first preset distance, each cluster corresponds to a topic, and V is a positive integer greater than M.

[0094] Optionally, the closer the vector distance between two vectors, the more similar the two vectors are. When both vectors are semantic vectors, this indicates that the two text statements corresponding to the two vectors are semantically similar, and thus two text statements with similar pronunciations can be categorized into the same topic. Based on this theory, this application can group all vectors in the first vector set into N clusters according to the vector distance between any two vectors. Each cluster consists of at least two vectors whose vector distance is less than a first preset distance, and each cluster corresponds to a topic. The first preset distance can be customized.

[0095] Step S703: Determine M first clusters from V clusters based on the number of vectors contained in each cluster.

[0096] In step S703, the number of vectors contained in each first cluster is greater than the number of vectors contained in other clusters. The other clusters are any clusters among the V clusters other than the M first clusters.

[0097] Optionally, to further improve the fitting effect of topic distribution, the text processing system can iteratively filter outliers during the topic clustering fitting process to prevent some abnormal data points from having too much impact on the topic distribution fitting. For example, clusters with fewer vector data can be directly filtered out. For instance, if V is 60 and M is 50, the text processing system can filter out the 10 clusters with the fewest vectors, ultimately obtaining the 50 clusters with the most vectors as the first cluster.

[0098] Step S704: Determine the M topics corresponding to the first vector set and the number of vectors corresponding to each topic in the M topics based on the M first clusters.

[0099] Optionally, the text processing system can perform vector filtering on each of the M first-class clusters to obtain M second-class clusters. The vector filtering is used to filter the first vector in each first-class cluster. The first vector is the vector in the first-class cluster whose distance from the centroid of the first-class cluster is greater than a second preset distance. The centroid of the cluster is the vector located at the center position among all vectors corresponding to the first-class cluster.

[0100] Then, the text processing system takes the topic corresponding to each second cluster as a topic corresponding to the first vector set, and obtains M topics corresponding to the first vector set. The number of vectors contained in each second cluster is taken as the number of vectors corresponding to the topic corresponding to the second cluster in the M topics, and the number of vectors corresponding to each topic in the M topics is obtained.

[0101] Specifically, for each first cluster, further vector filtering can be performed to remove discrete points within the cluster. The text processing system can iteratively traverse each cluster obtained by cluster fitting, calculate the distance between the cluster centroid and each element (i.e., each vector within the cluster), and then filter out cluster elements (i.e., the first vector within the cluster) that exceed a second preset distance.

[0102] Finally, the text processing system takes the first cluster that has completed vector filtering as the second cluster, takes the topic corresponding to the second cluster as a topic corresponding to the first vector set, and takes the number of vectors contained in each second cluster as the number of vectors corresponding to the topic corresponding to the second cluster among the M topics.

[0103] In one alternative embodiment, Figure 8 This is another optional target text processing flowchart according to an embodiment of this application, such as... Figure 8 As shown, for a new input target text, the target text is first segmented into K text sentences. Among them, text sentences with too short a length can be directly assigned the recognition result as insensitive. Then, the remaining target text sentences are fed into the text representation model for text vectorization to obtain the sentence vectors corresponding to the target text sentences. Gaussian models are used for inference and prediction to obtain the topic probability distribution information of each target text sentence in each topic direction. Combined with the previously obtained target topic probability distribution information P, the sensitivity probability of each target text sentence can be calculated.

[0104] It is important to note that in the field of data loss prevention, it is often necessary to identify and monitor users' sensitive and critical information to prevent accidental leakage of this data due to non-compliant or negligent operations, which could cause potential losses to users. Current technologies typically involve uploading user data, organizing and classifying it, identifying and defining sensitive data for each category, and developing different interception strategies based on these categories. Therefore, an efficient and accurate sensitive information identification method can improve the accuracy of matching and interception, reduce false positives and false negatives, and also reduce the manual costs associated with domain experts by automating the definition and identification process of sensitive information.

[0105] It should be noted that existing technologies typically define sensitive information data by pre-setting a sensitive information fingerprint database (vocabulary, text), or by extracting sensitive information fingerprint features through machine learning methods. These fingerprints are then matched with features extracted from the test text data, or the test text is directly judged to determine whether it contains sensitive information fingerprint features, thereby identifying whether the test data is sensitive. In this process, text representation is particularly important. Common representation methods include: frequency-based methods: representing text by statistically analyzing the frequencies of key features, such as TF-IDF; word vector-based methods: representing text using trained word vectors, such as word2vec; and pre-trained model-based methods: using pre-trained and pre-researched models to perform semantic representation of text, such as BERT.

[0106] However, existing technologies that use domain keywords and feature fingerprint databases to identify sensitive information require extensive manual processing and collection by domain experts, resulting in significant costs. Methods based on machine learning or deep learning to extract feature fingerprint databases and then perform fingerprint feature matching or determine whether feature information is contained have significant limitations. They heavily rely on feature information databases and are often applicable to vertical or limited domains, with limited performance in general scenarios. Furthermore, deep learning methods often require large amounts of labeled training data for supervised training.

[0107] The technical solution of this application provides a general method for identifying sensitive information data for text data in different industry sectors. This method can automatically mine sensitive information in user data, reduce the chances of false or missed interception by data security protection systems, and effectively protect the secure transmission of user data.

[0108] The technical solution of this application can be applied to many data leakage prevention and data security protection scenarios. It can detect and identify data transmitted and operated by users to determine whether it belongs to the user's sensitive information data, and prevent the accidental leakage of sensitive information due to non-compliant or negligent operation behavior. In addition, the text vectorization technology, topic clustering technology, and topic probability distribution statistical technology involved in this application can be replaced by other similar methods / techniques, which have strong scalability and practicality. Example 2

[0109] This embodiment provides an optional text processing device, wherein each implementation unit / module in the text processing device corresponds to each implementation step in Embodiment 1.

[0110] Figure 9 This is a schematic diagram of an optional text processing device provided according to an embodiment of this application, such as... Figure 9 As shown, it includes: an acquisition module 901, a vector processing module 902, and a determination module 903.

[0111] Specifically, the acquisition module 901 is used to acquire the target text to be processed and convert the N target text statements contained in the target text into statement vectors to obtain N statement vectors, where N is a positive integer greater than or equal to 1; the vector processing module 902 is used to process each statement vector through the target model to obtain the topic probability distribution information corresponding to each statement vector, where the topic probability distribution information corresponding to each statement vector represents the probability that the target text statement corresponding to the statement vector belongs to each of the M topics, and the target model is a mathematical model used to predict the probability that a text statement belongs to a topic, where M is a positive integer greater than or equal to 1; the determination module 903 is used to determine the probability that the target text statement corresponding to each statement vector belongs to the target type statement based on the topic probability distribution information and the target topic probability distribution information, where the target topic probability distribution information represents the probability that each of the M topics belongs to the target type topic, the target type statement is a statement containing text information that is prohibited from being published, and the target type topic is a topic involving the target type statement.

[0112] It should be noted that the target text to be processed is text that requires review. The review involves checking whether the target text contains sensitive information that should not be disclosed, such as user privacy data, company confidential data, etc. The target model in this application is a mathematical model constructed based on a probability distribution algorithm. Unlike neural network models, the target model in this application does not require extensive annotation of training data during construction, thereby saving labor costs and shortening the time for identifying sensitive information.

[0113] Furthermore, the target model in this application is used to predict the probability that the target text statement corresponding to the input statement vector belongs to a topic. For example, assuming M is 50, when the target model processes statement vector A, the output topic probability distribution information corresponding to statement vector A represents the probability that statement vector A belongs to each of the 50 topics. For example, the probability that the text statement corresponding to statement vector A belongs to topic 1 is P1, the probability that the text statement corresponding to statement vector A belongs to topic 2 is P2, the probability that the text statement corresponding to statement vector A belongs to topic 3 is P3, and so on.

[0114] Optionally, the target topic probability distribution information is used to characterize the probability that each topic among the M topics belongs to the target type topic. For example, when M is 50, the target topic probability distribution information is used to characterize the probability that each topic among the 50 topics belongs to the target type topic. For example, the probability that topic 1 belongs to the target type topic is P1.1, the probability that topic 2 belongs to the target type topic is P2.2, and the probability that topic 3 belongs to the target type topic is P3.3.

[0115] Furthermore, in this application, the target topic probability distribution information and the topic probability distribution information corresponding to each statement vector are both M-dimensional probability vectors. For example, the target topic probability distribution information is represented by vector H1, and the topic probability distribution information corresponding to statement vector A is represented by vector H-A1. Then, the probability that the target text statement corresponding to statement vector A belongs to the target type statement is the product of vector H1 and vector H-A1.

[0116] As the above analysis shows, this application employs a method of vectorizing text statements and then calculating the topic probability distribution of the text statements based on the statement vectors. First, the target text to be processed is obtained, and the N target text statements contained in the target text are converted into statement vectors, resulting in N statement vectors, where N is a positive integer. Then, each statement vector is processed by a target model to obtain the topic probability distribution information corresponding to each statement vector. The topic probability distribution information for each statement vector represents the probability that the target text statement corresponding to that statement vector belongs to each of the M topics. The target model is a mathematical model used to predict the probability that a text statement belongs to a topic, where M is a positive integer. Finally, based on the topic probability distribution information and the target topic probability distribution information for each statement vector, the probability that the target text statement corresponding to each statement vector belongs to the target type statement is determined. The target topic probability distribution information represents the probability that each of the M topics belongs to the target type topic. Target type statements are statements containing text information that is prohibited from being disclosed, and target type topics are topics related to target type statements.

[0117] As can be seen from the above, this application vectorizes text statements and processes the corresponding statement vectors through a mathematical model to calculate the topic probability distribution of each text statement. Combining the probability that each of the M topics may involve sensitive information (i.e., information that is prohibited from being disclosed), the sensitive information in the text can be automatically identified through mathematical model calculation. This not only eliminates the need for a large amount of manual cost in keyword collection but also eliminates the need to consider the completeness of the feature information database. Therefore, it can save a lot of time in identifying sensitive information and improve the efficiency of identifying sensitive information in the text.

[0118] Therefore, the technical solution of this application achieves the goal of identifying sensitive information in text without constructing a feature information database of sensitive information, thereby reducing the technical effect of sensitive information identification cost and identification time, and solving the technical problem of low identification efficiency in the prior art when identifying data that is prohibited from being disclosed in text.

[0119] Optionally, the vector processing module includes: a text preprocessing unit, a filtering unit, and a vectorization processing unit. The text preprocessing unit performs preprocessing operations on the target text, specifically dividing the target text into K text sentences based on punctuation marks. Each of the K text sentences includes at least N target text sentences, where K is a positive integer greater than or equal to 1. The filtering unit filters out the first text sentence from the K text sentences, where the number of characters is less than a preset number, to obtain the remaining N target text sentences. The vectorization processing unit performs vectorization processing on the N target text sentences using a text representation model, resulting in N sentence vectors. The text representation model is a language model that converts text sentences into semantic vectors.

[0120] Optionally, the text preprocessing unit can divide the text content of the target text into multiple text sentences based on the punctuation marks in the target text. For example, it can use common punctuation marks such as commas and periods as delimiters to perform sentence segmentation on the text content of the target text, thereby obtaining K text sentences corresponding to the target text.

[0121] It is important to note that this application uses text statements as the minimum transformation object for vector transformation. Compared to using characters or words as vector transformation objects, text statements have richer semantic information and are more complete in expression, thus they can better characterize whether the text contains sensitive information. For example, a single number cannot characterize a user's private data, but multiple numbers strung together may characterize a user's mobile phone number or ID card number.

[0122] Furthermore, to reduce the number of text statements processed and prevent irrelevant text statements from participating in the subsequent sensitive information identification process, the filtering unit can filter the first text statement among the K text statements whose character count is less than a preset character count, thereby obtaining the remaining N target text statements. The preset character count can be customized, for example, set to 2. This application does not impose any special limitations on the setting of the preset character count.

[0123] It should be noted that since statements with too few characters contain less semantic information, these statements can be directly filtered out without participating in the subsequent vector conversion process, thereby improving the efficiency of the system's computing resources.

[0124] Optionally, the text representation model in this application can be a pre-trained large-scale language representation model, such as the BERT model. Additionally, the text representation model can convert text sentences into sentence vectors (also known as semantic vectors).

[0125] Optionally, the text processing device further includes: a first acquisition module, a first processing module, a topic clustering module, and a generation module. The first acquisition module is used to acquire a first training set, wherein the first training set consists of multiple user data sets, each user data set being a text statement, and the multiple user data sets including at least one target type statement; the first processing module is used to perform vectorization processing on the user data in the first training set to obtain a first vector set corresponding to the first training set, wherein each vector in the first vector set corresponds to a user data set in the first training set; the topic clustering module is used to perform topic clustering operations on the first vector set to obtain M topics corresponding to the first vector set and the number of vectors corresponding to each of the M topics; the generation module is used to generate a target model based on the M topics corresponding to the first vector set and the number of vectors corresponding to each of the M topics.

[0126] Optionally, the generation module includes a first computation unit and a model processing unit. The first computation unit is used to perform probability distribution calculations on the M topics corresponding to the first vector set and the number of vectors corresponding to each of the M topics using a probability distribution model, thereby obtaining the topic probability distribution information corresponding to the first vector set. This topic probability distribution information characterizes the probability that each text statement in the first training set belongs to each of the M topics. The probability distribution model is a mathematical model constructed based on a probability fitting distribution algorithm. The model processing unit is used to take the probability distribution model that has completed the probability distribution calculation as the target model.

[0127] Optionally, the text processing device further includes: a second acquisition module, a second processing module, and a third processing module. The second acquisition module is used to acquire a second training set, wherein the second training set consists of multiple public data sets, each public data set being a text statement; the second processing module is used to vectorize the public data in the second training set to obtain a second vector set corresponding to the second training set, wherein each vector in the second vector set corresponds to a public data set in the second training set; the third processing module is used to process each vector in the second vector set based on the target model to obtain topic probability distribution information corresponding to the second vector set, wherein the topic probability distribution information corresponding to the second vector set is used to characterize the probability that each text statement in the second training set belongs to each of the M topics.

[0128] Optionally, the topic probability distribution information corresponding to the first vector set is represented by an M-dimensional first probability vector, and the topic probability distribution information corresponding to the second vector set is represented by an M-dimensional second probability vector. The text processing device further includes a first calculation module and a second calculation module. The first calculation module is used to calculate the reciprocal of the second probability vector to obtain a third probability vector; the second calculation module is used to perform a dot product calculation on the first and third probability vectors to obtain a fourth probability vector, wherein the fourth probability vector represents the target topic probability distribution information.

[0129] Optionally, the topic probability distribution information corresponding to each statement vector is represented by a fifth probability vector. The determination module includes a second calculation unit and a calculation result processing unit. The second calculation unit performs a dot product calculation on the fifth probability vector and the fourth probability vector to obtain the calculation result; the calculation result processing unit uses the calculation result as the probability that the target text statement corresponding to each statement vector belongs to the target type statement.

[0130] Optionally, the topic clustering module includes: a third calculation unit, a first determination unit, a second determination unit, a vector filtering unit, a topic determination unit, and a vector quantity determination unit. The third calculation unit is used to calculate the vector distance between every two vectors in the first vector set; the first determination unit is used to determine V clusters corresponding to the first vector set based on the vector distances, wherein each cluster consists of at least two vectors from the first vector set, the vector distance between at least two vectors in each cluster is less than a first preset distance, each cluster corresponds to a topic, and V is a positive integer greater than M; the second determination unit is used to determine M first clusters from the V clusters based on the number of vectors contained in each cluster, wherein the number of vectors contained in each first cluster is greater than the number of vectors contained in other clusters, and other clusters are any clusters in the V clusters other than the M first clusters; the vector filtering unit uses... The system performs vector filtering on each of the M first-class clusters to obtain M second-class clusters. The vector filtering process filters out the first vector in each first-class cluster. The first vector is a vector in the first-class cluster whose distance to the centroid of the first-class cluster is greater than a second preset distance. The centroid of the cluster is the vector located at the center of all vectors corresponding to the first-class cluster. A topic determination unit is used to treat the topic corresponding to each second-class cluster as a topic corresponding to a first vector set, obtaining M topics corresponding to the first vector set. A vector quantity determination unit is used to treat the number of vectors contained in each second-class cluster as the number of vectors corresponding to the topic in the M topics, obtaining the number of vectors corresponding to each topic in the M topics.

[0131] Optionally, the text processing device includes: a third acquisition module, a vector transformation module, a vector fitting module, a vector calculation module, and a model training module. The system comprises the following modules: a third acquisition module, which acquires multiple training texts and uses three consecutive text statements in each training text as a training subset, wherein each training text contains Y text statements, where Y is an integer greater than or equal to 3; a vector conversion module, which converts the three text statements in the training subset into semantic vectors, resulting in a first semantic vector, a second semantic vector, and a third semantic vector corresponding to the training subset, wherein the first semantic vector is the vector corresponding to the first text statement in the training subset, the second semantic vector is the vector corresponding to the second text statement in the training subset, and the third semantic vector is the vector corresponding to the third text statement in the training subset, with the first text statement preceding the second text statement and the second text statement preceding the third text statement; a vector fitting module, which performs vector fitting on the first and third semantic vectors to obtain a fourth semantic vector; a vector calculation module, which calculates the mean squared error loss between the fourth and second semantic vectors; and a model training module, which iteratively trains a preset language model based on the mean squared error loss, the fourth semantic vector, and the second semantic vector to obtain a text representation model. Example 3

[0132] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the text processing method of any one of the embodiments 1 above by executing the executable instructions. Example 4

[0133] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to execute any of the text processing methods in Embodiment 1 above.

[0134] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0135] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0136] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0137] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0138] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0139] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A text processing method, characterized in that, include: Obtain the target text to be processed, and convert the N target text statements contained in the target text into statement vectors to obtain N statement vectors, where N is a positive integer greater than or equal to 1; Obtain a first training set, wherein the first training set consists of multiple user data, each user data is a text statement, and the multiple user data includes at least one target type statement; perform vectorization processing on the user data in the first training set to obtain a first vector set corresponding to the first training set, wherein each vector in the first vector set corresponds to a user data in the first training set, and the topic probability distribution information corresponding to the first vector set is represented by an M-dimensional first probability vector; Obtain a second training set, which consists of multiple public data sets, each of which is a text statement; vectorize the public data in the second training set to obtain a second vector set corresponding to the second training set, wherein each vector in the second vector set corresponds to a public data set in the second training set; process each vector in the second vector set based on the target model to obtain the topic probability distribution information corresponding to the second vector set, wherein the topic probability distribution information corresponding to the second vector set is used to characterize the probability that each text statement in the second training set belongs to each of the M topics, and the topic probability distribution information corresponding to the second vector set is represented by an M-dimensional second probability vector; Calculate the reciprocal of the second probability vector to obtain the third probability vector; perform a dot product of the first probability vector and the third probability vector to obtain the fourth probability vector, wherein the fourth probability vector represents the probability distribution information of the target topic; Each statement vector is processed by the target model to obtain the topic probability distribution information corresponding to each statement vector. The topic probability distribution information corresponding to each statement vector represents the probability that the target text statement corresponding to the statement vector belongs to each of the M topics. The target model is a mathematical model for predicting the probability that a text statement belongs to a topic. M is a positive integer greater than or equal to 1. Based on the topic probability distribution information corresponding to each statement vector and the target topic probability distribution information, the probability that the target text statement corresponding to each statement vector belongs to the target type statement is determined. The target topic probability distribution information is used to characterize the probability that each topic among M topics belongs to the target type topic. The target type statement is a statement containing text information that is prohibited from being published. The target type topic is a topic that involves the target type statement.

2. The text processing method according to claim 1, characterized in that, The N target text statements contained in the target text are converted into statement vectors, resulting in N statement vectors, including: The target text is preprocessed, wherein the preprocessing operation is used to divide the text content of the target text into K text sentences according to the punctuation marks in the target text, wherein the K text sentences include at least the N target text sentences, and K is a positive integer greater than or equal to 1; Filter the first text statement among the K text statements whose number of characters is less than a preset number of characters to obtain the remaining N target text statements; The N target text statements are vectorized using a text representation model to obtain the N statement vectors. The text representation model is a language model that converts text statements into semantic vectors.

3. The text processing method according to claim 1, characterized in that, The target model is obtained through the following process: Perform topic clustering on the first vector set to obtain M topics corresponding to the first vector set and the number of vectors corresponding to each of the M topics; The target model is generated based on the M topics corresponding to the first vector set and the number of vectors corresponding to each of the M topics.

4. The text processing method according to claim 3, characterized in that, The target model is generated based on the M topics corresponding to the first vector set and the number of vectors corresponding to each of the M topics, including: The probability distribution model is used to fit the probability distribution of the M topics corresponding to the first vector set and the number of vectors corresponding to each of the M topics to obtain the topic probability distribution information of the first vector set. The topic probability distribution information of the first vector set is used to characterize the probability that each text statement in the first training set belongs to each of the M topics. The probability distribution model is a mathematical model constructed based on the probability fitting distribution algorithm. The probability distribution model that completes the calculation of the fitted probability distribution is used as the target model.

5. The text processing method according to claim 4, characterized in that, The topic probability distribution information corresponding to each statement vector is represented by a fifth probability vector. The probability that the target text statement corresponding to each statement vector belongs to the target type statement is determined based on the topic probability distribution information and the target topic probability distribution information, including: Perform a dot product calculation on the fifth probability vector and the fourth probability vector to obtain the calculation result; The calculation result is used as the probability that the target text statement corresponding to each statement vector belongs to the target type statement.

6. The text processing method according to claim 3, characterized in that, Perform topic clustering on the first vector set to obtain M topics corresponding to the first vector set and the number of vectors corresponding to each of the M topics, including: Calculate the vector distance between every two vectors in the first vector set; V clusters corresponding to the first vector set are determined based on the vector distance, wherein each cluster consists of at least two vectors from the first vector set, the vector distance between the at least two vectors in each cluster is less than a first preset distance, each cluster corresponds to a topic, and V is a positive integer greater than M; Based on the number of vectors contained in each cluster, M first clusters are determined from the V clusters, wherein the number of vectors contained in each first cluster is greater than the number of vectors contained in other clusters, and the other clusters are any clusters in the V clusters other than the M first clusters. Vector filtering is performed on each of the M first clusters to obtain M second clusters. The vector filtering is used to filter the first vector in each first cluster. The first vector is the vector in the first cluster whose distance from the centroid of the first cluster is greater than a second preset distance. The centroid of the cluster is the vector located at the center position among all vectors corresponding to the first cluster. Each topic corresponding to the second type of cluster is used as a topic corresponding to the first vector set, resulting in M ​​topics corresponding to the first vector set; The number of vectors contained in each second cluster is taken as the number of vectors corresponding to the topic in the M topics that corresponds to the second cluster, thus obtaining the number of vectors corresponding to each topic in the M topics.

7. The text processing method according to claim 2, characterized in that, The text representation model is obtained through the following process: Obtain multiple training texts and take three consecutive text statements in each training text as a training subset. Each training text contains Y text statements, where Y is an integer greater than or equal to 3. The three text statements in the training subset are converted into semantic vectors respectively to obtain the first semantic vector, the second semantic vector, and the third semantic vector corresponding to the training subset. The first semantic vector is the vector corresponding to the first text statement in the training subset, the second semantic vector is the vector corresponding to the second text statement in the training subset, and the third semantic vector is the vector corresponding to the third text statement in the training subset. The first text statement is located before the second text statement, and the second text statement is located before the third text statement. A fourth semantic vector is obtained by performing vector fitting on the first semantic vector and the third semantic vector. Calculate the mean squared error loss between the fourth semantic vector and the second semantic vector; The preset language model is iteratively trained based on the mean squared error loss value, the fourth semantic vector, and the second semantic vector to obtain the text representation model.

8. A text processing device, characterized in that, include: The acquisition module is used to acquire the target text to be processed and convert the N target text statements contained in the target text into statement vectors to obtain N statement vectors, where N is a positive integer greater than or equal to 1; A first acquisition module is used to acquire a first training set, wherein the first training set consists of multiple user data, each user data is a text statement, and the multiple user data includes at least one target type statement; a first processing module is used to perform vectorization processing on the user data in the first training set to obtain a first vector set corresponding to the first training set, wherein each vector in the first vector set corresponds to a user data in the first training set, and the topic probability distribution information corresponding to the first vector set is represented by an M-dimensional first probability vector; The second acquisition module is used to acquire a second training set, wherein the second training set consists of multiple public data, each of which is a text statement; the second processing module is used to vectorize the public data in the second training set to obtain a second vector set corresponding to the second training set, wherein each vector in the second vector set corresponds to a public data in the second training set; the third processing module is used to process each vector in the second vector set based on the target model to obtain topic probability distribution information corresponding to the second vector set, wherein the topic probability distribution information corresponding to the second vector set is used to characterize the probability that each text statement in the second training set belongs to each of the M topics, and the topic probability distribution information corresponding to the second vector set is represented by an M-dimensional second probability vector; The first calculation module is used to calculate the reciprocal of the second probability vector to obtain the third probability vector; the second calculation module is used to perform a dot product calculation on the first probability vector and the third probability vector to obtain the fourth probability vector, wherein the fourth probability vector represents the probability distribution information of the target topic. The vector processing module is used to process each statement vector through the target model to obtain the topic probability distribution information corresponding to each statement vector. The topic probability distribution information corresponding to each statement vector represents the probability that the target text statement corresponding to the statement vector belongs to each of the M topics. The target model is a mathematical model for predicting the probability that a text statement belongs to a topic. M is a positive integer greater than or equal to 1. The determination module is used to determine the probability that the target text statement corresponding to each statement vector belongs to the target type statement based on the topic probability distribution information and the target topic probability distribution information corresponding to each statement vector. The target topic probability distribution information is used to characterize the probability that each topic among M topics belongs to the target type topic. The target type statement is a statement containing text information that is prohibited from being published. The target type topic is a topic that involves the target type statement.

Images