Text detection method and device

Abstract

The embodiments of the present application provide a text detection method and device. The text detection method includes: obtaining multiple first texts and multiple second texts, the first texts are texts marked with labels, and the second texts are sensitive texts without labels; based on the labels marked on each first text, extracting the label feature vectors corresponding to each first text, and determining the sensitive texts in the multiple first texts based on the label feature vectors; based on the determined sensitive texts and multiple second texts, generating a sensitive reference text set, and detecting the similarity between each text in the sensitive reference text set and the text to be detected, to obtain multiple detection results; if it is determined based on the multiple detection results that there is a text in the sensitive reference text set whose similarity with the text to be detected meets the similarity condition, then the text to be detected is determined to be a sensitive text. The technical solution of the embodiments of the present application improves the accuracy and precision of sensitive text detection.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a text detection method and device. Background Art

[0002] With the advancement of digital transformation, more and more companies are moving various data files online, realizing electronic office work. This has brought with it a growing demand for sensitive data detection and leakage prevention technologies. However, current sensitive text detection technologies have low accuracy and are prone to numerous issues such as missed and false positives. Summary of the Invention

[0003] The embodiments of the present application provide a text detection method and apparatus, which can improve the accuracy and precision of sensitive text detection, at least to a certain extent.

[0004] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by practice of the present application.

[0005] According to one aspect of an embodiment of the present application, a text detection method is provided, including: obtaining multiple first texts and multiple second texts, the first texts being texts marked with labels, and the second texts being sensitive texts without labels; extracting label feature vectors corresponding to the respective first texts based on the labels marked with the respective first texts, and determining sensitive texts among the multiple first texts based on the label feature vectors; generating a sensitive reference text set based on the determined sensitive texts and the multiple second texts, and detecting the similarity between each text in the sensitive reference text set and a text to be detected to obtain multiple detection results; if it is determined based on the multiple detection results that there is a text in the sensitive reference text set whose similarity with the text to be detected meets a similarity condition, then determining that the text to be detected is a sensitive text.

[0006] According to one aspect of an embodiment of the present application, a text detection device is provided, including: an acquisition unit, configured to acquire multiple first texts and multiple second texts, the first texts being texts marked with labels, and the second texts being sensitive texts not marked with labels; a first determination unit, configured to extract label feature vectors corresponding to each first text based on the labels marked with each first text, and determine the sensitive texts in the multiple first texts according to the label feature vectors; a detection unit, configured to generate a sensitive reference text set based on the determined sensitive texts and the multiple second texts, and detect the similarity between each text in the sensitive reference text set and a text to be detected to obtain multiple detection results; a second determination unit, configured to determine that the text to be detected is a sensitive text if it is determined, based on the multiple detection results, that there is a text in the sensitive reference text set whose similarity with the text to be detected meets a similarity condition.

[0007] In some embodiments of the present application, based on the aforementioned scheme, the first determination unit includes: an input subunit, configured to input the label feature values ​​corresponding to the each first text into a trained text classification model, and obtain the sensitivity probability corresponding to the each first text output by the text classification model, the sensitive probability being the probability that the first text is a sensitive text, and the text classification model is trained based on multiple labeled training texts and text categories corresponding to each training text, and the text categories include sensitive text and non-sensitive text; a determination subunit, configured to determine the sensitive text among the multiple first texts based on the sensitive probability corresponding to the each first text.

[0008] In some embodiments of the present application, based on the aforementioned scheme, the determination subunit is configured to: take the first text among the multiple first texts whose sensitivity probability is greater than a first preset threshold as the sensitive text among the multiple first texts.

[0009] In some embodiments of the present application, based on the aforementioned scheme, the detection unit is configured to: determine the Hamming distance between each text in the sensitive reference text set and the text to be detected based on the first hash signature value corresponding to each text in the sensitive reference text set and the second hash signature value corresponding to the text to be detected; the second determination unit is configured to: if it is determined based on multiple Hamming distances that there is a text in the sensitive reference text set whose Hamming distance with the text to be detected is less than or equal to a second preset threshold, then determine that the text to be detected is a sensitive text.

[0010] In some embodiments of the present application, based on the aforementioned scheme, the device also includes: a first word segmentation unit, configured to perform word segmentation on each text in the sensitive reference text set to obtain a first feature word set corresponding to each text; a first processing unit, configured to perform hash processing on each feature word in the first feature word set to obtain a hash value of each feature word; a first generation unit, configured to generate a first hash signature value corresponding to each text in the sensitive reference text set based on the hash value of each feature word and the weight corresponding to each feature word.

[0011] In some embodiments of the present application, based on the aforementioned scheme, the first generation unit is configured to: perform weighted processing on the corresponding hash values ​​according to the weights corresponding to the respective feature words to obtain weighted numerical values ​​corresponding to the respective feature words; merge and accumulate the weighted numerical values ​​corresponding to the respective feature words to obtain accumulated numerical values, and perform dimensionality reduction processing on the accumulated numerical values ​​to obtain the first hash signature value corresponding to each text in the sensitive reference text set.

[0012] In some embodiments of the present application, based on the aforementioned scheme, the device also includes: a second word segmentation unit, configured to perform word segmentation on each text in the sensitive reference text set to obtain a first feature word set corresponding to each text; a second processing unit, configured to perform hash processing on the target feature word in the first feature word set to obtain a hash value of the target feature word; a second generation unit, configured to generate a first hash signature value corresponding to each text in the sensitive reference text set based on the hash value of the target feature word and the weight corresponding to the target feature word.

[0013] In some embodiments of the present application, based on the aforementioned scheme, the device further includes: a calculation unit, configured to calculate the word frequency inverse file frequency of each feature word in the first feature word set in the sensitive reference text set; a selection unit, configured to select a preset number of feature words from the first feature word set as the target feature words in the order of the word frequency inverse file frequency from high to low.

[0014] In some embodiments of the present application, based on the aforementioned scheme, the calculation unit is configured to: count the reverse file frequency of each feature word appearing in the sensitive reference text set, and count the frequency of each feature word appearing in the sensitive reference text set; and take the product of the reverse file frequency and the frequency as the word frequency reverse file frequency of each feature word in the sensitive reference text set.

[0015] According to one aspect of an embodiment of the present application, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the text detection method as described in the above embodiment is implemented.

[0016] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the text detection method as described in the above embodiments.

[0017] According to one aspect of an embodiment of the present application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the text detection method provided in the various optional embodiments described above.

[0018] In the technical solutions provided by some embodiments of the present application, a plurality of first texts and a plurality of second texts are obtained, wherein the first text is a text marked with a label and the second text is a sensitive text not marked with a label. Then, by extracting the label feature vector corresponding to each of the first texts, the sensitive texts in the plurality of first texts are determined, and then a sensitive reference text set is generated based on the determined sensitive texts and the plurality of second texts. The similarity between the text to be detected and each text in the sensitive reference text set is detected. If, based on the multiple detection results, it is determined that there is a text in the sensitive reference text set that satisfies the similarity condition with the text to be detected, then the text to be detected can be determined to be a sensitive text. Compared with the traditional method of matching text content using rules, the technical solutions of the embodiments of the present application do not require pre-configured rules, but only need to generate a sensitive reference text set, so that the detection of the text to be detected can be directly performed based on the similarity detection of each text in the sensitive reference text set. This can avoid the problem of missed detection caused by only data that meets the preset rules when detecting through preset rules, while other data cannot be detected. At the same time, it can also avoid the problem of false detection caused by errors in the preset rules, thereby improving the accuracy and precision of sensitive text detection.

[0019] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are incorporated into and constitute a part of the specification, illustrating embodiments consistent with the present application and, together with the specification, explaining the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can derive other drawings based on these drawings without inventive effort. In the drawings:

[0021] Figure 1 A schematic diagram showing an exemplary system architecture to which the technical solutions of the embodiments of the present application can be applied;

[0022] Figure 2 A flowchart of a text detection method according to an embodiment of the present application is shown;

[0023] Figure 3 A flowchart of a text detection method according to an embodiment of the present application is shown;

[0024] Figure 4 A flowchart of a text detection method according to an embodiment of the present application is shown;

[0025] Figure 5 A flowchart of a text detection method according to an embodiment of the present application is shown;

[0026] Figure 6 A flowchart of a text detection method according to an embodiment of the present application is shown;

[0027] Figure 7 A flowchart of a text detection method according to an embodiment of the present application is shown;

[0028] Figure 8 A flowchart of a text detection method according to an embodiment of the present application is shown;

[0029] Figure 9 A flowchart of a text detection method according to an embodiment of the present application is shown;

[0030] Figure 10 A logic diagram of a text detection method according to an embodiment of the present application is shown;

[0031] Figure 11 A block diagram of a text detection device according to an embodiment of the present application is shown;

[0032] Figure 12 A schematic diagram of the structure of a computer system suitable for implementing an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0033] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art.

[0034] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner.In the following description, many specific details are provided so as to provide a full understanding of the embodiments of the present application. However, it will be appreciated by those skilled in the art that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps etc. can be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0035] It should be noted that the terms used in the specification and claims of this application and the above-mentioned drawings are only used to describe the embodiments and are not intended to limit the scope of this application. It should be understood that the terms "comprise", "include", "have", etc., when used in this document, specify the presence of the stated features, wholes, steps, operations, elements, components and / or groups thereof, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof.

[0036] It will be further understood that, although the terms "first," "second," "third," etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the present invention, a first element may be referred to as a second element. Similarly, a second element may be referred to as a first element. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0037] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0038] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0039] It should be understood that, in the present application, “at least one (item)” refers to one or more, and “more” refers to two or more.

[0040] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0041] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0042] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0043] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning through demonstration.

[0044] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, drones, robots, smart medical care, smart customer service, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0045] The solutions provided in the embodiments of this application involve technologies such as machine learning based on artificial intelligence, and are specifically described through the following embodiments:

[0046] Figure 1 A schematic diagram showing an exemplary system architecture to which the technical solutions of the embodiments of the present application can be applied is shown in FIG. Figure 1 As shown, the system architecture 100 may include a terminal device 101 and a server 102 .

[0047] The terminal device 101 is connected to the server 102 via a network. The network is used to provide a medium for a communication link between the terminal device 101 and the server 102. The network can include various connection types, such as wired or wireless communication links or fiber optic cables. The terminal device 101 and the server 102 can be connected directly or indirectly via wired or wireless communication, which is not limited here.

[0048] The terminal device 101 can be a device such as a smartphone, a tablet computer, a smart TV, a desktop computer, an in-vehicle computer, and a portable computer, but is not limited thereto. The terminal device 101 can generally refer to one of multiple terminal devices. The embodiments of the present application only use the terminal device 101 as an example. Those skilled in the art will appreciate that the number of the above-mentioned terminal devices can be more or less. For example, the above-mentioned terminal device can be only one, or the above-mentioned terminal devices can be dozens, hundreds, or even more. In this case, other terminal devices are also included in the above-mentioned implementation environment. The embodiments of the present application do not limit the number and device types of terminal devices.

[0049] Server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms.

[0050] In some embodiments, the terminal device 101 obtains the text to be detected, which may be text determined based on user input of the terminal device 101, and then sends the text to be detected to the server 102 via the network. The server detects the text to be detected and determines whether it is sensitive text. In other embodiments, the terminal device 101 may directly detect the text to be detected after obtaining the text to be detected to determine whether it is sensitive text.

[0051] Figure 2 A flowchart of a text detection method according to an embodiment of the present application is shown. The text detection method can be executed by a server, which can be Figure 1 The server 102 shown in FIG. Figure 2 As shown, the text detection method includes:

[0052] Step S210: Acquire a plurality of first texts and a plurality of second texts, wherein the first texts are texts marked with labels, and the second texts are sensitive texts not marked with labels;

[0053] Step S220: extracting label feature vectors corresponding to the respective first texts based on the labels annotated with the respective first texts, and determining sensitive texts among the plurality of first texts according to the label feature vectors;

[0054] Step S230: generating a sensitive reference text set based on the determined sensitive text and the plurality of second texts, and detecting similarities between each text in the sensitive reference text set and the text to be detected, to obtain a plurality of detection results;

[0055] Step S240: If it is determined based on the multiple detection results that there is text in the sensitive reference text set whose similarity with the text to be detected meets the similarity condition, then the text to be detected is determined to be sensitive text.

[0056] These steps are described in detail below.

[0057] In step S210, a plurality of first texts and a plurality of second texts are obtained, wherein the first texts are texts marked with labels, and the second texts are sensitive texts not marked with labels.

[0058] The first text is pre-labeled text. The label may be a label based on the sensitivity of the first text, for example, the label may be top secret, confidential, private, etc. The second text is unlabeled sensitive text. Sensitive text refers to text containing sensitive content, for example, content involving user privacy, including user identity information, social relationship links, and user online behavior records. User identity information includes the user's social network account password, date of birth, ID number, age, place of birth, and permanent residence. User online behavior records include web browsing history and online shopping history.

[0059] In this embodiment, the text may be in the form of written language, such as an article, a sentence, a paragraph, or a phrase, etc. The text may be in English, Chinese, or French, etc., which is not limited in this embodiment of the present application.

[0060] In step S220 , based on the labels annotated on each first text, a label feature vector corresponding to each first text is extracted, and sensitive texts among the plurality of first texts are determined according to the label feature vector.

[0061] In this embodiment, if it is necessary to determine the sensitive text in multiple first texts, it is necessary to extract the label feature vector corresponding to each first text based on the label annotated by each first text. Specifically, the label feature vector is a vector composed of at least one pre-specified label feature. The specified label features include: label type, the position where the label appears in the text, the position where the label appears in the sentence, text format, text size, etc. It can be understood that: determining the sensitive text in the first text by combining multiple categories of features into a vector is more accurate than using a single category of features to determine the sensitive text in the first text, thereby avoiding misjudgment.

[0062] Tags can include first-, second-, and third-category tags. First-category tags can be classified based on confidentiality levels, including top secret, confidential, and private. Second-category tags can be tags with negative connotations, indicating that the document is prohibited from external dissemination. Examples include: "Not for distribution," "Not for public release," "Not for unauthorized dissemination in any form," "Not for unauthorized dissemination without written permission," "Not for unauthorized excerpting unless otherwise agreed upon," "For internal use only," "For review only," "Prohibited from external release," and "Lawful liability pursued." Third-category tags can be any other tags that indicate the sensitivity of the text. Tags can appear at the top, middle, or bottom of the text. Tags can appear on the left, middle, or right of a sentence. When a tag begins a sentence, it is positioned "left," when it ends a sentence, it is positioned "right," and when it appears elsewhere in the sentence, it is positioned "center." Text formats include PDF, DOCX, and PPTX.

[0063] The purpose of extracting the label feature vector is to automatically classify the first text based on the label feature vector. Therefore, after extracting the label feature vector corresponding to each first text, the multiple first texts can be further classified based on the label feature vector to determine sensitive text and non-sensitive text among the multiple first texts.

[0064] In step S230, a sensitive reference text set is generated based on the determined sensitive text and the multiple second texts, and the similarity between each text in the sensitive reference text set and the text to be detected is detected to obtain multiple detection results.

[0065] After determining the sensitive texts among the multiple first texts, a sensitive reference text set can be generated based on the determined sensitive texts and the multiple second texts to detect the similarity between each text in the sensitive reference text set and the text to be detected, thereby obtaining multiple detection results. Specifically, assuming that the determined sensitive texts are text A and text B, and the multiple second texts are text C, text D, and text E, then the sensitive reference text set can include text A, text B, text C, text D, and text E, and text A, text B, text C, text D, and text E are all sensitive texts.

[0066] Among them, the text to be detected can be the text to be detected uploaded by the terminal device, and the text to be detected can be the text input by the user of the terminal device, such as the text input by the user of the terminal device through a user interactive device such as a keyboard and a touch screen, or it can be the text obtained by recognizing the voice of the user of the terminal device. In other embodiments, the text can also be obtained by the terminal device through other means.

[0067] The logic of the above-mentioned text similarity detection can be: using a similarity algorithm to calculate the similarity detection result between two texts, for example, using a vector space model (VSM), first segmenting the text, then establishing a text vector, and converting the similarity calculation into a calculation of a certain feature vector distance, such as cosine angle, Euclidean distance, Jaccard similarity coefficient, etc.

[0068] In step S240, if it is determined based on the multiple detection results that there is text in the sensitive reference text set whose similarity with the text to be detected meets the similarity condition, then the text to be detected is determined to be sensitive text.

[0069] Among them, the similarity condition is a pre-set condition for classifying the text to be detected as sensitive text, that is, if there is text in the sensitive reference text set whose similarity with the text to be detected meets the similarity condition, the text to be detected can be classified as sensitive text.

[0070] When similarity is used to represent the detection results, the similarity condition can be a similarity threshold. The similarity threshold is a pre-set lower limit of similarity for classifying text into a certain category. That is, if there is text in the sensitive reference text set whose similarity with the text to be detected is greater than the similarity threshold, then the text to be detected can be determined to be sensitive text.

[0071] Compared with the traditional method of matching text content using rules, the technical solution of the above embodiment does not require pre-configuration of rules. It only needs to generate a sensitive reference text set, so that the detection of the text to be detected can be directly performed based on the similarity detection of each text in the sensitive reference text set. This can avoid the problem of missed detection caused by only data that meets the preset rules being detected when detecting through preset rules, while other data cannot be detected. It can also avoid the problem of false detection caused by errors in the preset rules, thereby improving the accuracy and precision of sensitive text detection.

[0072] In one embodiment of the present application, a method of a trained text classification model can be used to determine sensitive texts in a plurality of first texts, such as Figure 3 As shown, in this embodiment, the process may specifically include steps S310 to S320, which are described in detail as follows:

[0073] Step S310: Input the label feature values ​​corresponding to the respective first texts into a trained text classification model to obtain the sensitivity probabilities corresponding to the respective first texts output by the text classification model, wherein the sensitivity probability is the probability that the first text is a sensitive text. The text classification model is trained based on a plurality of labeled training texts and text categories corresponding to the respective training texts, wherein the text categories include sensitive texts and non-sensitive texts.

[0074] In this embodiment, it is necessary to use a trained text classification model, which is obtained by training a preset classification model based on a plurality of labeled training texts and the text categories corresponding to each training text. Among them, the preset classification model can be a support vector machine (SVM) classification model, a naive Bayes classification model, etc., which is not specifically limited in this embodiment of the present application. The text category corresponding to each training text is pre-set, and the text category can include sensitive text and non-sensitive text, that is, which training texts are sensitive texts and which training texts are non-sensitive texts are pre-set. When using the training text to train the preset classification model, it is also necessary to extract the label feature values ​​corresponding to each training text. It can be understood that the type of features contained in the label feature values ​​used during training is the same as the type of features contained in the label feature values ​​used when determining the sensitive text of the first text.

[0075] After the text classification model is trained, the label feature values ​​corresponding to each first text are input into the trained text classification model, and the sensitivity probability corresponding to each first text output by the text classification model can be obtained, wherein the sensitive probability is the probability that the first text is a sensitive text. The sensitive probability is a conditional probability. Under the conditions that the input layer, hidden layer and output layer in the text classification model have been set, and the sensitive text and non-sensitive text have been pre-set, after the label feature values ​​are input, the probability of the output text being a sensitive text is obtained.

[0076] Step S320: Determine sensitive texts among the multiple first texts according to the sensitivity probabilities corresponding to the respective first texts.

[0077] Specifically, after obtaining the sensitivity probabilities corresponding to the respective first texts, the sensitive texts among the plurality of first texts may be determined according to the sensitivity probabilities.

[0078] In one embodiment of the present application, the method of determining sensitive texts among the plurality of first texts according to the sensitivity probabilities corresponding to the respective first texts may specifically include:

[0079] The first text among the multiple first texts whose sensitivity probability is greater than a first preset threshold is used as the sensitive text among the multiple first texts.

[0080] Among them, the first preset threshold can be set according to actual needs. For example, the first preset threshold is set to 90%. Then, if there is a first text with a sensitivity probability greater than 90% among multiple first texts, the first text with a sensitivity probability greater than 90% can be used as the sensitive text among the multiple first texts.

[0081] In one embodiment of the present application, the method for detecting the similarity between each text in the sensitive reference text set and the text to be detected can be based on the comparison of the first hash signature value corresponding to each text in the sensitive reference text set and the second hash signature value corresponding to the text to be detected, specifically, Figure 4 As shown, the process may specifically include steps S410 to S420, which are described in detail as follows:

[0082] Step S410: Determine the Hamming distance between each text in the sensitive reference text set and the text to be detected based on the first hash signature value corresponding to each text in the sensitive reference text set and the second hash signature value corresponding to the text to be detected.

[0083] Calculating a hash signature value is a method of obtaining a minimum hash signature for a text using a single hash function. The result of this calculation is a unique hash value for the text. In this embodiment, the first hash signature value corresponding to each text in the sensitive reference text set can represent the specific content of each text, and the second hash signature value can represent the specific content of the text to be detected. Therefore, the first and second hash signature values ​​can be used to determine the similarity between each text in the sensitive reference text set and the text to be detected.

[0084] Since the hash signature value of a text is actually a multi-bit binary code, in information coding, the Hamming distance refers to the number of different codes at the corresponding positions of two codes. In other words, the Hamming distance refers to the number of characters required to replace one string of code to transform it into another string of code. Therefore, in this embodiment, the Hamming distance can be calculated using the first signature hash value and the second signature hash value to determine whether the text to be detected is similar to each text in the sensitive reference text set. For example, if the first hash signature value corresponding to a text in the sensitive reference text set is 1011101, and the second hash signature value corresponding to the text to be detected is 1001001, then it can be determined that the Hamming distance between the text in the sensitive reference text set and the text to be detected is 2.

[0085] Step S420: If it is determined based on multiple Hamming distances that there is text in the sensitive reference text set whose Hamming distance to the text to be detected is less than or equal to a second preset threshold, then the text to be detected is determined to be a sensitive text.

[0086] Among them, the second preset threshold can be a pre-set value. When the Hamming distance between two texts is less than or equal to the second preset threshold, the two text contents are considered similar. Otherwise, the two text contents are not similar. Furthermore, according to the calculation method of the Hamming distance and the similarity judgment rule, the second preset threshold can be set to 3.

[0087] In this embodiment, after calculating the Hamming distance between each text in the sensitive reference text set and the text to be detected and obtaining multiple Hamming distances, if it is determined based on the multiple Hamming distances that there is text in the sensitive reference text set whose Hamming distance with the text to be detected is less than or equal to a second preset threshold, that is, there is text in the sensitive reference text set that is similar to the text to be detected, then it can be determined that the text to be detected is sensitive text.

[0088] In one embodiment of the present application, Figure 5 As shown, the method for generating the first hash signature value corresponding to each text in the sensitive reference text set may specifically include steps S510 to S530, which are described in detail as follows:

[0089] Step S510: Segment each text in the sensitive reference text set to obtain a first feature word set corresponding to each text.

[0090] Specifically, each text in the sensitive reference text set is first segmented to obtain individual words, which then constitute the first feature word set corresponding to each text. In this embodiment, a variety of open source or non-open source word segmentation tools can be used to segment the sensitive reference text, such as StandardAnalyzer, ChineseAnalyzer, CJKAnalyzer, and other open source word segmentation tools, which are not specifically limited in this embodiment of the present application.

[0091] In some embodiments, pre-processing can be performed during the word segmentation process to filter out symbolic information and redundant semantic information within each text. For example, since emoticons and punctuation marks generally do not contain sensitive information, they can be pre-filtered to reduce computational effort. Furthermore, repeated word segmentations that are obvious typos can be filtered out to reduce computational effort.

[0092] Step S520: Perform hash processing on each feature word in the first feature word set to obtain a hash value of each feature word.

[0093] After segmenting each text in the sensitive reference text set to obtain the corresponding first feature word set, a hash operation can be performed on each feature word in the first feature word set to obtain a hash value for each feature word. The hash value is a binary number consisting of multiple bits. This converts the feature word from text into a vector representation, facilitating subsequent processing.

[0094] Among them, the hash function is an encryption algorithm mainly used in the field of information security. The hash function is also called a hash function. It is a function that converts an input of arbitrary length into an output of fixed length through a hash algorithm. Its output is a fixed-length string, called a hash value. The essence of this conversion is a compression mapping, that is, the space of hash values ​​is usually smaller than the space of input values. An important property of the hash function is irreversibility. That is, given a hash value, it is extremely difficult to calculate the message it corresponds to. Among them, the output values ​​corresponding to different input values ​​are basically different, and the input values ​​corresponding to different output values ​​are different. Therefore, it is impossible to determine the unique input value from the hash value. In other words, there is no way to get the original input data from the output hash value. Common hash algorithms based on the hash principle include Secure Hash Algorithm 1 (SHA-1), Secure Hash Algorithm 224 (SHA-224), Secure Hash Algorithm 256 (SHA-256), Secure Hash Algorithm 384 (SHA-384), Secure Hash Algorithm 512 (SHA-512), and MD5 Message-Digest Algorithm (MD5). The embodiments of the present application do not limit the type of hash algorithm.

[0095] Step S530: Generate a first hash signature value corresponding to each text in the sensitive reference text set according to the hash value of each feature word and the weight corresponding to each feature word.

[0096] Among them, the weight corresponding to each feature word refers to the number of times each feature word appears in the corresponding text. When calculating the i-th bit of the first hash signature value corresponding to each text, the value of the hash value of each feature word in the text at its i-th bit is obtained. Then, a weighted sum is performed based on the value of the hash value of each feature word at its i-th bit and the weight corresponding to each feature word to obtain the value of the i-th bit of the first hash signature value, thereby obtaining the first hash signature value corresponding to the text.

[0097] In one embodiment of the present application, Figure 6 As shown, step S530 may specifically include steps S610 to S620, which are described in detail as follows:

[0098] Step S610: performing weighted processing on the corresponding hash values ​​according to the weights corresponding to the respective feature words to obtain weighted values ​​corresponding to the respective feature words.

[0099] For example, let's take text A in the sensitive reference text set as an example. Assume that text A has n feature words, and the weights corresponding to each feature word are:

[0100] The weight corresponding to the first feature word is: w1;

[0101] The weight corresponding to the second feature word is: w2;

[0102] …

[0103] The weight corresponding to the nth feature word is w n ;

[0104] And the hash values ​​of each feature word are:

[0105] The hash value of the first feature word is: 100011;

[0106] The hash value of the second feature word is: 011011;

[0107] …

[0108] The hash value of the nth feature word is: 011011;

[0109] Then, the corresponding hash values ​​are weighted according to the weights corresponding to the feature words, and the weighted values ​​corresponding to the feature words are:

[0110] The weighted value corresponding to the first feature word is: w1-w1-w1-w1w1w1

[0111] The corresponding weighted value of the second feature word is: -w2w2w2-w2w2w2

[0112] …

[0113] The weighted value corresponding to the nth feature word is: -w n w n w n -w n w n w n

[0114] Step S620: Combine and accumulate the weighted values ​​corresponding to the feature words to obtain an accumulated value, and perform dimensionality reduction processing on the accumulated value to obtain a first hash signature value corresponding to each text in the sensitive reference text set.

[0115] After calculating the weighted values ​​corresponding to each feature word through step S610, the weighted values ​​can be merged and accumulated to obtain an accumulated value, that is, multiple weighted data are merged and accumulated into one value, and then the accumulated value is subjected to dimensionality reduction processing, wherein the dimensionality reduction processing can specifically be to take 0 when the accumulated value is negative and take 1 when it is positive, and finally obtain the first hash signature value corresponding to each text.

[0116] Continuing with the example of step S610 above, assuming that the accumulated values ​​are 13, 108, -22, -5, -32, 55, dimensionality reduction processing can be performed on the accumulated values, and finally the first hash signature value corresponding to text A is 110001.

[0117] In another embodiment of the present application, Figure 7 As shown, the method for generating the first hash signature value corresponding to each text in the sensitive reference text set may further specifically include:

[0118] Step S710: Segment each text in the sensitive reference text set to obtain a first feature word set corresponding to each text;

[0119] Step S720: performing hash processing on the target feature word in the first feature word set to obtain a hash value of the target feature word;

[0120] Step S730: Generate a first hash signature value corresponding to each text in the sensitive reference text set according to the hash value of the target feature word and the weight corresponding to the target feature word.

[0121] It should be noted that step S710 and step S730 are respectively Figure 5 The contents described in step S510 and step S530 in the shown embodiment are similar. The difference is that in this embodiment, the target feature words in the first feature word set are hashed, and then the first hash signature value corresponding to each text in the sensitive reference text set is generated based on the hash value of the target feature word and the weight corresponding to the target feature word. The reason for doing this is that the computational amount of hash processing can be reduced and the processing speed can be improved.

[0122] In one embodiment of the present application, the target feature words may be selected based on the word frequency inverse document frequency of each feature word, such as Figure 8 As shown, it may specifically include steps S810 to S820, which are detailed as follows:

[0123] Step S810: Calculate the word frequency inverse file frequency of each feature word in the first feature word set in the sensitive reference text set.

[0124] Term frequency-inverse document frequency, also known as term frequency-inverse document frequency, is the combination of term frequency (TF) and inverse document frequency (IDF). The main idea of ​​the term frequency-inverse document frequency method is that if a word or phrase appears frequently in an article and rarely appears in other articles, it is considered that this word or phrase has good category distinction ability and is suitable for classification. Inverse document frequency mainly means that if there are fewer articles containing a certain word or phrase, the greater the inverse document frequency, which means that the word or phrase has good category distinction ability. Therefore, using the term frequency-inverse document frequency method, the importance of a word or phrase in a certain article can be calculated.

[0125] Specifically in this embodiment, by calculating the word frequency inverse file frequency of each feature word in the first feature word set in the sensitive reference text set, that is, calculating the importance of each feature word to the sensitive reference text set, the target feature word can be further selected according to the importance.

[0126] Step S820: Select a preset number of feature words from the first feature word set as the target feature words in descending order of the frequency of the word frequency inverse file.

[0127] After calculating the word frequency inverse file frequency of each feature word in the sensitive reference text set, a preset number of feature words can be selected from the first feature word set as target feature words in descending order of the word frequency inverse file frequency.

[0128] Among them, the selection method can be to select a preset number of feature words with high ranking as target feature words. The higher the word frequency inverse file frequency of the feature words with high ranking, the higher the importance of the feature words in the sensitive reference text set. Since the first hash signature value can represent the specific content of each text in the sensitive reference text set, the feature words with high importance are used as target feature words, and then the target feature words are hashed to generate the first hash signature value corresponding to each text in the sensitive reference text set. This can make the first hash signature value more consistent with the specific content of each text and more accurate.

[0129] In one embodiment of the present application, Figure 9 As shown, step S810 may specifically include steps S910 to S920, which are described in detail as follows:

[0130] Step S910: Count the reverse file frequencies of each feature word appearing in the sensitive reference text set, and count the frequencies of each feature word appearing in the sensitive reference text set.

[0131] To calculate the inverse file frequency of each feature word in the sensitive reference text set, we first count the inverse file frequency of each feature word in the sensitive reference text set. Specifically, the inverse file frequency of a particular word can be calculated by dividing the total number of documents by the number of documents containing the word, and then taking the logarithm of the resulting quotient. Next, we further count the frequency of each feature word in the sensitive reference text set. The resulting frequency is the number of times each feature word appears divided by the total number of feature words.

[0132] For example, assuming that there are 1,000 texts in the sensitive reference text set, each text in the sensitive reference text set is segmented to obtain the first feature word set corresponding to each text. According to the first feature word set corresponding to each text, it can be counted that the total number of feature words is 10,000. Further assuming that the feature word "Xiao Ming" appears 18 times in the sensitive reference text set, and the feature word "Xiao Ming" appears in 10 texts in the sensitive reference text set, then the frequency of occurrence of the feature word "Xiao Ming" can be counted as TF = 18 / 10000 = 0.0018, and the inverse file frequency of the feature word "Xiao Ming" is IDF = log (1000 / 10) = 2.

[0133] Step S920: The product of the reverse file frequency and the frequency is used as the word frequency reverse file frequency of each feature word in the sensitive reference text set.

[0134] Finally, the product of the calculated inverse document frequency and the frequency can be used as the inverse document frequency of each feature word in the sensitive reference text set. In the example of step S910, the calculated frequency TF = 0.0018 and the inverse document frequency IDF = 2. The inverse document frequency of the feature word "Xiao Ming" can be calculated as TF - IDF = 2 × 0.0018 = 0.0036.

[0135] In one embodiment of the present application, a method for generating a second hash signature value corresponding to the text to be detected may specifically include:

[0136] Segmenting the text to be detected to obtain a second feature word set;

[0137] Performing hash processing on each feature word in the second feature word set to obtain a hash value of each feature word;

[0138] A second hash signature value corresponding to the text to be detected is generated according to the hash value of each feature word and the weight corresponding to each feature word.

[0139] Specifically, since the method for generating the second hash signature value corresponding to the text to be detected in this embodiment is the same as Figure 5 The method for generating the first hash signature value corresponding to the sensitive reference text in the illustrated embodiment is similar and will not be described again here.

[0140] In another embodiment of the present application, the method for generating a second hash signature value corresponding to the text to be detected may further specifically include:

[0141] Segmenting the text to be detected to obtain a second feature word set;

[0142] Performing hash processing on the target feature word in the second feature word set to obtain a hash value of the target feature word;

[0143] A second hash signature value corresponding to the text to be detected is generated according to the hash value of the target feature word and the weight corresponding to each feature word.

[0144] Since the method for generating the second hash signature value corresponding to the text to be detected in this embodiment is the same as Figure 7 The method for generating the first hash signature value corresponding to the sensitive reference text in the illustrated embodiment is similar and will not be described again here.

[0145] Figure 10 A logic diagram of a text detection method according to an embodiment of the present application is shown. Figure 10 As shown, in this embodiment, the text detection method may specifically include the following steps:

[0146] S1. Acquire multiple first texts and multiple second texts, wherein the first texts are texts marked with labels, and the second texts are sensitive texts not marked with labels.

[0147] S2. Extract the label feature vector corresponding to each first text.

[0148] S3. Determine sensitive text and non-sensitive text based on the label feature vector.

[0149] S4. Generate a sensitive reference text set based on the determined sensitive text and multiple second texts.

[0150] S5. Detect similarities between each text in the sensitive reference text set and the text to be detected, and obtain multiple detection results.

[0151] S6. If it is determined based on multiple detection results that there is text in the sensitive reference text set whose similarity with the text to be detected meets the similarity condition, then the text to be detected is determined to be sensitive text.

[0152] The following describes an embodiment of the device of the present application, which can be used to perform the text detection method in the above embodiment of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the text detection method in the above embodiment of the present application.

[0153] Figure 11 A block diagram of a text detection device according to an embodiment of the present application is shown. Figure 11 As shown, a text detection device 1100 according to an embodiment of the present application includes: an acquisition unit 1102 , a first determination unit 1104 , a detection unit 1106 and a second determination unit 1108 .

[0154] Among them, the acquisition unit 1102 is configured to acquire multiple first texts and multiple second texts, the first texts are texts marked with labels, and the second texts are sensitive texts without labels; the first determination unit 1104 is configured to extract the label feature vectors corresponding to the respective first texts based on the labels marked with the respective first texts, and determine the sensitive texts in the multiple first texts according to the label feature vectors; the detection unit 1106 is configured to generate a sensitive reference text set based on the determined sensitive texts and the multiple second texts, and detect the similarity between each text in the sensitive reference text set and the text to be detected to obtain multiple detection results; the second determination unit 1108 is configured to determine that the text to be detected is a sensitive text if it is determined according to the multiple detection results that there is a text in the sensitive reference text set whose similarity with the text to be detected meets the similarity condition.

[0155] In some embodiments of the present application, the first determination unit 1104 includes: an input subunit, configured to input the label feature values ​​corresponding to the respective first texts into a trained text classification model, and obtain the sensitivity probabilities corresponding to the respective first texts output by the text classification model, wherein the sensitive probability is the probability that the first text is a sensitive text, and the text classification model is trained based on a plurality of labeled training texts and text categories corresponding to the respective training texts, and the text categories include sensitive texts and non-sensitive texts; a determination subunit, configured to determine the sensitive texts among the plurality of first texts based on the sensitivity probabilities corresponding to the respective first texts.

[0156] In some embodiments of the present application, the determining subunit is configured to: take the first text among the multiple first texts whose sensitivity probability is greater than a first preset threshold as the sensitive text among the multiple first texts.

[0157] In some embodiments of the present application, the detection unit 1106 is configured to: determine the Hamming distance between each text in the sensitive reference text set and the text to be detected based on the first hash signature value corresponding to each text in the sensitive reference text set and the second hash signature value corresponding to the text to be detected; the second determination unit is configured to: if it is determined based on multiple Hamming distances that there is a text in the sensitive reference text set whose Hamming distance with the text to be detected is less than or equal to a second preset threshold, then determine that the text to be detected is a sensitive text.

[0158] In some embodiments of the present application, the device also includes: a first word segmentation unit, configured to perform word segmentation on each text in the sensitive reference text set to obtain a first feature word set corresponding to each text; a first processing unit, configured to perform hash processing on each feature word in the first feature word set to obtain a hash value of each feature word; and a first generation unit, configured to generate a first hash signature value corresponding to each text in the sensitive reference text set based on the hash value of each feature word and the weight corresponding to each feature word.

[0159] In some embodiments of the present application, the first generation unit is configured to: perform weighted processing on the corresponding hash values ​​according to the weights corresponding to the respective feature words to obtain weighted numerical values ​​corresponding to the respective feature words; merge and accumulate the weighted numerical values ​​corresponding to the respective feature words to obtain an accumulated numerical value, and perform dimensionality reduction processing on the accumulated numerical value to obtain a first hash signature value corresponding to each text in the sensitive reference text set.

[0160] In some embodiments of the present application, the device also includes: a second word segmentation unit, configured to perform word segmentation on each text in the sensitive reference text set to obtain a first feature word set corresponding to each text; a second processing unit, configured to perform hash processing on the target feature word in the first feature word set to obtain a hash value of the target feature word; and a second generation unit, configured to generate a first hash signature value corresponding to each text in the sensitive reference text set based on the hash value of the target feature word and the weight corresponding to the target feature word.

[0161] In some embodiments of the present application, the device further includes: a calculation unit, configured to calculate the word frequency inverse file frequency of each feature word in the first feature word set in the sensitive reference text set; a selection unit, configured to select a preset number of feature words from the first feature word set as the target feature words in the order of the word frequency inverse file frequency from high to low.

[0162] In some embodiments of the present application, the calculation unit is configured to: count the reverse file frequency of each feature word appearing in the sensitive reference text set, and count the frequency of each feature word appearing in the sensitive reference text set; and take the product of the reverse file frequency and the frequency as the word frequency reverse file frequency of each feature word in the sensitive reference text set.

[0163] Figure 12 A schematic diagram of the structure of a computer system suitable for implementing an electronic device according to an embodiment of the present application is shown.

[0164] It should be noted that Figure 12 The computer system 1200 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0165] like Figure 12 As shown, the computer system 1200 includes a central processing unit (CPU) 1201, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1202 or the program loaded from the storage part 1208 to the random access memory (RAM) 1203, such as executing the method described in the above embodiment. Various programs and data required for system operation are also stored in the RAM 1203. The CPU 1201, ROM 1202 and RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0166] The following components are connected to the I / O interface 1205: an input section 1206 including a keyboard, a mouse, and the like; an output section 1207 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 1208 including a hard disk; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the I / O interface 1205 as needed. Removable media 1211, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 1210 as needed, so that computer programs read from the removable media can be installed in the storage section 1208 as needed.

[0167] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1209, and / or installed from a removable medium 1211. When the computer program is executed by the central processing unit (CPU) 1201, the various functions defined in the system of the present application are executed.

[0168] It should be noted that the computer-readable medium shown in the embodiments of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable computer program. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. A computer program embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0169] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. Among them, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, and the above-mentioned module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0170] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. In some cases, the names of these units do not constitute limitations on the units themselves.

[0171] As another aspect, the present application further provides a computer-readable medium, which may be included in the electronic device described in the above embodiments, or may exist independently without being incorporated into the electronic device. The computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device implements the method described in the above embodiments.

[0172] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the application, the features and functions of two or more modules or units described above can be concretized in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0173] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the embodiments of the present application.

[0174] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art that are not disclosed herein.

[0175] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A text detection method, characterized in that: The method comprises: Acquire multiple first texts and multiple second texts, wherein the first texts are texts marked with labels, the labels are labels that mark the sensitivity of the first texts, and the labels are located in the first texts, and the second texts are sensitive texts that are not marked with labels; Extracting, based on the labels annotated with each first text, label feature vectors corresponding to each first text, and determining sensitive texts among the plurality of first texts according to the label feature vectors; Generating a sensitive reference text set based on the determined sensitive text and the plurality of second texts, and detecting similarities between each text in the sensitive reference text set and the text to be detected, to obtain a plurality of detection results; If it is determined based on the multiple detection results that there is text in the sensitive reference text set whose similarity with the text to be detected meets the similarity condition, then the text to be detected is determined to be sensitive text.

2. The method according to claim 1, characterized in that Determining sensitive texts among the plurality of first texts according to the label feature vector includes: Inputting the label feature values ​​corresponding to the respective first texts into a trained text classification model, obtaining a sensitivity probability corresponding to the respective first texts output by the text classification model, wherein the sensitivity probability is a probability that the first text is a sensitive text, the text classification model being trained based on a plurality of labeled training texts and text categories corresponding to the respective training texts, wherein the text categories include sensitive text and non-sensitive text; Determine sensitive texts among the multiple first texts according to the sensitivity probabilities corresponding to the respective first texts.

3. The method according to claim 2, characterized in that Determining sensitive texts among the plurality of first texts according to the sensitivity probabilities corresponding to the respective first texts includes: The first text among the multiple first texts whose sensitivity probability is greater than a first preset threshold is used as the sensitive text among the multiple first texts.

4. The method according to claim 1, wherein Detecting similarities between each text in the sensitive reference text set and the text to be detected to obtain multiple detection results, including: determining a Hamming distance between each text in the sensitive reference text set and the text to be detected based on a first hash signature value corresponding to each text in the sensitive reference text set and a second hash signature value corresponding to the text to be detected; If it is determined based on the multiple detection results that there is text in the sensitive reference text set whose similarity with the text to be detected meets the similarity condition, then the text to be detected is determined to be sensitive text, including: if it is determined based on the multiple Hamming distances that there is text in the sensitive reference text set whose Hamming distance with the text to be detected is less than or equal to a second preset threshold, then the text to be detected is determined to be sensitive text.

5. The method according to claim 4, characterized in that The method further comprises: Performing word segmentation on each text in the sensitive reference text set to obtain a first feature word set corresponding to each text; Performing hash processing on each feature word in the first feature word set to obtain a hash value of each feature word; According to the hash value of each feature word and the weight corresponding to each feature word, a first hash signature value corresponding to each text in the sensitive reference text set is generated.

6. The method according to claim 5, characterized in that Generating a first hash signature value corresponding to each text in the sensitive reference text set according to the hash value of each feature word and the weight corresponding to each feature word includes: Perform weighted processing on the corresponding hash value according to the weight corresponding to each feature word to obtain the weighted value corresponding to each feature word; The weighted values ​​corresponding to the characteristic words are combined and accumulated to obtain an accumulated value, and the accumulated value is subjected to dimensionality reduction processing to obtain a first hash signature value corresponding to each text in the sensitive reference text set.

7. The method according to claim 4, characterized in that The method further comprises: Performing word segmentation on each text in the sensitive reference text set to obtain a first feature word set corresponding to each text; Performing hash processing on the target feature word in the first feature word set to obtain a hash value of the target feature word; According to the hash value of the target feature word and the weight corresponding to the target feature word, a first hash signature value corresponding to each text in the sensitive reference text set is generated.

8. The method according to claim 7, characterized in that The method further comprises: Calculating the word frequency inverse file frequency of each feature word in the first feature word set in the sensitive reference text set; A preset number of feature words are selected from the first feature word set as the target feature words in descending order of the frequency of the word frequency inverse file.

9. The method according to claim 8, characterized in that Calculating the word frequency inverse file frequency of each feature word in the first feature word set in the sensitive reference text set includes: Counting the reverse file frequency of each feature word appearing in the sensitive reference text set, and counting the frequency of each feature word appearing in the sensitive reference text set; The product of the inverse file frequency and the frequency is used as the word frequency inverse file frequency of each feature word in the sensitive reference text set.

10. A text detection device, characterized in that: The device comprises: an acquiring unit configured to acquire a plurality of first texts and a plurality of second texts, wherein the first texts are texts annotated with a label, the annotated label being a label indicating the sensitivity of the first text, the label being located within the first text, and the second texts are sensitive texts that are not annotated with a label; a first determining unit configured to extract, based on a label annotated with each first text, a label feature vector corresponding to each first text, and determine a sensitive text among the plurality of first texts according to the label feature vector; a detection unit configured to generate a sensitive reference text set based on the determined sensitive text and the plurality of second texts, and detect similarities between each text in the sensitive reference text set and the text to be detected, to obtain a plurality of detection results; The second determining unit is configured to determine that the text to be detected is a sensitive text if it is determined based on the multiple detection results that there is text in the sensitive reference text set whose similarity with the text to be detected meets a similarity condition.

11. A computer-readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the text detection method according to any one of claims 1 to 9 is implemented.

12. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the text detection method according to any one of claims 1 to 9.

13. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the text detection method according to any one of claims 1 to 9.

Images