A text detection method, device and equipment

Abstract

The embodiment of the present application discloses a text detection method, device and equipment, which first obtains sensitive judgment items, and determines the sensitive judgment items that can be constituted by sensitive words included in the target text as sensitive judgment items in the target text; then determines the error judgment coefficient and distance decay coefficient of the sensitive judgment items in the target text, and uses the error judgment coefficient, distance decay coefficient and sensitivity level to calculate the sensitivity score of the target text; finally, uses the sensitivity score to determine whether the target text is a sensitive text. Through the pre-defined sensitive judgment items, it is possible to adjust the detection of the target text in a timely manner as the sensitive judgment items change, thereby ensuring the timeliness of the detection of the target text. Moreover, by determining the error judgment coefficient, distance decay coefficient and sensitivity level of the sensitive judgment items in the target text, it is possible to more accurately determine whether the target text is a sensitive text, reduce misjudgment of the target text, and improve the accuracy of the detection of the target text.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to a text detection method, device and equipment. Background Art

[0002] Information texts on the Internet may contain sensitive content that should not be publicly displayed or used at will, and that affects the network environment. In order to ensure the use environment of the Internet, it is necessary to detect texts posted on the Internet to determine whether the text is sensitive text. Detected sensitive texts should be deleted or prevented from being posted to ensure a healthy and civilized network environment.

[0003] At present, when performing sensitive text detection, the detection results may not be accurate enough, and the detection method cannot quickly respond to changes in sensitive content and make corresponding adjustments to the detection. Therefore, how to achieve accurate detection of sensitive text on the basis of easy adjustment is an urgent problem to be solved. Summary of the invention

[0004] In view of this, the embodiments of the present application provide a text detection method, apparatus and device, which can timely adjust the target text detection process according to changes in sensitive content, thereby improving the accuracy of target text detection.

[0005] To solve the above problems, the technical solutions provided in the embodiments of the present application are as follows:

[0006] A text detection method, the method comprising:

[0007] Acquire sensitive determination items, each of which includes at least one sensitive word, and each of which corresponds to a sensitivity level;

[0008] Detect and obtain sensitive words included in the target text;

[0009] When determining the sensitive determination item that can be formed by the sensitive word element included in the target text, determining the formed sensitive determination item as the sensitive determination item included in the target text;

[0010] Determining the error determination coefficient of the sensitive determination item included in the target text;

[0011] Calculating the distance attenuation coefficient of the sensitive judgment item included in the target text;

[0012] Calculate the sensitivity score of the target text according to the error determination coefficient, distance attenuation coefficient and sensitivity level of each sensitive determination item included in the target text;

[0013] If the sensitivity score of the target text is greater than a preset threshold, the target text is determined to be a sensitive text.

[0014] In a possible implementation, the method further includes:

[0015] Adding each of the sensitive word units to an automaton; a path of the automaton includes at least one Chinese character node; the Chinese characters included in a path of the automaton correspond to each of the Chinese characters included in a sensitive word unit in sequence; each path of the automaton corresponds to a different sensitive word unit;

[0016] The detection obtains sensitive words included in the target text, including:

[0017] The target text is input into the automaton. If a first path matching the continuous Chinese characters included in the target text can be obtained in each path of the automaton, the sensitive word element corresponding to the first path is determined as the sensitive word element included in the target text.

[0018] In a possible implementation, the method further includes:

[0019] Adding the Chinese character pinyin string of each Chinese character in each of the sensitive words to a pinyin prefix tree; a path of the pinyin prefix tree includes at least one character node; the character nodes included in a path of the pinyin prefix tree correspond to each pinyin character included in the Chinese character pinyin string in sequence; the Chinese character pinyin string corresponding to each path of the pinyin prefix tree corresponds to one or more homophonic Chinese characters;

[0020] Assigning Chinese character serial numbers to the Chinese characters corresponding to each path of the pinyin prefix tree;

[0021] Adding the Chinese character sequence number to the Chinese character node of the corresponding Chinese character in the automaton;

[0022] The detection obtains sensitive words included in the target text, including:

[0023] Identify non-Chinese character strings in the target text;

[0024] Input the non-Chinese character string into the pinyin prefix tree, and if a second path matching the continuous characters included in the non-Chinese character string can be obtained in each path of the pinyin prefix tree, use the Chinese character sequence number corresponding to the second path as the Chinese character sequence number corresponding to the target text;

[0025] The Chinese character serial number corresponding to the target text is input into the automatic machine. If a third path matching the Chinese character serial number corresponding to the target text can be obtained in each path of the automatic machine, the sensitive word corresponding to the third path is determined as the sensitive word included in the target text.

[0026] In a possible implementation, after detecting and obtaining the sensitive word element included in the target text, the method further includes:

[0027] Acquire a first sensitive word element that is a numeric word element among the sensitive words elements included in the target text;

[0028] When it is determined that the first sensitive word belongs to the number information in the target text, the first sensitive word is discarded.

[0029] In a possible implementation, after detecting and obtaining the sensitive word element included in the target text, the method further includes:

[0030] Acquire a second sensitive word unit that is a single-word word unit from the sensitive word units included in the target text;

[0031] Acquire the segmented words including the second sensitive word in the target text;

[0032] When it is determined that the segmented word including the second sensitive word is a high-frequency misjudged word, the second sensitive word is discarded.

[0033] In a possible implementation, determining the error determination coefficient of the sensitive determination item included in the target text includes:

[0034] Segmenting the target text to obtain a segmentation sequence;

[0035] According to the overlap between the j-th sensitive word element of the target sensitive determination item and the marginal characters of the word segmentation in the word segmentation sequence, the misjudgment coefficient of the j-th sensitive word element of the target sensitive determination item is obtained; the target sensitive determination item is any one of the sensitive determination items included in the target text; the value of j is an integer from 1 to m, and m is the number of sensitive word elements included in the target sensitive determination item;

[0036] The average value of the error determination coefficients of each sensitive word element of the target sensitive determination item is calculated to obtain the error determination coefficient of the target sensitive determination item.

[0037] In a possible implementation, the calculating the distance attenuation coefficient of the sensitive determination item included in the target text includes:

[0038] If the number of sensitive word units included in the target sensitive determination item is 1, the distance attenuation coefficient of the target sensitive determination item is 1; the target sensitive determination item is any one of the sensitive determination items included in the target text;

[0039] If the number of sensitive words included in the target sensitivity determination item is greater than 1, obtain the position distance between any two sensitive words included in the target sensitivity determination item in the target text, and obtain the minimum value of the position distance;

[0040] The distance attenuation coefficient of the target sensitivity determination item is determined according to the minimum value in the position distance, and the distance attenuation coefficient of the target sensitivity determination item is inversely proportional to the minimum value in the position distance.

[0041] In a possible implementation manner, determining the distance attenuation coefficient of the target sensitivity determination item according to the minimum value of the position distances includes:

[0042] The distance attenuation coefficient of the target sensitive determination item is determined according to the minimum value of the position distance, the number of sensitive word elements included in the target sensitive determination item, and the length of each sensitive word element included in the target sensitive determination item.

[0043] In a possible implementation, the method further includes:

[0044] The sensitivity score of the target text is adjusted according to the length of the target text, and the adjusted sensitivity score of the target text is in inverse proportion to the length of the target text.

[0045] A text detection device, comprising:

[0046] A first acquisition unit is used to acquire sensitive determination items, each of which includes at least one sensitive word, and each of which corresponds to a sensitivity level;

[0047] A detection unit, used for detecting sensitive words included in the target text;

[0048] A first determination unit is used to, when determining the sensitive determination item that can be constituted by the sensitive word element included in the target text, determine the constituted sensitive determination item as the sensitive determination item included in the target text;

[0049] A second determination unit is used to determine the error determination coefficient of the sensitive determination item included in the target text;

[0050] A first calculation unit, used to calculate the distance attenuation coefficient of the sensitive judgment item included in the target text;

[0051] A second calculation unit is used to calculate the sensitivity score of the target text according to the error determination coefficient, distance attenuation coefficient and sensitivity level of each sensitive determination item included in the target text;

[0052] The determination unit is used to determine that the target text is a sensitive text if the sensitivity score of the target text is greater than a preset threshold.

[0053] In a possible implementation manner, the device further includes:

[0054] A first adding unit is used to add each of the sensitive word elements to an automaton; a path of the automaton includes at least one Chinese character; the Chinese characters included in a path of the automaton correspond to each of the Chinese characters included in a sensitive word element in sequence; each path of the automaton corresponds to a different sensitive word element;

[0055] The detection unit is specifically used to input the target text into the automaton, and if a first path matching the continuous Chinese characters included in the target text can be obtained in each path of the automaton, the sensitive word element corresponding to the first path is determined as the sensitive word element included in the target text.

[0056] In a possible implementation manner, the device further includes:

[0057] A second adding unit is used to add the Chinese character pinyin string of each Chinese character in each of the sensitive words to the pinyin prefix tree; a path of the pinyin prefix tree includes at least one character node; the character nodes included in a path of the pinyin prefix tree correspond to each pinyin character included in the Chinese character pinyin string in sequence; the Chinese character pinyin string corresponding to each path of the pinyin prefix tree corresponds to one or more homophonic Chinese characters;

[0058] The allocating unit allocates Chinese character serial numbers to the Chinese characters corresponding to each path of the pinyin prefix tree;

[0059] A third adding unit, used for adding the Chinese character sequence number to the Chinese character node of the corresponding Chinese character in the automatic machine;

[0060] The detection unit comprises:

[0061] Recognition subunit, used to recognize non-Chinese character strings in the target text;

[0062] A first input subunit is used to input the non-Chinese character string into the pinyin prefix tree, and if a second path matching the continuous characters included in the non-Chinese character string can be obtained in each path of the pinyin prefix tree, the Chinese character sequence number corresponding to the second path is used as the Chinese character sequence number corresponding to the target text;

[0063] The second input sub-unit is used to input the Chinese character serial number corresponding to the target text into the automatic machine. If a third path matching the Chinese character serial number corresponding to the target text can be obtained in each path of the automatic machine, the sensitive word corresponding to the third path is determined as the sensitive word included in the target text.

[0064] In a possible implementation manner, the device further includes:

[0065] A second acquisition unit is used to acquire a first sensitive word element that is a numeric word element from the sensitive word elements included in the target text;

[0066] The first discarding unit is configured to discard the first sensitive word when it is determined that the first sensitive word belongs to the number information in the target text.

[0067] In a possible implementation manner, the device further includes:

[0068] A third acquisition unit is used to acquire a second sensitive word unit that is a single-word word unit from the sensitive word units included in the target text;

[0069] A fourth acquisition unit, configured to acquire, from the target text, a word segment including the second sensitive word element;

[0070] The second discarding unit is configured to discard the second sensitive word when it is determined that the segmented word including the second sensitive word is a high-frequency misjudged word.

[0071] In a possible implementation manner, the second determining unit includes:

[0072] A word segmentation subunit, used for segmenting the target text to obtain a word segmentation sequence;

[0073] The first determination subunit is used to obtain the error determination coefficient of the j-th sensitive word element of the target sensitive determination item according to the overlap between the j-th sensitive word element of the target sensitive determination item and the marginal characters of the word segmentation in the word segmentation sequence; the target sensitive determination item is any one of the sensitive determination items included in the target text; the value of j is an integer from 1 to m, and m is the number of sensitive word elements included in the target sensitive determination item;

[0074] The calculation subunit is used to calculate the average value of the error determination coefficients of each sensitive word element of the target sensitive determination item to obtain the error determination coefficient of the target sensitive determination item.

[0075] In a possible implementation, the first computing unit includes:

[0076] A second determination subunit is used for, if the number of sensitive word units included in the target sensitive determination item is 1, then the distance attenuation coefficient of the target sensitive determination item is 1; the target sensitive determination item is any one of the sensitive determination items included in the target text;

[0077] an acquisition subunit, configured to acquire, if the number of sensitive words included in the target sensitivity determination item is greater than 1, a position distance between any two sensitive words included in the target sensitivity determination item in the target text, and obtain a minimum value of the position distances;

[0078] The third determination subunit is used to determine the distance attenuation coefficient of the target sensitivity determination item according to the minimum value in the position distance, and the distance attenuation coefficient of the target sensitivity determination item is inversely proportional to the minimum value in the position distance.

[0079] In a possible implementation, the third determination subunit is specifically used to determine the distance attenuation coefficient of the target sensitive judgment item based on the minimum value of the position distance, the number of sensitive words included in the target sensitive judgment item, and the length of each sensitive word included in the target sensitive judgment item.

[0080] In a possible implementation manner, the device further includes:

[0081] The adjustment unit is used to adjust the sensitivity score of the target text according to the length of the target text, and the sensitivity score of the adjusted target text is in inverse proportion to the length of the target text.

[0082] A device for text detection includes a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors to perform the above-mentioned text detection method.

[0083] A computer-readable medium stores instructions which, when executed by one or more processors, enable a device to perform the above-mentioned text detection method.

[0084] It can be seen that the embodiments of the present application have the following beneficial effects:

[0085] The embodiments of the present application provide a text detection method, device and equipment, which first obtains sensitive judgment items, and determines the sensitive judgment items that can be constituted by sensitive words included in the target text as sensitive judgment items in the target text; then determines the error judgment coefficient and distance decay coefficient of the sensitive judgment items in the target text, and uses the error judgment coefficient, distance decay coefficient and sensitivity level to calculate the sensitivity score of the target text; finally, uses the sensitivity score to determine whether the target text is a sensitive text. Through the pre-defined sensitive judgment items, it is possible to adjust the detection of the target text in a timely manner as the sensitive judgment items change, ensure the timeliness of the target text detection, and meet the detection needs of sudden sensitive content. In addition, by determining the error judgment coefficient, distance decay coefficient and sensitivity level of the sensitive judgment items in the target text, it is possible to more accurately determine whether the target text is a sensitive text, reduce misjudgment of the target text, and improve the accuracy of target text detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1A schematic diagram of a framework of an exemplary application scenario of a text detection method provided in an embodiment of the present application;

[0087] Figure 2 A flowchart of a text detection method provided in an embodiment of the present application;

[0088] Figure 3 A schematic diagram of the structure of an automatic machine provided in an embodiment of the present application;

[0089] Figure 4 A schematic diagram of the structure of another automatic machine provided in an embodiment of the present application;

[0090] Figure 5 A schematic diagram of the structure of a pinyin prefix tree provided in an embodiment of the present application;

[0091] Figure 6 A schematic diagram of a function for calculating a distance attenuation coefficient provided in an embodiment of the present application;

[0092] Figure 7 A structural schematic diagram of a text detection device provided in an embodiment of the present application;

[0093] Figure 8 A schematic diagram of the structure of a device for text detection provided in an embodiment of the present application;

[0094] Fig. 9 A schematic diagram of the structure of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0095] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0096] To facilitate understanding of the technical solution provided by this application, the background technology of this application will be described below.

[0097] After studying the traditional text detection method, the inventor found that the current text detection can be achieved by the keyword matching method or based on the text detection model. The detection personnel can pre-set the keywords included in the sensitive text, and determine whether the target text to be detected is a sensitive text through keyword matching. However, the detection effect of the keyword matching method is overly dependent on the setting of the keywords in the target text, and it is easy to ignore the contextual relationship in the target text. In the process of detection, missed detection and false detection may occur, and the detection result of the target text is not accurate enough. The detection method based on the text detection model needs to rely on the data quality of the training data used when training the text detection model. When the sensitive text changes, new training data needs to be constructed, and a large amount of cost is spent to retrain the model. The text detection model cannot be updated quickly, resulting in the inability to detect the target text in time for changes in sensitive content, affecting the accuracy of the target text detection.

[0098] Based on this, the embodiments of the present application provide a text detection method, device and equipment, which first obtains sensitive judgment items, and determines the sensitive judgment items that can be constituted by sensitive words included in the target text as sensitive judgment items in the target text; then determines the error judgment coefficient and distance attenuation coefficient of the sensitive judgment items in the target text, and uses the error judgment coefficient, distance attenuation coefficient and sensitivity level to calculate the sensitivity score of the target text; finally, uses the sensitivity score to determine whether the target text is a sensitive text. Through the pre-defined sensitive judgment items, it is possible to adjust the detection of the target text in a timely manner as the sensitive judgment items change, ensure the timeliness of the target text detection, and meet the detection needs of sudden sensitive content. In addition, by determining the error judgment coefficient, distance attenuation coefficient and sensitivity level of the sensitive judgment items in the target text, it is possible to more accurately determine whether the target text is a sensitive text, reduce misjudgment of the target text, and improve the accuracy of target text detection.

[0099] In order to facilitate understanding of the text detection method provided in the embodiment of the present application, Figure 1 The application scenario of the text detection method provided in the embodiment of the present application is described. Figure 1 A schematic diagram of a framework of an exemplary application scenario of a text detection method provided in an embodiment of the present application. The text detection method provided in an embodiment of the present application can be applied to the server 20.

[0100] In practical applications, the server 20 first obtains predetermined sensitive determination items from the database, each sensitive determination item includes at least one sensitive word element and corresponds to a sensitivity level. The server 20 detects the target text that needs to be detected from the client 10 or other servers 20 to obtain the sensitive word elements included in the target text. The sensitive word elements included in the target text are combined, and the sensitive determination items that can be constituted by the sensitive word elements are determined as the sensitive determination items included in the target text. And determine the error determination coefficient and distance decay coefficient of the sensitive determination items included in the target text. Based on the error determination coefficient, distance decay coefficient and sensitivity level of the sensitive determination items included in the target text, calculate the sensitivity score of the target text. Finally, the sensitivity score of the target text is compared with the threshold value. If the sensitivity score is greater than the threshold value, the target text is determined to be a sensitive text. In addition, the sensitivity score of the target text and the included sensitive determination items can also be sent to the client 10 or other servers 20 to facilitate the determination of sensitive content in the target text.

[0101] Those skilled in the art will understand that Figure 1 The framework diagram shown is only an example in which the embodiments of the present application can be implemented. The scope of application of the embodiments of the present application is not limited by any aspect of the framework.

[0102] It should be noted that the client 10 can be any user device that is currently available, under development, or developed in the future, and can interact with each other through any form of wired and / or wireless connection (e.g., Wi-Fi, LAN, cellular, coaxial cable, etc.), including but not limited to: smart wearable devices, smart phones, non-smart phones, tablet computers, laptop personal computers, desktop personal computers, minicomputers, mid-range computers, mainframe computers, etc. that are currently available, under development, or developed in the future. The implementation of the present application is not limited in this regard. It should also be noted that the server 20 in the embodiment of the present application can be an example of a device that is currently available, under development, or developed in the future and can perform the above operations. The implementation of the present application is not limited in this regard.

[0103] To facilitate understanding of the technical solution provided by the embodiment of the present application, the text detection method provided by the embodiment of the present application will be described below in conjunction with the accompanying drawings.

[0104] See also Figure 2 , which is a flow chart of a text detection method provided by an embodiment of the present application. Figure 2 As shown, the method may include S201-S207:

[0105] S201: Acquire sensitive determination items, each of which includes at least one sensitive word, and each of which corresponds to a sensitivity level.

[0106] A sensitive determination item is text that is composed of sensitive word elements and is predefined for determining sensitive content in text. The sensitive determination item can be adjusted according to the sensitive content to be detected, and the sensitive determination item can be updated with the updated sensitive content to ensure the timeliness of sensitive text detection.

[0107] It should be noted that the sensitive determination item can be pre-stored in the corresponding database. The embodiments of the present application do not limit the specific implementation manner of obtaining the sensitive determination item, and the obtaining of the sensitive determination item can be triggered when detecting the target text.

[0108] Each sensitive determination item includes at least one sensitive word element. Among them, the sensitive word element can be a character or a word that is pre-determined to represent sensitive content. If the sensitive determination item is composed of multiple sensitive word elements, in order to facilitate the determination of each sensitive word element, the sensitive word elements can be divided by a delimiter. For example, a sensitive determination item is composed of two sensitive word elements, "sky" and "Chengdu", and this sensitive determination item can be expressed as "sky|Chengdu".

[0109] Each sensitive determination item also has a corresponding sensitive level. The sensitive level is used to determine the sensitivity of the sensitive determination item. For example, the sensitive levels can be set as the first level, the second level, and the third level. Among them, the third level can indicate that the sensitivity of this sensitive determination item is the highest, and the first level can indicate that the sensitivity of this sensitive determination item is the lowest.

[0110] In a possible implementation manner, the sensitive determination item can be a line of text in a pre-established sensitive word list. Each line in the sensitive word list has one or more sensitive word elements, and the sensitive word elements in this line can form a sensitive determination item. Each line in the sensitive word list also has a corresponding sensitive level, which is used to determine the sensitive level corresponding to the sensitive determination item formed by the sensitive word elements in this line.

[0111] S202: Detect the sensitive word elements included in the target text.

[0112] The target text is the text to be detected, and the target text may contain sensitive content. The embodiments of the present application do not limit the length of the target text. The target text can be the text corresponding to a sentence to be detected, or the text corresponding to an article to be detected.

[0113] To improve the detection accuracy of the target text, before detecting the sensitive word elements in the target text, the target text can be normalized to obtain a more standard target text. The normalization operation can specifically include removing invalid symbols, converting traditional Chinese to simplified Chinese, and converting uppercase to lowercase, etc. Among them, invalid symbols refer to symbols that do not have the meaning of specific punctuation marks. For example, the "&" in "sky&space" only serves to separate "sky" and does not have the specific meaning of the corresponding punctuation mark. By normalizing the target text, the diverse settings for sensitive determination items can be reduced, and the management and configuration cost of sensitive determination items can be lowered.

[0114] The target text may include sensitive word elements. Detect the target text to determine the sensitive word elements included in the target text.

[0115] If the target text does not have sensitive word elements, it means that the target text does not contain sensitive content, and the target text is a safe text.

[0116] In a possible implementation manner, the target text can be input into an automaton to implement the detection of sensitive word elements. Correspondingly, the embodiments of the present application provide an implementation manner of S202, that is, the specific implementation manner of detecting the sensitive word elements included in the target text. For details, please refer to the following.

[0117] In addition, when determining sensitive text through Chinese character detection, for the situation where some target texts have non-Chinese character strings, in another possible implementation manner: the target text can be first input into a pinyin prefix tree to convert the non-Chinese character string into the corresponding Chinese characters, and then the converted text is input into the automaton to implement the detection of sensitive word elements. Correspondingly, the embodiments of the present application provide another implementation manner of S202, that is, the specific implementation manner of detecting the sensitive word elements included in the target text. For details, please refer to the following.

[0118] S203: When it is determined that the sensitive word elements included in the target text can form a sensitive determination item, the formed sensitive determination item is determined as the sensitive determination item included in the target text.

[0119] After detecting the sensitive word elements included in the target text, further determine whether the obtained sensitive word elements can form a sensitive determination item. If one or more sensitive word elements included in the target text can form a sensitive determination item, it means that the target text has sensitive content. Then, determine the sensitive determination item included in the target text according to the formed sensitive determination item. If the obtained sensitive word elements cannot form a sensitive determination item, it means that the target text does not have a complete sensitive determination item and does not constitute sensitive content.

[0120] Specifically, it is possible to determine whether each detected sensitive word element can form a sensitive determination item one by one. If it can form a sensitive determination item, the sensitive determination item included in the target text is determined; if the sensitive word element cannot form a sensitive determination item, the sensitive word element is discarded.

[0121] It should be noted that in some possible cases, in order to avoid the detection of sensitive text, the sensitive word elements of the sensitive determination items in some target texts are not continuous. Based on this situation, the embodiments of the present application do not limit the positions and orders of the sensitive word elements that form the same sensitive determination item in the target text. For example, the sensitive determination item is "sky|Chengdu", and the target text is "The sky in Chengdu is very beautiful". The target text includes two sensitive word elements, "Chengdu" and "sky", which can form the sensitive determination item of "sky|Chengdu".

[0122] S204: Determine the misjudgment coefficient of the sensitive determination item included in the target text.

[0123] It can be understood that there may be errors in determining the sensitive determination item included in the target text through the detected sensitive word elements. For example, a detected sensitive word element composed of two characters may be composed of the ending character of a word and the starting character of a word, and they are only adjacent in position, which does not represent the sensitive meaning of the sensitive word element itself, and there is a possibility of misjudgment. For example, if "tiantian" is a sensitive word element, and the target text is "The weather is very good today", detecting the target text can obtain "tiantian", but "tiantian" is actually composed of the ending character of "today" and the starting character of "weather", and does not have the meaning of the sensitive word element "tiantian".

[0124] To ensure the accuracy of the detection of the target text, based on the sensitive determination item included in the target text determined in step 203, the corresponding misjudgment coefficient is determined. The misjudgment coefficient is used to measure the possibility that the sensitive determination item included in the target text is incorrect. Based on the misjudgment coefficient, a more accurate sensitive score of the target text can be obtained.

[0125] In a possible implementation manner, the misjudgment coefficient can be determined by the coincidence between the sensitive word elements included in the target text and the word segmentation of the target text. The embodiments of the present application provide an S204, that is, a specific implementation manner for determining the misjudgment coefficient of the sensitive determination item included in the target text. Please refer to the following text.

[0126] S205: Calculate the distance attenuation coefficient of the sensitive determination item included in the target text.

[0127] In addition, if the sensitive judgment item is composed of multiple sensitive words, the distance between the sensitive words will also affect the possibility that the target text is a sensitive text. For example, when the distance between multiple sensitive words is long, there is a possibility that the sensitive words are not related. Correspondingly, the possibility that the target text is a sensitive text will be reduced.

[0128] According to the sensitive judgment items included in the target text determined in step S203, the corresponding distance decay coefficient is calculated. Based on the distance decay coefficient, a more accurate sensitivity score of the target text can be obtained.

[0129] In a possible implementation, the embodiment of the present application provides a specific implementation of S205, namely, calculating the distance attenuation coefficient of the sensitive judgment item included in the target text, please refer to the following.

[0130] S206: Calculate the sensitivity score of the target text according to the false positive coefficient, distance attenuation coefficient and sensitivity level of each sensitive judgment item included in the target text.

[0131] The error determination coefficient, distance decay coefficient and sensitivity level of each sensitive determination item included in the target text will affect the determination of whether the target text is a sensitive text. Based on the error determination coefficient, distance decay coefficient and sensitivity level of each sensitive determination item, the sensitivity score of the target text is calculated. The sensitivity score of the target text can be used to determine whether the target text is a sensitive text.

[0132] In a possible implementation, the sensitivity score of the target text may be calculated using formula (1).

[0133]

[0134] Where i represents the i-th sensitive judgment item in the target text, 1≤i≤n, i is a positive integer, and n is the total number of sensitive judgment items in the target text. s represents the sensitivity score of the target text. m i Represents the sensitivity level of the i-th sensitive judgment item. i Represents the error determination coefficient of the i-th sensitive determination item. i Represents the distance attenuation coefficient of the i-th sensitive judgment item.

[0135] S207: If the sensitivity score of the target text is greater than a preset threshold, the target text is determined to be a sensitive text.

[0136] The threshold is a predetermined value used to determine whether the target text is sensitive text. When the sensitivity score of the target text is greater than the threshold, the target text is determined to be sensitive text. When the sensitivity score of the target text is less than or equal to the threshold, it can be determined that the target text is not sensitive text and is safe text.

[0137] The embodiment of the present application does not limit the specific value of the threshold, and the threshold can be adjusted accordingly according to the detection situation of sensitive text.

[0138] Based on the above S201-S207, it can be known that when the sensitive content changes, the predefined sensitive judgment items can be adjusted in time, and the sensitive judgment items included in the target text can be determined through the adjusted sensitive judgment items, which improves the timeliness of the adjustment of the target text detection and can meet the detection needs of the changes in sensitive content. The sensitive score is calculated by the error determination coefficient, distance attenuation coefficient and sensitivity level of the sensitive judgment items included in the target text, and a more accurate sensitive score can be obtained based on the contextual content of the target text and the sensitivity of the sensitive judgment items, thereby improving the accuracy and recall rate of sensitive text detection.

[0139] The target text to be detected may be composed of Chinese characters. For such target text, an automatic machine can be established to detect sensitive words.

[0140] In a possible implementation, the text detection method further includes: constructing a sensitive word unit detection automaton, specifically including:

[0141] The sensitive word units included in the target text obtained in step S202 are added to the automaton. A path of the automaton includes at least one Chinese character node. The Chinese characters included in a path of the automaton correspond to the Chinese characters included in a sensitive word unit in sequence. Each path of the automaton corresponds to a different sensitive word unit.

[0142] It can be seen that the automaton in the embodiment of the present application is constructed based on various sensitive word units and is used to detect sensitive word units in the target text composed of Chinese characters. Figure 3 As shown in FIG. 1 , this figure is a schematic diagram of the structure of an automaton provided in an embodiment of the present application. The automaton is constructed based on three sensitive words: "weather", "sky" and "the day before yesterday".

[0143] The automaton may include multiple paths, each path including at least one Chinese character. The Chinese characters included in each path correspond to the Chinese characters included in a sensitive word in order. For example, Figure 3 As shown in the figure, in the order starting from the root node, the Chinese characters on each path form three sensitive words: "weather", "sky" and "the day before yesterday". Each path in the automaton corresponds to a sensitive word, and each path corresponds to a different sensitive word.

[0144] In a possible implementation, the Chinese character nodes in the automaton can be represented by the Chinese character serial numbers corresponding to the Chinese characters. For example, if "天" corresponds to "01", "气" corresponds to "02", "空" corresponds to "04", and "前" corresponds to "03". The automaton obtained by adding the three sensitive word elements "天气", "天空", and "前天" to the automaton is as Figure 4 shown. Figure 4 It is a schematic structural diagram of another automaton provided by an embodiment of the present application. By using Chinese character serial numbers to represent Chinese characters, the specific Chinese characters corresponding to the Chinese character nodes included in the automaton can be kept confidential, ensuring the security of sensitive word elements and avoiding the problem of maliciously obtaining sensitive word elements through the automaton and circumventing sensitive content.

[0145] Each path of the automaton also corresponds to the sensitive determination item to which the sensitive word element belongs and the sensitive level corresponding to each sensitive determination item. Through each path of the automaton, the sensitive word element and the relevant information of the sensitive determination item to which the sensitive word element belongs can be determined.

[0146] Based on the sensitive word element detection automaton constructed above, an embodiment of the present application provides an S202, that is, a specific implementation method for detecting the sensitive word elements included in the target text, including:

[0147] Input the target text into the automaton. If a first path that matches the consecutive Chinese characters included in the target text can be obtained in each path of the automaton, determine the sensitive word element corresponding to the first path as the sensitive word element included in the target text.

[0148] The target text is composed of various characters. When the target text is input into the automaton, the automaton can sequentially match each Chinese character in the target text with the Chinese character nodes in each path according to the order in the target text. If there is a first path in the multiple paths included in the automaton that can match the consecutive Chinese characters included in the target text, the sensitive word element corresponding to the first path can be determined as the sensitive word element included in the target text.

[0149] Taking the above-mentioned automaton as an example, if the target text is "The weather was nice before", then each Chinese character in the target text is matched with the Chinese character nodes in each path of the automaton in sequence ({之 / 前 / 的 / 天 / 气 / 很 / 好}). "之" has no matching node, "前" matches the first node in the path where "前天" is located in the automaton, "的" has no matching node, "天" matches the first node in the paths where "天气" and "天空" are located in the automaton, "气" matches the second node in the path where "天气" is located in the automaton, and neither "很" nor "好" has a matching node. It can be seen that the path where "天气" is located in the automaton can match the consecutive Chinese characters "天气" in the target text, and the path where "天气" is located is taken as the first path. And the sensitive word element "天气" corresponding to the path where "天气" is located is taken as the sensitive word element included in the target text.

[0150] It should be noted that if the Chinese character nodes of the automaton are represented by the Chinese character numbers corresponding to the Chinese character characters, when matching the Chinese characters in the target text, the conversion between Chinese characters and the corresponding Chinese character numbers can be automatically performed first, and the converted Chinese character numbers are used to match the paths in the automaton. And the Chinese character numbers corresponding to the Chinese character nodes in the automaton can be obtained according to the Chinese pinyin string of the Chinese character corresponding to the Chinese character node. For the specific implementation, reference can be made to the description of the subsequent embodiments.

[0151] In the embodiments of the present application, through the automaton constructed according to the sensitive word elements, the detection of sensitive word elements in the target text composed of Chinese character characters can be performed. The automaton can be correspondingly adjusted based on the change of the sensitive word elements in the sensitive determination items, so as to adapt to the update of sensitive content and improve the accuracy and recall rate of target text detection.

[0152] It can be understood that in order to avoid the detection of sensitive content in the target text, some Chinese character strings may be replaced by pinyin strings in the target text.

[0153] In view of the above situation, the embodiments of the present application also provide a text detection method. In addition to the above steps, the method further includes the following three steps A1-A3:

[0154] A1: Add the Chinese pinyin string of each Chinese character in each sensitive word element to the pinyin prefix tree; a path of the pinyin prefix tree includes at least one character node; the character nodes included in a path of the pinyin prefix tree correspond to each pinyin character included in a Chinese pinyin string in sequence; the Chinese pinyin string corresponding to each path of the pinyin prefix tree corresponds to one or more homophonic Chinese characters.

[0155] According to the Chinese pinyin strings corresponding to each Chinese character in each sensitive word element, a pinyin prefix tree is established. The pinyin prefix tree may include multiple paths, and each path includes at least one character node, and each character node corresponds to a pinyin character. Refer to Figure 5 As shown, this figure is a schematic structural diagram of a pinyin prefix tree provided by an embodiment of the present application. The pinyin prefix tree is constructed according to the Chinese pinyin strings corresponding to each Chinese character in the three sensitive word elements of "weather", "add", and "the day before yesterday".

[0156] A2: Assign Chinese character numbers to the Chinese characters corresponding to each path of the pinyin prefix tree.

[0157] Each path of the pinyin prefix tree may correspond to one or more homophonic Chinese characters. In order to distinguish different Chinese characters and facilitate the subsequent recognition of sensitive word elements by the automaton, corresponding Chinese character numbers are assigned to the Chinese characters corresponding to each path.

[0158] Taking the above pinyin prefix tree as an example, "01" can be assigned to "tian", "02" to "qi", "03" to "qian", "04" to "tian", and "05" to "jia".

[0159] A3: Add the Chinese character numbers to the Chinese character nodes of the corresponding Chinese characters in the automaton.

[0160] Add the assigned Chinese character numbers to the Chinese character nodes in the automaton. Each Chinese character node corresponds to a Chinese character and its Chinese character number. Through the Chinese character numbers, the Chinese character nodes corresponding to the Chinese pinyin string can be conveniently determined in the automaton.

[0161] Based on the pinyin prefix tree established by using the Chinese pinyin strings corresponding to the Chinese characters in the sensitive word elements, another S202 is provided in an embodiment of the present application, that is, a specific implementation manner for detecting the sensitive word elements included in the target text, including the following three steps B1 - B3:

[0162] B1: Identify non-Chinese character strings in the target text.

[0163] Some target texts may have non-Chinese character strings composed of pinyin characters. Identify the non-Chinese character strings in the target text to obtain the non-Chinese character strings in the target text.

[0164] B2: Input the non-Chinese character string into the pinyin prefix tree. If a second path that matches the consecutive characters included in the non-Chinese character string can be obtained in each path of the pinyin prefix tree, use the Chinese character number corresponding to the second path as the Chinese character number corresponding to the target text.

[0165] Input the identified non-Chinese character string into the pinyin prefix tree to identify the Chinese characters in the possible sensitive word elements corresponding to the non-Chinese character string.

[0166] The pinyin prefix tree can match each character in the non-Chinese character string with the character nodes in each path of the pinyin prefix tree in sequence according to the character order in the non-Chinese character string. If there is a second path that matches the continuous characters included in the non-Chinese character string, the Chinese character sequence number corresponding to the second path can be used as the Chinese character sequence number corresponding to the non-Chinese character string in the target text.

[0167] Taking the above-mentioned Pinyin prefix tree as an example, if the target text is "tianqi", the non-Chinese character string in the target text is first identified to obtain "tianqi". "tianqi" is input into the Pinyin prefix tree, and the Pinyin prefix tree uses each character in "tianqi" to match the character nodes in the path. "t", "i", "a" and "n" are matched in sequence with the character nodes in the path where "tian" of the Pinyin prefix tree is located. The path where "tian" is located can be used as the second path, and the corresponding "01" can be used as the Chinese character sequence number corresponding to the non-Chinese character string. "q" and "i" are matched in sequence with the character nodes in the path where "qi" of the Pinyin prefix tree is located. The path where "qi" is located can be used as the second path, and the corresponding "02" can be used as the Chinese character sequence number corresponding to the non-Chinese character string. Finally, the two Chinese character sequence numbers "01" and "02" corresponding to the target text are obtained.

[0168] B3: Input the Chinese character serial number corresponding to the target text into the automaton. If a third path matching the Chinese character serial number corresponding to the target text can be obtained in each path of the automaton, the sensitive word corresponding to the third path is determined as the sensitive word included in the target text.

[0169] After converting the non-Chinese character string into the corresponding Chinese character serial number, the Chinese character serial number corresponding to the target text is input into the automaton, and the automaton is used to determine the sensitive words included in the target text.

[0170] The automaton matches each path using the Chinese character sequence number corresponding to the input target text. If there is a third path that matches the Chinese character sequence number corresponding to the target text, the sensitive word corresponding to the third path is determined as the sensitive word included in the target text.

[0171] by Figure 4 Taking the automaton shown as an example, the Chinese character serial numbers corresponding to the target text are "01" and "02". The automaton matches the Chinese character serial numbers corresponding to the target text "01" and "02" with the Chinese character serial numbers corresponding to the Chinese characters in each path. "01" and "02" can match the path corresponding to "0102" of the automaton, and the path corresponding to "0102" is used as the third path, and the sensitive word "weather" corresponding to the third path is used as the sensitive word included in the target text.

[0172] Based on the above content, it can be seen that by establishing a pinyin prefix tree, the conversion between pinyin characters and Chinese character serial numbers can be realized, thereby realizing the detection of sensitive words corresponding to non-Chinese character strings, supporting the detection of multiple types of sensitive content, and improving the accuracy of target text detection.

[0173] In some possible cases, sensitive determination items are composed of numeric words. However, for numbers composed of multiple digits, such as telephone numbers and express numbers, there may be a numeric string that is consistent with the numeric words. This can easily lead to incorrect determination of numeric sensitive words in the number, affecting the accuracy of target text detection.

[0174] In view of the above problem, in a possible implementation, the embodiment of the present application provides a text detection method. In addition to the above steps, in S202, that is, after detecting and obtaining the sensitive word element included in the target text, the method further includes:

[0175] Obtaining the first sensitive word that is a numeric word among the sensitive words included in the target text;

[0176] When it is determined that the first sensitive word belongs to the number information in the target text, the first sensitive word is discarded.

[0177] The digital word is a sensitive word in digital form, which may be in Arabic numeral form. After the sensitive word included in the target text is detected, the sensitive word that is a digital word is obtained and used as the first sensitive word.

[0178] The embodiment of the present application does not limit the specific implementation method of obtaining the first sensitive word element. In a possible implementation method, the first sensitive word element can be obtained by matching the sensitive word element in the target text with a predetermined numeric character.

[0179] The target text may contain number information, where the number information is information corresponding to a numeric string corresponding to a specific coded meaning in the target text. For example, if the target text contains a numeric string of an order number, the order number can be used as the number information.

[0180] It is determined whether the number information includes the first sensitive word. If the first sensitive word belongs to the number information, it can be determined that the first sensitive word is used to form the number information and does not represent sensitive content. The first sensitive word is discarded and is not used to determine subsequent sensitive judgment items.

[0181] For example, the first sensitive word "01" is a telephone number "010-88888888" in the target text, and the first sensitive word "01" belongs to the number information "010-88888888", so the first sensitive word "01" is discarded.

[0182] In the embodiment of the present application, by determining whether the first sensitive word belongs to number information, it is possible to determine whether the detected digital word is a sensitive word, thereby preventing the digital character string from being misjudged as a sensitive word and improving the accuracy of the target text detection.

[0183] For a single character, it is easy to be determined as a sensitive word unit of the corresponding single-word word unit when it is used normally with other characters to form a vocabulary.

[0184] In view of such a situation, in a possible implementation, the embodiment of the present application provides a text detection method. In addition to the above steps, in S202, that is, after detecting and obtaining the sensitive words included in the target text, the method further includes the following three steps C1-C3:

[0185] C1: Obtain the second sensitive word that is a single-word word among the sensitive words included in the target text.

[0186] A single-word word is a sensitive word consisting of a single Chinese character. After the sensitive word included in the target text is detected, the sensitive word that is a single-word word is obtained and used as the second sensitive word.

[0187] C2: Obtain the word segmentation including the second sensitive word in the target text.

[0188] In order to determine the contextual relationship of the characters corresponding to the second sensitive word, the target text may be segmented, and the segmented words including the second sensitive word are obtained from the segmented words included in the target text.

[0189] C3: When it is determined that the segmented word including the second sensitive word is a high-frequency misjudged word, the second sensitive word is discarded.

[0190] Frequently misjudged words are predetermined, including single-word words in sensitive word units, but not words involving sensitive content. Frequently misjudged words can be determined by the detection results of historical target texts, and can be updated accordingly with the detection results of the target texts.

[0191] It is judged whether the acquired segmented word including the second sensitive word is a high-frequency misjudged word. If the segmented word including the second sensitive word is a high-frequency misjudged word, it means that the second sensitive word may be misjudged, and the second sensitive word is discarded.

[0192] Take "The weather is very good today" as the target text. If the detected sensitive words include "weather" and "very", and "very" is a single word, "very" is used as the second sensitive word. Segment the target text to get "today", "weather" and "very good". "Very good" is the word segmentation that includes the second sensitive word. If "very good" is a high-frequency misjudgment word, it means that "very" is used as a word with "good" and is not a sensitive content when it represents a single word. "Very" is discarded and not used as a sensitive word.

[0193] Based on the above content, it can be known that by determining whether the second sensitive word is a high-frequency misjudged word, it is possible to determine whether the detected single-character word is a sensitive word, thereby preventing the single characters in the high-frequency misjudged words from being misjudged as sensitive words, thereby improving the accuracy of target text detection.

[0194] It is understandable that when determining the sensitive judgment items included in the target text through the detected sensitive word elements, the detected sensitive word elements may be composed of marginal characters of the vocabulary. Such sensitive word elements may not represent sensitive content in the target text, and there is a possibility of misjudgment.

[0195] In a possible implementation, the error determination coefficient of the sensitive determination item can be determined according to the overlap between each sensitive word in the sensitive determination item and the marginal vocabulary of the word segmentation in the target text. Specifically, the embodiment of the present application provides a specific implementation of S204, i.e., determining the error determination coefficient of the sensitive determination item included in the target text, including the following three steps D1-D3:

[0196] D1: Segment the target text to obtain a segmentation sequence.

[0197] The embodiment of the present application does not limit the specific implementation method of segmenting the target text. The target text can be segmented by a segmentation tool to obtain a segmentation sequence after segmentation. The segmentation sequence can be composed of segmentations and separators in the target text, and each segmentation is divided by a separator.

[0198] D2: According to the overlap between the j-th sensitive word element of the target sensitive decision item and the marginal characters of the word segmentation in the word segmentation sequence, the misjudgment coefficient of the j-th sensitive word element of the target sensitive decision item is obtained; the target sensitive decision item is any one of the sensitive decision items included in the target text; the value of j is an integer from 1 to m, and m is the number of sensitive words included in the target sensitive decision item.

[0199] The target text may include one or more sensitive judgment items, and each sensitive judgment item is composed of one or more sensitive word units. The overlap between each sensitive word unit in each sensitive judgment item and the word segmentation sequence is determined respectively, and then the error determination coefficient of each sensitive word unit in each sensitive judgment item is determined.

[0200] Any sensitive judgment item included in the target text is taken as the target sensitive judgment item. Then determine the overlap between the j-th sensitive word element in the target sensitive judgment item and the marginal characters of the word segmentation in the word segmentation sequence in the target text. Wherein, 1≤j≤m, and j is a positive integer. m is the total number of sensitive word elements included in the target sensitive judgment item. The marginal characters of a word segmentation are the starting characters or the ending characters of the word segmentation. If the marginal characters of a word segmentation are the same as the characters in the j-th sensitive word element, then the j-th sensitive word element overlaps with the marginal characters of the word segmentation.

[0201] Specifically, according to the specific type of the edge characters that are the same as the j-th sensitive word, the overlap can be divided into the following three cases:

[0202] Case 1: The starting character of the jth sensitive word coincides with the ending character of the word segmentation, which can correspond to the left overlap case;

[0203] Case 2: The end character of the jth sensitive word coincides with the start character of the word segmentation, which corresponds to the right overlap case;

[0204] Case 3: The j-th sensitive word overlaps with the ending character and the starting character of the adjacent word segmentation, which can correspond to the left-right overlap situation.

[0205] For different overlap situations, the misjudgment coefficient of the jth sensitive word can be determined. The misjudgment coefficient can be used to indicate the possibility of a sensitive word being misjudged. The value of the misjudgment coefficient can be between 0 and 1. The smaller the value, the greater the possibility of a corresponding misjudgment, indicating that it is a sensitive word. In a possible implementation, if there is no overlap, the misjudgment coefficient of the corresponding jth sensitive word can be set to 1.

[0206] Taking the above three situations as examples, the left-right overlap situation is more likely to have a possibility of misjudgment than the left-right overlap situation or the right-right overlap situation. The misjudgment coefficient of the j-th sensitive word corresponding to the left-right overlap situation can be smaller than the misjudgment coefficient of the j-th sensitive word corresponding to the left-right overlap situation or the right-right overlap situation. For example, if the misjudgment coefficient of the j-th sensitive word corresponding to the left-right overlap situation or the right-right overlap situation is 0.8, the misjudgment coefficient of the j-th sensitive word corresponding to the left-right overlap situation can be 0.2.

[0207] D3: Calculate the average value of the error determination coefficients of each sensitive word of the target sensitive determination item to obtain the error determination coefficient of the target sensitive determination item.

[0208] After obtaining the error determination coefficient of each sensitive word in the target sensitive determination item, the average value of the error determination coefficient of each sensitive word in the target sensitive determination item is calculated to obtain the error determination coefficient of the target sensitive determination item.

[0209] Specifically, the error determination coefficient of the target sensitive determination item can be calculated by formula (2).

[0210]

[0211] Among them, z i represents the misjudgment coefficient of the i-th sensitive judgment item, represents the misjudgment coefficient of the jth sensitive word of the i-th sensitive judgment item, count(w i ) represents the number of sensitive words in the i-th sensitive judgment item.

[0212] In the embodiment of the present application, the error determination coefficient of the sensitive word element is determined by the overlap of each sensitive word element in each sensitive determination item with the marginal characters of the word segmentation in the word segmentation sequence, and then the error determination coefficient of the sensitive determination item is determined. In this way, the error determination coefficient is used to measure the error determination, so that the sensitivity score calculated by the error determination coefficient is more accurate.

[0213] There may be a certain position distance between sensitive words in the sensitive judgment item. When the position distance is large, the correlation between sensitive words may be weak, and the impact of the determined sensitive judgment item on the detection of the target text should be small. Correspondingly, when the position distance is small, the correlation between sensitive words may be strong, and the impact of the determined sensitive judgment item on the detection of the target text should be large.

[0214] In a possible implementation, the distance decay coefficient of the sensitive judgment item can be determined by the positional relationship between sensitive word elements. The present application embodiment provides a specific implementation of S205, that is, calculating the distance decay coefficient of the sensitive judgment item included in the target text, which specifically includes the following three steps E1-E3:

[0215] E1: If the number of sensitive words included in the target sensitive judgment item is 1, the distance attenuation coefficient of the target sensitive judgment item is 1; the target sensitive judgment item is any one of the sensitive judgment items included in the target text.

[0216] Any one of the sensitive judgment items included in the target text is used as the target sensitive judgment item. When the number of sensitive words included in the target sensitive judgment item is 1, there is no position distance between sensitive words, and the distance attenuation coefficient of the target sensitive judgment item is 1.

[0217] E2: If the number of sensitive words included in the target sensitive judgment item is greater than 1, obtain the position distance between any two sensitive words included in the target sensitive judgment item in the target text, and obtain the minimum value of the position distance.

[0218] When the number of sensitive words included in the target sensitive judgment item is greater than 1, there is a position distance between each sensitive word. Obtain the position distance between any two sensitive words included in the target sensitive judgment item in the target text. Among them, the position distance can be determined by the number of characters between the two sensitive words. For example, in the target text "The weather is very good today", the position distance between the sensitive word "today" and the sensitive word "very good" is 2. When obtaining the position distance between any two sensitive words, multiple position distances may be obtained. Among them, it may include different position distances between the same two sensitive words, and / or position distances between two different sensitive words, and the minimum value is selected from the obtained position distances.

[0219] E3: Determine the distance attenuation coefficient of the target sensitivity determination item according to the minimum value in the position distance. The distance attenuation coefficient of the target sensitivity determination item is inversely proportional to the minimum value in the position distance.

[0220] The minimum value in the position distance is used to determine the distance attenuation coefficient of the target sensitive judgment item. It should be noted that the position distance and the degree of influence of the sensitive judgment item on the determination of sensitive text are inversely proportional. If the position distance is smaller, the degree of influence of the target sensitive judgment item on the determination of sensitive text is higher; conversely, the larger the position distance, the lower the degree of influence of the target sensitive judgment item on the determination of sensitive text. Correspondingly, the distance attenuation coefficient of the target sensitive judgment item is inversely proportional to the minimum value in the position distance. If the position distance is smaller, the corresponding distance attenuation coefficient is larger; if the position distance is larger, the corresponding distance attenuation coefficient is smaller.

[0221] The embodiment of the present application also provides a kind of E3, namely the specific implementation method of determining the distance attenuation coefficient of the target sensitivity judgment item according to the minimum value in the position distance, please refer to the following.

[0222] In the embodiment of the present application, the distance decay coefficient determined by the position distance between sensitive words in the target sensitive determination item can be used to measure the influence of the sensitive determination item on the determination of sensitive text, so as to achieve a more accurate sensitivity score of the target text calculated by the distance decay coefficient.

[0223] Furthermore, the embodiment of the present application also provides a specific implementation method E3, that is, determining the distance attenuation coefficient of the target sensitivity determination item according to the minimum value in the position distance, including:

[0224] The distance attenuation coefficient of the target sensitive determination item is determined according to the minimum value in the position distance, the number of sensitive word elements included in the target sensitive determination item, and the length of each sensitive word element included in the target sensitive determination item.

[0225] In addition to the position distance, the number of sensitive words included in the target sensitive judgment item and the length of each sensitive word will affect the degree of influence of the target sensitive judgment item on the determination of sensitive text. Among them, the number of sensitive words will affect the number of position distances between sensitive words. The length of sensitive words will affect the accuracy of the positioning of sensitive words in the target text.

[0226] The distance decay coefficient can be calculated using the minimum value of the position distance, the number of sensitive words in the target sensitive judgment item, and the length of the sensitive words.

[0227] Specifically, the distance attenuation coefficient can be calculated by formula (3), and the curve corresponding to formula (3) is as follows: Figure 6 As shown, this figure is a schematic diagram of a function for calculating the distance attenuation coefficient provided in an embodiment of the present application.

[0228]

[0229] Among them, r i represents the distance attenuation coefficient of the i-th sensitive judgment item, represents the position distance between the jth sensitive word and the kth sensitive word in the i-th sensitive judgment item, Indicates the minimum value of the position distance, count(w i ) represents the number of sensitive words contained in the i-th sensitive judgment item, Indicates the length of the jth sensitive word in the i-th sensitive judgment item; represents the jth sensitive word in the i-th sensitive judgment item; Represents the kth sensitive word in the i-th sensitive judgment item.

[0230] Based on the above content, it can be seen that the distance decay coefficient of the sensitive judgment item calculated by the minimum value in the position distance, the number of sensitive words included in the target sensitive judgment item, and the length of each sensitive word included in the target sensitive judgment item is more accurate. The sensitivity score of the target text calculated based on the distance decay coefficient is more accurate, thereby improving the accuracy of detecting the target text.

[0231] The length of the target text will also affect the sensitivity score of the target text. It is understandable that the longer the target text is, the higher the possibility that the target text includes sensitive judgment items.

[0232] In view of the influence of the length of the target text on the sensitivity score, the embodiment of the present application further provides a text detection method. In addition to the above steps, the method further includes:

[0233] The sensitivity score of the target text is adjusted according to the length of the target text, and the sensitivity score of the adjusted target text is in inverse proportion to the length of the target text.

[0234] The sensitivity score of the target text is adjusted by using the length of the target text, so that the sensitivity score of the target text is inversely proportional to the length of the target text, thereby reducing the influence of the length of the target text on the sensitivity score. The length of the target text can be represented by the number of characters included in the target text.

[0235] Specifically, the sensitivity score can be adjusted by formula (4).

[0236]

[0237] Among them, f represents the sensitivity score of the adjusted target text, s represents the sensitivity score of the unadjusted target text, and l represents the length of the target text.

[0238] In an embodiment of the present application, by normalizing the sensitivity score of the target text, the influence of the length of the target text and the sensitivity score can be eliminated, making the obtained sensitivity score more accurate, thereby improving the accuracy of target text detection.

[0239] Based on a text detection method provided in the above method embodiment, the embodiment of the present application also provides a text detection device, which will be described below in conjunction with the accompanying drawings.

[0240] See also Figure 7 As shown in FIG. 1 , this figure is a schematic diagram of the structure of a text detection device provided in an embodiment of the present application. Figure 7 As shown, the text detection device comprises:

[0241] The first acquisition unit 701 is used to acquire sensitive determination items, each of which includes at least one sensitive word, and each of which corresponds to a sensitivity level;

[0242] A detection unit 702, configured to detect sensitive words included in the target text;

[0243] A first determining unit 703 is used to, when determining the sensitive determination item that can be constituted by the sensitive word element included in the target text, determine the constituted sensitive determination item as the sensitive determination item included in the target text;

[0244] The second determination unit 704 is used to determine the error determination coefficient of the sensitive determination item included in the target text;

[0245] A first calculation unit 705 is used to calculate the distance attenuation coefficient of the sensitive judgment item included in the target text;

[0246] A second calculation unit 706 is used to calculate the sensitivity score of the target text according to the error determination coefficient, the distance decay coefficient and the sensitivity level of each sensitive determination item included in the target text;

[0247] The determination unit 707 is configured to determine that the target text is a sensitive text if the sensitivity score of the target text is greater than a preset threshold.

[0248] In a possible implementation manner, the device further includes:

[0249] A first adding unit is used to add each of the sensitive word elements to an automaton; a path of the automaton includes at least one Chinese character; the Chinese characters included in a path of the automaton correspond to each of the Chinese characters included in a sensitive word element in sequence; each path of the automaton corresponds to a different sensitive word element;

[0250] The detection unit 702 is specifically used to input the target text into the automaton, and if a first path matching the continuous Chinese characters included in the target text can be obtained in each path of the automaton, the sensitive word corresponding to the first path is determined as the sensitive word included in the target text.

[0251] In a possible implementation manner, the device further includes:

[0252] A second adding unit is used to add the Chinese character pinyin string of each Chinese character in each of the sensitive words to the pinyin prefix tree; a path of the pinyin prefix tree includes at least one character node; the character nodes included in a path of the pinyin prefix tree correspond to each pinyin character included in the Chinese character pinyin string in sequence; the Chinese character pinyin string corresponding to each path of the pinyin prefix tree corresponds to one or more homophonic Chinese characters;

[0253] An allocating unit, used for respectively allocating Chinese character serial numbers to the Chinese characters corresponding to each path of the pinyin prefix tree;

[0254] A third adding unit, used for adding the Chinese character sequence number to the Chinese character node of the corresponding Chinese character in the automatic machine;

[0255] The detection unit 702 includes:

[0256] Recognition subunit, used to recognize non-Chinese character strings in the target text;

[0257] A first input subunit is used to input the non-Chinese character string into the pinyin prefix tree, and if a second path matching the continuous characters included in the non-Chinese character string can be obtained in each path of the pinyin prefix tree, the Chinese character sequence number corresponding to the second path is used as the Chinese character sequence number corresponding to the target text;

[0258] The second input sub-unit is used to input the Chinese character serial number corresponding to the target text into the automatic machine. If a third path matching the Chinese character serial number corresponding to the target text can be obtained in each path of the automatic machine, the sensitive word corresponding to the third path is determined as the sensitive word included in the target text.

[0259] In a possible implementation manner, the device further includes:

[0260] A second acquisition unit is used to acquire a first sensitive word element that is a numeric word element from the sensitive word elements included in the target text;

[0261] The first discarding unit is configured to discard the first sensitive word when it is determined that the first sensitive word belongs to the number information in the target text.

[0262] In a possible implementation manner, the device further includes:

[0263] A third acquisition unit is used to acquire a second sensitive word unit that is a single-word word unit from the sensitive word units included in the target text;

[0264] A fourth acquisition unit, configured to acquire, from the target text, a word segment including the second sensitive word element;

[0265] The second discarding unit is configured to discard the second sensitive word when it is determined that the segmented word including the second sensitive word is a high-frequency misjudged word.

[0266] In a possible implementation, the second determining unit 704 includes:

[0267] A word segmentation subunit, used for segmenting the target text to obtain a word segmentation sequence;

[0268] The first determination subunit is used to obtain the error determination coefficient of the j-th sensitive word element of the target sensitive determination item according to the overlap between the j-th sensitive word element of the target sensitive determination item and the marginal characters of the word segmentation in the word segmentation sequence; the target sensitive determination item is any one of the sensitive determination items included in the target text; the value of j is an integer from 1 to m, and m is the number of sensitive word elements included in the target sensitive determination item;

[0269] The calculation subunit is used to calculate the average value of the error determination coefficients of each sensitive word element of the target sensitive determination item to obtain the error determination coefficient of the target sensitive determination item.

[0270] In a possible implementation, the first calculating unit 705 includes:

[0271] A second determination subunit is used for, if the number of sensitive word units included in the target sensitive determination item is 1, then the distance attenuation coefficient of the target sensitive determination item is 1; the target sensitive determination item is any one of the sensitive determination items included in the target text;

[0272] an acquisition subunit, configured to acquire, if the number of sensitive words included in the target sensitivity determination item is greater than 1, a position distance between any two sensitive words included in the target sensitivity determination item in the target text, and obtain a minimum value of the position distances;

[0273] The third determination subunit is used to determine the distance attenuation coefficient of the target sensitivity determination item according to the minimum value in the position distance, and the distance attenuation coefficient of the target sensitivity determination item is inversely proportional to the minimum value in the position distance.

[0274] In a possible implementation, the third determination subunit is specifically used to determine the distance attenuation coefficient of the target sensitive judgment item based on the minimum value of the position distance, the number of sensitive words included in the target sensitive judgment item, and the length of each sensitive word included in the target sensitive judgment item.

[0275] In a possible implementation manner, the device further includes:

[0276] The adjustment unit is used to adjust the sensitivity score of the target text according to the length of the target text, and the sensitivity score of the adjusted target text is in inverse proportion to the length of the target text.

[0277] Figure 8 A schematic diagram of the structure of a device 800 for text detection is shown. For example, the device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0278] Reference Figure 8 Device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0279] The processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0280] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0281] The power supply component 806 provides power to the various components of the device 800. The power supply component 806 can include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 800.

[0282] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0283] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the device 800 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0284] The I / O interface provides an interface between the processing component 802 and the peripheral interface module, which may be a keyboard, a click wheel, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0285] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the device 800, and the sensor assembly 814 can also detect the position change of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and the temperature change of the device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0286] The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0287] In an exemplary embodiment, the device 800 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the following methods:

[0288] Acquire sensitive determination items, each of which includes at least one sensitive word, and each of which corresponds to a sensitivity level;

[0289] Detect and obtain sensitive words included in the target text;

[0290] When determining the sensitive determination item that can be formed by the sensitive word element included in the target text, determining the formed sensitive determination item as the sensitive determination item included in the target text;

[0291] Determining the error determination coefficient of the sensitive determination item included in the target text;

[0292] Calculating the distance attenuation coefficient of the sensitive judgment item included in the target text;

[0293] Calculate the sensitivity score of the target text according to the error determination coefficient, distance attenuation coefficient and sensitivity level of each sensitive determination item included in the target text;

[0294] If the sensitivity score of the target text is greater than a preset threshold, the target text is determined to be a sensitive text.

[0295] In a possible implementation, the method further includes:

[0296] Adding each of the sensitive word units to an automaton; a path of the automaton includes at least one Chinese character node; the Chinese characters included in a path of the automaton correspond to each of the Chinese characters included in a sensitive word unit in sequence; each path of the automaton corresponds to a different sensitive word unit;

[0297] The detection obtains sensitive words included in the target text, including:

[0298] The target text is input into the automaton. If a first path matching the continuous Chinese characters included in the target text can be obtained in each path of the automaton, the sensitive word element corresponding to the first path is determined as the sensitive word element included in the target text.

[0299] In a possible implementation, the method further includes:

[0300] Adding the Chinese character pinyin string of each Chinese character in each of the sensitive words to a pinyin prefix tree; a path of the pinyin prefix tree includes at least one character node; the character nodes included in a path of the pinyin prefix tree correspond to each pinyin character included in the Chinese character pinyin string in sequence; the Chinese character pinyin string corresponding to each path of the pinyin prefix tree corresponds to one or more homophonic Chinese characters;

[0301] Assigning Chinese character serial numbers to the Chinese characters corresponding to each path of the pinyin prefix tree;

[0302] Adding the Chinese character sequence number to the Chinese character node of the corresponding Chinese character in the automaton;

[0303] The detection obtains sensitive words included in the target text, including:

[0304] Identify non-Chinese character strings in the target text;

[0305] Input the non-Chinese character string into the pinyin prefix tree, and if a second path matching the continuous characters included in the non-Chinese character string can be obtained in each path of the pinyin prefix tree, use the Chinese character sequence number corresponding to the second path as the Chinese character sequence number corresponding to the target text;

[0306] The Chinese character serial number corresponding to the target text is input into the automatic machine. If a third path matching the Chinese character serial number corresponding to the target text can be obtained in each path of the automatic machine, the sensitive word corresponding to the third path is determined as the sensitive word included in the target text.

[0307] In a possible implementation, after detecting and obtaining the sensitive word element included in the target text, the method further includes:

[0308] Acquire a first sensitive word element that is a numeric word element among the sensitive words elements included in the target text;

[0309] When it is determined that the first sensitive word belongs to the number information in the target text, the first sensitive word is discarded.

[0310] In a possible implementation, after detecting and obtaining the sensitive word element included in the target text, the method further includes:

[0311] Acquire a second sensitive word unit that is a single-word word unit from the sensitive word units included in the target text;

[0312] Acquire the segmented words including the second sensitive word in the target text;

[0313] When it is determined that the segmented word including the second sensitive word element is a high-frequency misjudged word, the second sensitive word element is discarded.

[0314] In a possible implementation, determining the error determination coefficient of the sensitive determination item included in the target text includes:

[0315] Segmenting the target text to obtain a segmentation sequence;

[0316] According to the overlap between the j-th sensitive word element of the target sensitive determination item and the marginal characters of the word segmentation in the word segmentation sequence, the misjudgment coefficient of the j-th sensitive word element of the target sensitive determination item is obtained; the target sensitive determination item is any one of the sensitive determination items included in the target text; the value of j is an integer from 1 to m, and m is the number of sensitive word elements included in the target sensitive determination item;

[0317] The average value of the error determination coefficients of each sensitive word element of the target sensitive determination item is calculated to obtain the error determination coefficient of the target sensitive determination item.

[0318] In a possible implementation, the calculating the distance attenuation coefficient of the sensitive determination item included in the target text includes:

[0319] If the number of sensitive word units included in the target sensitive determination item is 1, the distance attenuation coefficient of the target sensitive determination item is 1; the target sensitive determination item is any one of the sensitive determination items included in the target text;

[0320] If the number of sensitive words included in the target sensitivity determination item is greater than 1, obtain the position distance between any two sensitive words included in the target sensitivity determination item in the target text, and obtain the minimum value of the position distance;

[0321] The distance attenuation coefficient of the target sensitivity determination item is determined according to the minimum value in the position distance, and the distance attenuation coefficient of the target sensitivity determination item is inversely proportional to the minimum value in the position distance.

[0322] In a possible implementation manner, determining the distance attenuation coefficient of the target sensitivity determination item according to the minimum value of the position distances includes:

[0323] The distance attenuation coefficient of the target sensitive determination item is determined according to the minimum value of the position distance, the number of sensitive word elements included in the target sensitive determination item, and the length of each sensitive word element included in the target sensitive determination item.

[0324] In a possible implementation, the method further includes:

[0325] The sensitivity score of the target text is adjusted according to the length of the target text, and the adjusted sensitivity score of the target text is in inverse proportion to the length of the target text.

[0326] Fig. 9 9 is a schematic diagram of the structure of the server in the embodiment of the present invention. The server 900 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 922 (for example, one or more processors) and memory 932, and one or more storage media 930 (for example, one or more mass storage devices) storing application programs 942 or data 944. Among them, the memory 932 and the storage medium 930 can be short-term storage or permanent storage. The program stored in the storage medium 930 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 922 can be configured to communicate with the storage medium 930, and execute a series of instruction operations in the storage medium 930 on the server 900 for executing the above-mentioned method for building a knowledge base or information query method.

[0327] The server 900 may also include one or more power supplies 926, one or more wired or wireless network interfaces 950, one or more input and output interfaces 956, one or more keyboards 956, and / or one or more operating systems 941, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0328] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system or device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0329] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0330] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0331] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0332] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A text detection method, characterized in that: The method comprises: Acquire sensitive determination items, each of which includes at least one sensitive word, and each of which corresponds to a sensitivity level; Detect and obtain sensitive words included in the target text; When determining the sensitive determination item that can be formed by the sensitive word element included in the target text, determining the formed sensitive determination item as the sensitive determination item included in the target text; Determining the error determination coefficient of the sensitive determination item included in the target text; Calculating the distance attenuation coefficient of the sensitive determination item included in the target text, including: if the number of sensitive words included in the target sensitive determination item is 1, then the distance attenuation coefficient of the target sensitive determination item is 1; the target sensitive determination item is any one of the sensitive determination items included in the target text; if the number of sensitive words included in the target sensitive determination item is greater than 1, obtaining the position distance between any two sensitive words included in the target sensitive determination item in the target text, and obtaining the minimum value in the position distance; determining the distance attenuation coefficient of the target sensitive determination item according to the minimum value in the position distance, and the distance attenuation coefficient of the target sensitive determination item is inversely proportional to the minimum value in the position distance; Calculate the sensitivity score of the target text according to the error determination coefficient, distance attenuation coefficient and sensitivity level of each sensitive determination item included in the target text; If the sensitivity score of the target text is greater than a preset threshold, the target text is determined to be a sensitive text.

2. The method according to claim 1, characterized in that The method further comprises: Adding each of the sensitive word units to an automaton; a path of the automaton includes at least one Chinese character node; the Chinese characters included in a path of the automaton correspond to each of the Chinese characters included in a sensitive word unit in sequence; each path of the automaton corresponds to a different sensitive word unit; The detection obtains sensitive words included in the target text, including: The target text is input into the automaton. If a first path matching the continuous Chinese characters included in the target text can be obtained in each path of the automaton, the sensitive word element corresponding to the first path is determined as the sensitive word element included in the target text.

3. The method according to claim 2, characterized in that The method further comprises: Adding the Chinese character pinyin string of each Chinese character in each of the sensitive words to a pinyin prefix tree; a path of the pinyin prefix tree includes at least one character node; the character nodes included in a path of the pinyin prefix tree correspond to each pinyin character included in the Chinese character pinyin string in sequence; the Chinese character pinyin string corresponding to each path of the pinyin prefix tree corresponds to one or more homophonic Chinese characters; Assigning Chinese character serial numbers to the Chinese characters corresponding to each path of the pinyin prefix tree; Adding the Chinese character sequence number to the Chinese character node of the corresponding Chinese character in the automaton; The detection obtains sensitive words included in the target text, including: Identify non-Chinese character strings in the target text; Input the non-Chinese character string into the pinyin prefix tree, and if a second path matching the continuous characters included in the non-Chinese character string can be obtained in each path of the pinyin prefix tree, use the Chinese character sequence number corresponding to the second path as the Chinese character sequence number corresponding to the target text; The Chinese character serial number corresponding to the target text is input into the automatic machine. If a third path matching the Chinese character serial number corresponding to the target text can be obtained in each path of the automatic machine, the sensitive word corresponding to the third path is determined as the sensitive word included in the target text.

4. The method according to claim 1, characterized in that: After detecting and obtaining the sensitive words included in the target text, the method further includes: Acquire a first sensitive word element that is a numeric word element among the sensitive words elements included in the target text; When it is determined that the first sensitive word belongs to the number information in the target text, discarding the first sensitive word; or, Acquire a second sensitive word unit that is a single-word word unit from the sensitive word units included in the target text; Acquire the segmented words including the second sensitive word in the target text; When it is determined that the segmented word including the second sensitive word element is a high-frequency misjudged word, the second sensitive word element is discarded.

5. The method according to claim 1, characterized in that The step of determining the error determination coefficient of the sensitive determination item included in the target text includes: Segmenting the target text to obtain a segmentation sequence; According to the overlap between the j-th sensitive word element of the target sensitive determination item and the marginal characters of the word segmentation in the word segmentation sequence, the misjudgment coefficient of the j-th sensitive word element of the target sensitive determination item is obtained; the target sensitive determination item is any one of the sensitive determination items included in the target text; the value of j is an integer from 1 to m, and m is the number of sensitive word elements included in the target sensitive determination item; The average value of the error determination coefficients of each sensitive word element of the target sensitive determination item is calculated to obtain the error determination coefficient of the target sensitive determination item.

6. The method according to claim 1, characterized in that The determining the distance attenuation coefficient of the target sensitivity determination item according to the minimum value of the position distances includes: The distance attenuation coefficient of the target sensitive determination item is determined according to the minimum value of the position distance, the number of sensitive word elements included in the target sensitive determination item, and the length of each sensitive word element included in the target sensitive determination item.

7. A text detection device, characterized in that: The device comprises: A first acquisition unit is used to acquire sensitive determination items, each of which includes at least one sensitive word, and each of which corresponds to a sensitivity level; A detection unit, used for detecting sensitive words included in the target text; A first determination unit is used to, when determining the sensitive determination item that can be constituted by the sensitive word element included in the target text, determine the constituted sensitive determination item as the sensitive determination item included in the target text; A second determination unit is used to determine the error determination coefficient of the sensitive determination item included in the target text; A first calculation unit, used to calculate the distance attenuation coefficient of the sensitive judgment item included in the target text; A second calculation unit is used to calculate the sensitivity score of the target text according to the error determination coefficient, distance attenuation coefficient and sensitivity level of each sensitive determination item included in the target text; A determination unit, configured to determine that the target text is a sensitive text if the sensitivity score of the target text is greater than a preset threshold; The first computing unit comprises: A second determination subunit is used for, if the number of sensitive word units included in the target sensitive determination item is 1, then the distance attenuation coefficient of the target sensitive determination item is 1; the target sensitive determination item is any one of the sensitive determination items included in the target text; an acquisition subunit, configured to acquire, if the number of sensitive words included in the target sensitivity determination item is greater than 1, a position distance between any two sensitive words included in the target sensitivity determination item in the target text, and obtain a minimum value of the position distances; The third determination subunit is used to determine the distance attenuation coefficient of the target sensitivity determination item according to the minimum value in the position distance, and the distance attenuation coefficient of the target sensitivity determination item is inversely proportional to the minimum value in the position distance.

8. The device according to claim 7, characterized in that The device also includes: A first adding unit is used to add each of the sensitive word elements to an automaton; a path of the automaton includes at least one Chinese character; the Chinese characters included in a path of the automaton correspond to each of the Chinese characters included in a sensitive word element in sequence; each path of the automaton corresponds to a different sensitive word element; The detection unit is specifically used to input the target text into the automaton, and if a first path matching the continuous Chinese characters included in the target text can be obtained in each path of the automaton, the sensitive word element corresponding to the first path is determined as the sensitive word element included in the target text.

9. The device according to claim 8, characterized in that The device also includes: A second adding unit is used to add the Chinese character pinyin string of each Chinese character in each of the sensitive words to the pinyin prefix tree; a path of the pinyin prefix tree includes at least one character node; the character nodes included in a path of the pinyin prefix tree correspond to each pinyin character included in the Chinese character pinyin string in sequence; the Chinese character pinyin string corresponding to each path of the pinyin prefix tree corresponds to one or more homophonic Chinese characters; The allocating unit allocates Chinese character serial numbers to the Chinese characters corresponding to each path of the pinyin prefix tree; A third adding unit, used for adding the Chinese character sequence number to the Chinese character node of the corresponding Chinese character in the automatic machine; The detection unit comprises: Recognition subunit, used to recognize non-Chinese character strings in the target text; A first input subunit is used to input the non-Chinese character string into the pinyin prefix tree, and if a second path matching the continuous characters included in the non-Chinese character string can be obtained in each path of the pinyin prefix tree, the Chinese character sequence number corresponding to the second path is used as the Chinese character sequence number corresponding to the target text; The second input sub-unit is used to input the Chinese character serial number corresponding to the target text into the automatic machine. If a third path matching the Chinese character serial number corresponding to the target text can be obtained in each path of the automatic machine, the sensitive word corresponding to the third path is determined as the sensitive word included in the target text.

10. The device according to claim 7, characterized in that The device also includes: A second acquisition unit is used to acquire a first sensitive word element that is a numeric word element from the sensitive word elements included in the target text; A first discarding unit, configured to discard the first sensitive word when it is determined that the first sensitive word belongs to the number information in the target text; Alternatively, the device further comprises: A third acquisition unit is used to acquire a second sensitive word unit that is a single-word word unit from the sensitive word units included in the target text; A fourth acquisition unit, configured to acquire, from the target text, a word segment including the second sensitive word element; The second discarding unit is configured to discard the second sensitive word when it is determined that the segmented word including the second sensitive word is a high-frequency misjudged word.

11. The device according to claim 7, characterized in that The second determining unit includes: A word segmentation subunit, used for segmenting the target text to obtain a word segmentation sequence; The first determination subunit is used to obtain the error determination coefficient of the j-th sensitive word element of the target sensitive determination item according to the overlap between the j-th sensitive word element of the target sensitive determination item and the marginal characters of the word segmentation in the word segmentation sequence; the target sensitive determination item is any one of the sensitive determination items included in the target text; the value of j is an integer from 1 to m, and m is the number of sensitive word elements included in the target sensitive determination item; The calculation subunit is used to calculate the average value of the error determination coefficients of each sensitive word element of the target sensitive determination item to obtain the error determination coefficient of the target sensitive determination item.

12. A device for text detection, characterized in that: The device comprises a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors to perform the text detection method according to any one of claims 1 to 6.

13. A computer-readable medium having instructions stored thereon, which, when executed by one or more processors, causes a device to perform the text detection method according to any one of claims 1 to 6.

Images