A data violation risk intelligent detection method and system
By hierarchical clustering and running encoding of the prohibited word database and text information, dictionaries are constructed to improve the difference between prohibited words and characters, the problem of low detection efficiency of illegal word in the existing technology is solved, and the risk of faster data violations is achieved Test.
Patent Information
- Application Number
- CN202410912377.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-09
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-07-09
AI Technical Summary
In the detection of illegal word, the existing technology has a huge amount of character data in the prohibited word database, resulting in cumbersome calculation process, which affects the timeliness of the detection results.
By obtaining the database and text information of prohibited words, performing hierarchical clustering and running encoding, dictionaries are constructed to improve the difference between prohibited words and characters, and thus improving the efficiency of data violation detection.
Through hierarchical clustering and coding processing, the differences between prohibited and non-banned characters in text information are improved, and the timeliness of data violation risk detection results are improved.
Smart Images

Figure CN118467741B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of electronic digital data processing, and in particular to a data violation risk intelligent detection method and system. Background Art
[0002] When detecting illegal words, a banned word database is usually established to process and compare the data on the network platform to generate a comparison structure. If they are the same, they are marked as 1, and if they are different, they are marked as 0. However, due to the huge amount of character data in the banned word database, a large amount of calculation process is required in the process of comparing illegal words, and the detection results cannot be obtained in time, affecting the timeliness of the data violation detection results. Summary of the invention
[0003] The present invention provides a data violation risk intelligent detection method and system to solve the existing problems.
[0004] A data violation risk intelligent detection method and system of the present invention adopts the following technical solutions:
[0005] An embodiment of the present invention provides a method for intelligently detecting data violation risks, the method comprising the following steps:
[0006] Acquire a banned word database and text information in a network platform, wherein the banned word database contains a number of banned words, each banned word contains a number of banned characters, and the text information includes a number of characters;
[0007] According to the number of banned words in the banned word database containing different banned words, the co-occurrence rate between different banned words is obtained; according to the co-occurrence rate between different banned words, all banned words in the banned word database are hierarchically clustered to obtain a first-level tree; according to the sum of the co-occurrence rates between different banned words in each layer of the first-level tree and in each cluster of each layer, a clustering coefficient of each layer in the first-level tree is obtained; according to the size of the clustering coefficient, a target layer in the first-level tree is obtained; according to the co-occurrence rate between each banned word in the target layer and other banned words, the banned words in each cluster in the target layer are sorted to obtain a second ordinal value of each banned word; and a first dictionary is constructed based on all characters in the text information and all banned words in the banned word database in combination with the second ordinal value;
[0008] Encode the text information to obtain new text information, perform run-length encoding on the new text information to obtain encoded data, obtain the run-length value of each character in the encoded data, hierarchically cluster all characters in the new text information according to the order of the characters under each run-length value in the first dictionary to obtain a second-level tree, record the path from any tree node to any leaf node in the second-level tree as a branch path, and obtain a first difference value of the branch path according to the difference in the order of all characters in different clusters in the second-level tree in the first dictionary;
[0009] According to the size of the first difference value of the branch path, the characters in the branch path are compared with the banned word database to obtain a data violation detection result.
[0010] Furthermore, the method of obtaining the co-occurrence rate between different forbidden words according to the number of forbidden words containing different forbidden words in the forbidden word database includes the following specific methods:
[0011] Record any banned word in the banned word database as a target banned word, record all banned words in the banned word database that contain the target banned word as a first set of target banned words, record the number of banned words in the first set as a value N, record any banned word that can form a banned word with the target banned word as an associated banned word of the target banned word, record a banned word that contains both the target banned word and any associated banned word of the target banned word in the first set as a combined banned word of the target forbidden word and the associated banned word, obtain the number of combined banned words of the target forbidden word and the associated forbidden word, record it as a value n, and record the ratio of the value n to the value N as the co-occurrence rate of the target forbidden word and the associated forbidden word.
[0012] Furthermore, the method of hierarchically clustering all banned words in the banned word database according to the co-occurrence rates between different banned words to obtain a first-level tree includes the following specific methods:
[0013] The co-occurrence rate between any two banned words in the banned word database is used as the similarity between the two banned words, and all banned words in the banned word database are clustered by hierarchical clustering, and the obtained clustering result is recorded as the first hierarchical tree.
[0014] Furthermore, the clustering coefficient of each layer in the first-level tree is obtained according to the sum of the co-occurrence rates of different banned words in each layer and each cluster in each layer, and the target layer in the first-level tree is obtained according to the size of the clustering coefficient, including the specific method of:
[0015] Record any layer in the first level tree as layer A, obtain the cumulative sum of similarities between all different banned words in layer A, record it as the total similarity ZX1 of layer A, record |1-ZX1| as the total difference parameter ZC of layer A, obtain the cumulative sum of similarities between different banned words in each cluster in layer A, record it as the intra-class similarity ZX2 of the corresponding cluster, record |1-ZX2| as the intra-class difference parameter of the corresponding cluster, obtain the cumulative sum of the intra-class difference parameters of all clusters in layer A, record it as the total intra-class difference parameter ZL of layer A, and record |ZC-ZL| as the cluster division coefficient of layer A;
[0016] The layer corresponding to the maximum clustering coefficient in the first-level tree is recorded as the target layer.
[0017] Furthermore, the method of sorting the banned words of each cluster in the target layer according to the co-occurrence rate between each banned word and other banned words in the target layer to obtain a second ordinal value of each banned word, and constructing a first dictionary based on all characters in the text information and all banned words in the banned word database in combination with the second ordinal value includes the following specific methods:
[0018] Record any cluster in the target layer as the target cluster, take the average of the co-occurrence rates between all forbidden words in the target cluster as the co-occurrence degree of the target cluster to which all the forbidden words belong, sort all the clusters in the target layer according to the size of the co-occurrence degree of the clusters and in ascending order to obtain the first ordinal value of all the clusters in the target layer; record the cumulative sum of the co-occurrence rates between any forbidden word in the target cluster and all other forbidden words in the target cluster as the co-occurrence value of the forbidden word, sort all the forbidden words in the target cluster according to the size of the co-occurrence value and in ascending order to obtain the second ordinal value of all the forbidden words in the target cluster;
[0019] Construct a dictionary, store all characters in the text information and all banned words in the banned word database in the dictionary, put the banned words behind the non-banned words in the dictionary, and sort the banned words in the dictionary according to the second sequence value of the banned words to form a first dictionary.
[0020] Further, the encoding process of the text information to obtain new text information, the run-length encoding of the new text information to obtain the encoded data, the run-length value of each character in the encoded data, and the hierarchical clustering of all characters in the new text information according to the order of the characters under each run-length value in the first dictionary to obtain the second hierarchical tree include the following specific methods:
[0021] Encode the text information by using a BWT algorithm to obtain new text information, perform run-length encoding on the new text information, obtain the run-length value of each character in the encoded data, sort all the run-length values according to the order of each character in the new text information, and obtain a run-length sequence formed by all the run-length values;
[0022] Obtain the order value of each character in the first dictionary in the first dictionary, record it as the dictionary order value of the character, record the character corresponding to each run value in the run sequence as the initial character of the run value, obtain the dictionary order value of the initial character in the first dictionary, take the difference between the dictionary order values of any two different initial characters as the similarity of the two initial characters, perform bottom-up hierarchical clustering on all initial characters based on the similarity of the two initial characters, and record the obtained clustering result as the second level tree.
[0023] Furthermore, the first difference of the branch path is obtained according to the difference in the order of all characters in different clusters in the second level tree in the first dictionary, including the specific method of:
[0024] Each branch path contains several clusters. Any cluster in any branch path is recorded as a branch cluster. The maximum difference of the dictionary order values of all different characters in the branch cluster is recorded as the first distance value of the branch cluster. The first distance values of all branch clusters on the corresponding branch path are sorted in order from the tree node to the leaf node to obtain the distance sequence of the branch path. The absolute value of the difference between the i-th and i-1-th first distance values in the distance sequence of the branch path is obtained to obtain the difference sequence of the distance sequence. The maximum value of the elements in the difference sequence is recorded as the first difference value of the branch path.
[0025] Furthermore, the character in the branch path is compared with the banned word database according to the size of the first difference of the branch path to obtain the data violation detection result, including the specific method of:
[0026] The characters in all branch paths in the second-level tree are compared with the banned words in the banned word database in order from large to small according to the first difference value. If the character in the branch path is the same as the banned word in the banned word database, the character is marked as 1, if not the same, it is marked as 0. When there are at least two consecutive characters marked as 1 in the text information, the two consecutive characters are marked as banned words, and a label and warning are given.
[0027] Furthermore, the specific method for obtaining the text information is:
[0028] When the text information exists in the form of text data, the text data is directly used as the text information; when the text information exists in the form of image data or video data, the text information in the image data and the text information of each frame in the video data are extracted through OCR technology.
[0029] A data violation risk intelligent detection system comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements any one of the steps of the data violation risk intelligent detection method when executing the computer program.
[0030] The beneficial effect of the technical solution of the present invention is: by performing hierarchical clustering processing on different banned words in a banned word database and characters in text information, the corresponding order of banned words or characters in the obtained hierarchical tree reflects the association characteristics between different banned words or characters, and by obtaining the co-occurrence rate between different banned words, the degree of association between different banned words is reflected. The greater the co-occurrence rate, the greater the probability that banned words can form banned words. Therefore, through hierarchical clustering, banned words that can form banned words in the first hierarchical tree are more likely to be clustered into a cluster. Furthermore, the clustering coefficient is used to describe each layer in the first hierarchical tree. The degree to which banned characters that can form banned words are divided together, the larger the cluster division coefficient, the more likely the banned characters in the corresponding layer are to form banned words. Therefore, the second ordinal value obtained by sorting by co-occurrence value reflects the probability that banned characters can form banned words with other banned characters. The larger the second ordinal value, the greater the probability that banned characters form banned words with other banned characters. By converting, encoding and clustering text information, the difference between banned characters and non-banned characters in text information in the second-level tree is improved, the efficiency of subsequent comparison of characters in text information with banned words in the banned word database is improved, and the timeliness of data violation risk detection results is improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0032] Figure 1 A flowchart of the steps of a data violation risk intelligent detection method of the present invention;
[0033] Figure 2 A data violation risk detection flow chart provided for one embodiment of the present invention. DETAILED DESCRIPTION
[0034] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of a data violation risk intelligent detection method and system proposed by the present invention, its specific implementation method, structure, features and effects, in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.
[0035] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.
[0036] The specific scheme of a data violation risk intelligent detection method and system provided by the present invention is described in detail below with reference to the accompanying drawings.
[0037] See also Figure 1 , which shows a flowchart of a method for intelligent detection of data violation risks provided by an embodiment of the present invention, the method comprising the following steps:
[0038] Step S001: Obtain the banned word database and text information in the network platform.
[0039] It should be noted that when there are illegal words in the text information on the Internet platform, the text information usually exists in different data forms, including: voice, pictures, text, video, etc. Therefore, in order to facilitate data analysis and illegal word detection, when the text information is in voice, pictures and videos, text extraction operations are usually required to facilitate subsequent analysis of the text data to further extract the illegal words in the text data.
[0040] Specifically, in order to implement a data violation risk intelligent detection method proposed in this embodiment, it is necessary to first collect text information. The specific process is as follows:
[0041] First, obtain the text information in the network platform.
[0042] The specific method for obtaining the text information is:
[0043] When the text information exists in the form of text data, the text data is directly used as the text information.
[0044] When the text information exists in the form of image data or video data, for each frame of the image data and video data, the text area is obtained through a semantic segmentation algorithm, and then the text information in the text area is extracted through OCR (Optical Character Recognition) technology.
[0045] When the text information exists in the form of voice data, the voice is converted into text information through voice recognition technology.
[0046] The speech recognition technology includes Hidden Markov Model, Connectionist Temporal Classification (CTC), etc.
[0047] Then, a banned word database is obtained, wherein the banned word database contains a number of banned words, and the banned words contain a number of banned characters.
[0048] So far, the text information and banned word database are obtained through the above method.
[0049] Step S002: obtaining the co-occurrence rate between different banned words according to the number of banned words in the banned word database, hierarchically clustering all banned words in the banned word database according to the co-occurrence rate between different banned words to obtain a first hierarchical tree, obtaining a clustering coefficient for each layer in the first hierarchical tree according to the sum of the co-occurrence rates between different banned words in each layer of the first hierarchical tree and in each cluster of each layer, obtaining a target layer in the first hierarchical tree according to the size of the clustering coefficient, sorting the banned words in each cluster in the target layer according to the co-occurrence rate between each banned word and other banned words in the target layer to obtain a second ordinal value for each banned word, and constructing a first dictionary based on the second ordinal value and all characters in the text information and all banned words in the banned word database.
[0050] It should be noted that in order to improve the difference between banned words and non-banned words and facilitate subsequent violation detection of characters in text information, this embodiment clusters the banned words in the banned word database through the combination relationship between different banned words in different banned words, and analyzes the tree structure obtained after clustering.
[0051] Specifically, in step 2.1, first, any banned word in the banned word database is recorded as a target banned word, all banned words in the banned word database that contain the target banned word are recorded as a first set of target banned words, the number of banned words in the first set is recorded as a numerical value N, any banned word that can form a banned word with the target forbidden word is recorded as an associated forbidden word of the target forbidden word, a banned word that contains both the target forbidden word and any associated forbidden word of the target forbidden word in the first set is recorded as a combined forbidden word of the target forbidden word and the associated forbidden word, the number of combined forbidden words of the target forbidden word and the associated forbidden word is obtained, recorded as a numerical value n, and the ratio of the numerical value n to the numerical value N is recorded as the co-occurrence rate of the target forbidden word and the associated forbidden word.
[0052] Then, the co-occurrence rate between any two banned words in the banned word database is used as the similarity between the two banned words, and all banned words in the banned word database are clustered by hierarchical clustering, and the obtained clustering result is recorded as the first hierarchical tree.
[0053] It should be noted that by performing hierarchical clustering on the data, a clustering result with a tree structure can be obtained. The tree structure contains multiple levels, each level contains multiple nodes, one node corresponds to a cluster, and in this embodiment, a cluster contains several characters. Since hierarchical clustering is an existing clustering method, this embodiment does not go into details about hierarchical clustering.
[0054] Step 2.2, first, any layer in the first level tree is recorded as layer A, and the clustering coefficient of layer A is obtained according to the distribution of similarities between different banned words in layer A. The specific acquisition method is:
[0055] Obtain the cumulative sum of the similarities between all different banned words in layer A, recorded as the total similarity ZX1 of layer A, record |1-ZX1| as the total difference parameter ZC of layer A, obtain the cumulative sum of the similarities between different banned words in each cluster in layer A, record it as the intra-class similarity ZX2 of the corresponding cluster, record |1-ZX2| as the intra-class difference parameter of the corresponding cluster, obtain the cumulative sum of the intra-class difference parameters of all clusters in layer A, recorded as the total intra-class difference parameter ZL of layer A, and record |ZC-ZL| as the cluster division coefficient of layer A.
[0056] Then, the layer corresponding to the maximum clustering coefficient in the first level tree is recorded as the target layer.
[0057] Finally, any cluster in the target layer is recorded as the target cluster, the mean of the co-occurrence rates between all forbidden words in the target cluster is taken as the co-occurrence degree of the target cluster to which all the forbidden words belong, and all the clusters in the target layer are sorted according to the size of the co-occurrence degree of the clusters and in ascending order to obtain the first ordinal value of all the clusters in the target layer; the cumulative sum of the co-occurrence rates between any forbidden word in the target cluster and all other forbidden words in the target cluster is recorded as the co-occurrence value of the forbidden word, and all the forbidden words in the target cluster are sorted according to the size of the co-occurrence value and in ascending order to obtain the second ordinal value of all the forbidden words in the target cluster;
[0058] It should be noted that the co-occurrence rate described in this embodiment reflects the degree of association between different banned characters, and is used to describe the probability that different banned characters can be combined into banned words. The larger the co-occurrence rate, the greater the probability that banned characters can form banned words. Therefore, the banned characters with a higher probability of forming banned words are clustered through hierarchical clustering, so that the banned characters that can form banned words in the first level tree are more likely to be gathered into a cluster. Furthermore, the cluster division coefficient is used to describe the degree to which each layer in the first level tree divides the banned characters that can form banned words together. The larger the cluster division coefficient, the more likely the banned characters in the corresponding layer are to form banned words. Therefore, the second ordinal value obtained by sorting according to the size of the co-occurrence value reflects the probability that a banned character can form a banned word with other banned characters. The larger the second ordinal value, the greater the probability that a banned character forms a banned word with other banned characters, and vice versa.
[0059] Step 2.3, construct a dictionary, store all characters in the text information and all banned words in the banned word database into the dictionary, put the banned words behind the non-banned words in the dictionary, and sort the banned words in the dictionary according to the second order value of the banned words to form a first dictionary, obtain the order value of each character in the first dictionary in the first dictionary, and record it as the dictionary order value of the character.
[0060] It should be noted that, in the first dictionary, prohibited words are also referred to as characters.
[0061] Step S003: Encode the text information to obtain new text information, perform run-length encoding on the new text information to obtain encoded data, obtain the run-length value of each character in the encoded data, hierarchically cluster all characters in the new text information according to the order of the characters under each run-length value in the first dictionary to obtain a second hierarchical tree, and obtain a first difference according to the difference in the order of all characters in different clusters in the second hierarchical tree in the first dictionary.
[0062] It should be noted that for an actual segment of initial data, the order distance of characters is used as the clustering criterion. Since the order values between forbidden characters are small, while the order values between non-forbidden characters are large, the forbidden character category and the non-forbidden character category are better distinguished by the hierarchical clustering method. Each character, whether it is a forbidden character or a non-forbidden character, may appear multiple times in the initial data. Therefore, the order values of the same character are first made similar through BWT encoding, and then the character types are obtained, that is, how many types of characters are there in the initial data.
[0063] Specifically, in step 3.1, first, the text information is encoded by the BWT algorithm to obtain new text information, the new text information is run-length encoded, the run value of each character in the encoded data is obtained, all the run values are sorted according to the order of each character in the new text information, and the run sequence formed by all the run values is obtained.
[0064] Then, the character corresponding to each run value in the run sequence is recorded as the initial character of the run value, the dictionary order value of the initial character in the first dictionary is obtained, the difference between the dictionary order values of any two different initial characters is used as the similarity of the two initial characters, and all initial characters are hierarchically clustered from bottom to top based on the similarity of the two initial characters, and the obtained clustering result is recorded as the second level tree.
[0065] It should be noted that when performing hierarchical clustering on the initial characters, the similarity between any two initial characters at the bottom layer of the second hierarchical tree is the difference between the dictionary order values of the initial characters in the first dictionary. When there are nodes containing multiple characters in the second hierarchical tree, the maximum similarity of all two different initial characters in the two nodes is taken as the similarity between the two nodes, so as to facilitate hierarchical clustering of the nodes.
[0066] In addition, an example is given to illustrate the method of obtaining the run sequence: if there is new text information aabbc, the first type of character in the new text information is a, and the corresponding run value is 2. Similarly, the run values of the second type of character b and the third type of character c are 2 and 1 respectively. Therefore, the run sequence of the new text information aabbc is [221], where the character corresponding to the first 2 in the run sequence is a.
[0067] It should be noted that since the BWT (Burrows-Wheeler Transform) algorithm is an existing encoding algorithm for data conversion, this embodiment will not be described in detail; in addition, the new text information obtained by processing the text information through the BWT algorithm makes the order values of the same character in the new text information similar.
[0068] Step 3.2, record the path from any tree node to any leaf node in the second-level tree as a branch path, each branch path contains several clusters, record any cluster in any branch path as a branch cluster, record the maximum difference of the dictionary order values of all different characters in the branch cluster as the first distance value of the branch cluster, sort the first distance values of all branch clusters on the corresponding branch path in the order from the tree node to the leaf node, and obtain the distance sequence of the branch path, obtain the absolute value of the difference between the i-th and i-1-th first distance values in the distance sequence of the branch path, obtain the difference sequence of the distance sequence, and record the maximum value of the elements in the difference sequence as the first difference value of the branch path.
[0069] It should be noted that the larger the first difference is, the more likely it is that there are banned words in the branch, and a higher priority examination is required.
[0070] It should be noted that, in the present embodiment, if there is a branch path in the second-level tree that contains banned words in the cluster, then the branch path starting from the tree node will have a turning point with a large change in the first distance value. This is because in the process of hierarchical clustering from bottom to top, the banned words will be classified together due to their similar order values, and then with further clustering, they will be classified into the same category as non-banned words. At this time, due to the large difference in the order values of non-banned words and banned words, a turning point will appear in the distance sequence of the corresponding branch path, resulting in excessive difference in the numerical values of the elements in the distance sequence, which corresponds to the first difference.
[0071] So far, the first difference value of each branch path in the second level tree is obtained through the above method.
[0072] Step S004: Compare the characters in the branch path with the banned word database according to the size of the first difference of the branch path to obtain a data violation detection result.
[0073] Specifically, the characters in all branch paths in the second level tree are compared with the banned words in the banned word database in order from large to small according to the first difference value. If the characters in the branch paths are the same as the banned words in the banned word database, the characters are marked as 1, and if they are different, they are marked as 0. When there are at least two characters marked as 1, the two characters are marked as banned words, and are marked and warned.
[0074] At this point, the data violation detection result is obtained through the above method.
[0075] Through the above steps, data violation detection on the network platform is completed.
[0076] See also Figure 2 , which shows a data violation risk detection flow chart provided by an embodiment of the present invention. In the present invention, a data violation risk intelligent detection system includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, steps S001 to S004 in a data violation risk intelligent detection method are implemented.
[0077] In this embodiment, hierarchical clustering is performed on different banned words in the banned word database and characters in text information. The corresponding order of banned words or characters in the obtained hierarchical tree reflects the association characteristics between different banned words or characters. The co-occurrence rate between different banned words is obtained to reflect the association degree between different banned words. The greater the co-occurrence rate, the greater the probability that banned words can form banned words. Therefore, hierarchical clustering makes it more likely that banned words that can form banned words in the first hierarchical tree are clustered into a cluster. Furthermore, the clustering coefficient is used to describe the probability that each layer in the first hierarchical tree will form banned words. The degree to which banned words in a word are divided together, the larger the cluster division coefficient, the more likely the banned words in the corresponding layer are to form banned words. Therefore, the second ordinal value obtained by sorting by co-occurrence value reflects the probability that banned words can form banned words with other banned words. The larger the second ordinal value, the greater the probability that banned words form banned words with other banned words. By converting, encoding and clustering text information, the difference between banned words and non-banned words in text information in the second-level tree is improved, the efficiency of subsequent comparison of characters in text information with banned words in the banned word database is improved, and the timeliness of data violation risk detection results is improved.
[0078] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A data violation risk intelligent detection method, characterized in that: The method comprises the following steps: Acquire a banned word database and text information in a network platform, wherein the banned word database contains a number of banned words, each banned word contains a number of banned characters, and the text information includes a number of characters; According to the number of banned words in the banned word database containing different banned words, the co-occurrence rate between different banned words is obtained; according to the co-occurrence rate between different banned words, all banned words in the banned word database are hierarchically clustered to obtain a first-level tree; according to the sum of the co-occurrence rates between different banned words in each layer of the first-level tree and in each cluster of each layer, a clustering coefficient of each layer in the first-level tree is obtained; according to the size of the clustering coefficient, a target layer in the first-level tree is obtained; according to the co-occurrence rate between each banned word in the target layer and other banned words, the banned words in each cluster in the target layer are sorted to obtain a second ordinal value of each banned word; and a first dictionary is constructed based on all characters in the text information and all banned words in the banned word database in combination with the second ordinal value; Encode the text information to obtain new text information, perform run-length encoding on the new text information to obtain encoded data, obtain the run-length value of each character in the encoded data, hierarchically cluster all characters in the new text information according to the order of the characters under each run-length value in the first dictionary to obtain a second-level tree, record the path from any tree node to any leaf node in the second-level tree as a branch path, and obtain a first difference value of the branch path according to the difference in the order of all characters in different clusters in the second-level tree in the first dictionary; According to the size of the first difference value of the branch path, the characters in the branch path are compared with the banned word database to obtain a data violation detection result; The method of obtaining the co-occurrence rate between different forbidden words according to the number of forbidden words containing different forbidden words in the forbidden word database includes the following specific methods: Record any banned word in the banned word database as a target banned word, record all banned words in the banned word database that contain the target banned word as a first set of target banned words, record the number of banned words in the first set as a value N, record any banned word that can form a banned word with the target banned word as an associated banned word of the target banned word, record a banned word that contains both the target banned word and any associated banned word of the target banned word in the first set as a combined banned word of the target banned word and the associated banned word, obtain the number of combined banned words of the target banned word and the associated banned word, record it as a value n, and record the ratio of the value n to the value N as the co-occurrence rate of the target banned word and the associated banned word; The method of obtaining the clustering coefficient of each layer in the first-level tree according to the sum of the co-occurrence rates of different banned words in each layer and each cluster in each layer, and obtaining the target layer in the first-level tree according to the size of the clustering coefficient, includes the following specific methods: Record any layer in the first level tree as layer A, obtain the cumulative sum of similarities between all different banned words in layer A, record it as the total similarity ZX1 of layer A, record |1-ZX1| as the total difference parameter ZC of layer A, obtain the cumulative sum of similarities between different banned words in each cluster in layer A, record it as the intra-class similarity ZX2 of the corresponding cluster, record |1-ZX2| as the intra-class difference parameter of the corresponding cluster, obtain the cumulative sum of the intra-class difference parameters of all clusters in layer A, record it as the total intra-class difference parameter ZL of layer A, and record |ZC-ZL| as the cluster division coefficient of layer A; The layer corresponding to the maximum clustering coefficient in the first-level tree is recorded as the target layer.
2. According to claim 1, a data violation risk intelligent detection method is characterized in that: The method of hierarchically clustering all banned words in the banned word database according to the co-occurrence rates between different banned words to obtain a first-level tree includes the following specific methods: The co-occurrence rate between any two banned words in the banned word database is used as the similarity between the two banned words, and all banned words in the banned word database are clustered by hierarchical clustering, and the obtained clustering result is recorded as the first hierarchical tree.
3. According to claim 1, a data violation risk intelligent detection method is characterized in that: The method of sorting the banned words of each cluster in the target layer according to the co-occurrence rate between each banned word and other banned words in the target layer to obtain the second ordinal value of each banned word, and building a first dictionary based on all characters in the text information and all banned words in the banned word database in combination with the second ordinal value includes the following specific methods: Record any cluster in the target layer as the target cluster, take the average of the co-occurrence rates between all forbidden words in the target cluster as the co-occurrence degree of the target cluster to which all the forbidden words belong, sort all the clusters in the target layer according to the size of the co-occurrence degree of the clusters and in ascending order to obtain the first ordinal value of all the clusters in the target layer; record the cumulative sum of the co-occurrence rates between any forbidden word in the target cluster and all other forbidden words in the target cluster as the co-occurrence value of the forbidden word, sort all the forbidden words in the target cluster according to the size of the co-occurrence value and in ascending order to obtain the second ordinal value of all the forbidden words in the target cluster; Construct a dictionary, store all characters in the text information and all banned words in the banned word database in the dictionary, put the banned words behind the non-banned words in the dictionary, and sort the banned words in the dictionary according to the second sequence value of the banned words to form a first dictionary.
4. According to claim 1, a data violation risk intelligent detection method is characterized in that: The method of encoding the text information to obtain new text information, performing run-length encoding on the new text information to obtain encoded data, obtaining the run-length value of each character in the encoded data, and hierarchically clustering all characters in the new text information according to the order of the characters under each run-length value in the first dictionary to obtain a second hierarchical tree includes the following specific methods: Encode the text information by using a BWT algorithm to obtain new text information, perform run-length encoding on the new text information, obtain the run-length value of each character in the encoded data, sort all the run-length values according to the order of each character in the new text information, and obtain a run-length sequence formed by all the run-length values; Obtain the order value of each character in the first dictionary in the first dictionary, record it as the dictionary order value of the character, record the character corresponding to each run value in the run sequence as the initial character of the run value, obtain the dictionary order value of the initial character in the first dictionary, take the difference between the dictionary order values of any two different initial characters as the similarity of the two initial characters, perform bottom-up hierarchical clustering on all initial characters based on the similarity of the two initial characters, and record the obtained clustering result as the second level tree.
5. According to claim 1, a data violation risk intelligent detection method is characterized in that: The specific method of obtaining the first difference of the branch path according to the difference in the order of all characters in different clusters in the second level tree in the first dictionary includes: Each branch path contains several clusters. Any cluster in any branch path is recorded as a branch cluster. The maximum difference of the dictionary order values of all different characters in the branch cluster is recorded as the first distance value of the branch cluster. The first distance values of all branch clusters on the corresponding branch path are sorted in order from the tree node to the leaf node to obtain the distance sequence of the branch path. The absolute value of the difference between the i-th and i-1-th first distance values in the distance sequence of the branch path is obtained to obtain the difference sequence of the distance sequence. The maximum value of the elements in the difference sequence is recorded as the first difference value of the branch path.
6. According to claim 1, a data violation risk intelligent detection method is characterized in that: The method of comparing the characters in the branch path with the banned word database according to the size of the first difference of the branch path to obtain the data violation detection result includes: The characters in all branch paths in the second-level tree are compared with the banned words in the banned word database in order from large to small according to the first difference value. If the character in the branch path is the same as the banned word in the banned word database, the character is marked as 1, if not the same, it is marked as 0. When there are at least two consecutive characters marked as 1 in the text information, the two consecutive characters are marked as banned words, and a label and warning are given.
7. According to claim 1, a data violation risk intelligent detection method is characterized in that: The specific method for obtaining the text information is as follows: When the text information exists in the form of text data, the text data is directly used as the text information; when the text information exists in the form of image data or video data, the text information in the image data and the text information of each frame in the video data are extracted through OCR technology.
8. A data violation risk intelligent detection system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.
