Data matching method and device and electronic equipment

Abstract

The invention provides a data matching method and device and electronic equipment, and relates to the technical field of data processing. The method comprises the following steps: firstly, obtaining a dictionary tree, and receiving to-be-recognized text content; dividing the text content to be recognized into a plurality of words, and converting the plurality of words according to a set conversion rule to obtain a pinyin sequence to be recognized corresponding to each word; and finally, matching the to-be-recognized pinyin in each to-be-recognized pinyin sequence with the dictionary tree according to the respective sequence of the to-be-recognized pinyin, and outputting a matching result. According to the method, the dictionary tree of the pinyin is constructed, the input text to be recognized is converted into the pinyin format after word segmentation, and matching is performed in the pinyin form, so that the scale of a sensitive word bank and the size of the constructed dictionary tree can be reduced, meanwhile, the recognition capability of character deformation with similar pronunciation is enhanced, the misjudgment of a system is reduced, and the reliability of information monitoring is ensured.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a data matching method, device and electronic equipment. Background Art

[0002] Sensitive word recognition (detection) technology is usually applied to various scenarios where text content monitoring and filtering are required, such as online services, social media, news publishing platforms, e-books, etc.

[0003] By real-time monitoring of the text content generated by the target object, the platform can ensure that the generated text content complies with laws, regulations and social moral standards, and prevent the spread of inappropriate speech and information.

[0004] The traditional sensitive word identification method includes the following steps: organizing a sensitive dictionary; first constructing a sensitive word tree based on the sensitive dictionary; then using a deterministic finite automaton (DFA) algorithm to match sensitive words to the input text, and finally outputting the matching result.

[0005] However, the traditional sensitive word identification method has the following problems when using the DFA algorithm to match sensitive words in the input text:

[0006] 1. Problem of identifying homophonetic variations of sensitive words: Traditional sensitive word recognition methods are difficult to identify homophonetic variations of sensitive words.

[0007] 2. Word context ambiguity problem: When matching sensitive words, mainstream DFA algorithms only consider the meaning of the word itself and ignore the context of the word. This is not only easy to cause misjudgment, but may also mistakenly mark words that meet the standards as sensitive words in a specific context.

[0008] 3. Performance and efficiency issues in sensitive word recognition: Due to the huge size of the sensitive word library, existing technologies face performance bottlenecks when processing large-scale text content, affecting word matching efficiency and response time, which is not conducive to real-time monitoring and filtering of sensitive words. Summary of the invention

[0009] The present invention provides a data matching method, device and electronic device to enhance the recognition capability of similar-sounding characters and improve the accuracy and efficiency of sensitive word recognition. The specific technical solution is as follows:

[0010] In a first aspect, the present application provides a data matching method, comprising:

[0011] Get the dictionary tree and receive the text content to be recognized;

[0012] The text content to be recognized is divided into a plurality of words, and the plurality of words are converted according to a set conversion rule to obtain a pinyin sequence to be recognized corresponding to each word;

[0013] The pinyin to be recognized in each pinyin sequence to be recognized is matched with the dictionary tree according to the sequence order of each pinyin to be recognized, and the matching result is output.

[0014] By constructing a Pinyin dictionary tree, the input text to be recognized is segmented and converted into Pinyin format, and matching is performed directly in Pinyin form, which effectively reduces the size of the sensitive vocabulary and the size of the dictionary tree. At the same time, it enhances the recognition ability of deformations of similar-sounding characters, reduces system misjudgments, and ensures the reliability of information monitoring.

[0015] In a possible implementation, obtaining the dictionary tree includes:

[0016] Obtain a sample word set;

[0017] According to the set conversion rules, each sample word in the sample word set is converted to obtain a sample pinyin sequence corresponding to each sample word;

[0018] Based on each of the sample pinyin sequences, the dictionary tree is constructed.

[0019] Based on the above method, each sample word can be converted into a sample pinyin sequence corresponding to each sample word, and based on the sample pinyin sequence, a dictionary tree is constructed for subsequent sensitive word matching.

[0020] In a possible implementation, converting each sample word in the sample word set to obtain a sample pinyin sequence corresponding to each sample word includes:

[0021] Mapping any sample word to be selected in the sample word set according to a word mapping method;

[0022] If the mapping is successful, the sample pinyin sequence corresponding to the sample word to be selected is obtained;

[0023] If the mapping fails, the sample word to be selected is split into multiple characters, and each of the multiple characters is mapped according to the character mapping method to obtain the pinyin of each character; the pinyin of each character is combined to obtain a sample pinyin sequence corresponding to the sample word to be selected;

[0024] After polling each sample word in the sample word set, a sample pinyin sequence corresponding to each sample word is obtained.

[0025] Based on the above method, the phonetic deformation of sensitive words can be quickly identified, the size of the sensitive word library can be reduced, and each sample word in the sample word set can be converted according to the set conversion rules to obtain the sample pinyin sequence corresponding to each sample word.

[0026] In a possible implementation, for any one of the candidate sample pinyin sequences in each of the sample pinyin sequences, the following operations are repeatedly performed to construct the dictionary tree:

[0027] Determine the root node of the dictionary tree, and obtain the first sample pinyin in the sample pinyin sequence to be selected;

[0028] Determine whether the first sample pinyin exists in the child nodes of the root node;

[0029] If yes, determine whether the second sample pinyin in the sample pinyin sequence to be selected exists in the next level node of the child node, and when it is determined that the second sample pinyin does not exist in the next level node, insert the second sample pinyin into the next level node;

[0030] If not, the first sample pinyin is inserted into the child node, wherein the first sample pinyin is earlier in sequence order than the second sample pinyin.

[0031] Based on the above method, a pinyin-based dictionary tree can be designed, allowing the system to quickly locate potential sensitive words during the text matching process, optimize the matching speed, and reduce the size of the dictionary tree and the consumption of computing resources.

[0032] In a possible implementation, matching the pinyin to be recognized in each pinyin sequence to be recognized with the dictionary tree according to the sequence order of each pinyin to be recognized includes:

[0033] Selecting a first pinyin sequence to be recognized in each of the pinyin sequences to be recognized, and determining a first pinyin to be recognized in the first pinyin sequence to be recognized;

[0034] Taking the root node of the dictionary tree as the starting node, determining whether the sample pinyin in the child nodes of the root node matches the first pinyin to be recognized;

[0035] If yes, then according to the sequence order of the respective pinyins to be recognized, the next first pinyin to be recognized in the first pinyin sequence to be recognized is selected for matching until each first pinyin to be recognized in the first pinyin sequence to be recognized is polled;

[0036] If not, then select the second pinyin sequence to be recognized in each of the pinyin sequences to be recognized, and perform matching based on the first pinyin to be recognized in the second pinyin sequence to be selected, until each of the pinyin sequences to be recognized is polled.

[0037] By matching the pinyin to be recognized in each pinyin sequence to be recognized through the improved DFA algorithm, the accuracy of the matching process can be improved; matching is performed according to the sequence order of the pinyin to be recognized, avoiding recognition errors caused by ignoring the context.

[0038] In a second aspect, the present application provides a data matching device, comprising:

[0039] A data acquisition module is used to obtain a dictionary tree and receive text content to be recognized;

[0040] A data conversion module is used to divide the text content to be recognized into multiple words, and convert the multiple words according to the set conversion rules to obtain the pinyin sequence to be recognized corresponding to each word;

[0041] The data matching module is used to match the pinyin to be recognized in each pinyin sequence to be recognized with the dictionary tree according to the sequence order of each pinyin to be recognized, and output the matching result.

[0042] In a possible implementation, the data acquisition module is specifically used to:

[0043] Obtain a sample word set;

[0044] According to the set conversion rule, each sample word in the sample word set is converted to obtain a sample pinyin sequence corresponding to each sample word;

[0045] Based on each of the sample pinyin sequences, the dictionary tree is constructed.

[0046] In a possible implementation, the data acquisition module is specifically used to:

[0047] Mapping any sample word to be selected in the sample word set according to a word mapping method;

[0048] If the mapping is successful, the sample pinyin sequence corresponding to the sample word to be selected is obtained;

[0049] If the mapping fails, the sample word to be selected is split into multiple characters, and each of the multiple characters is mapped according to the character mapping method to obtain the pinyin of each character; the pinyin of each character is combined to obtain a sample pinyin sequence corresponding to the sample word to be selected;

[0050] After polling each sample word in the sample word set, a sample pinyin sequence corresponding to each sample word is obtained.

[0051] In a possible implementation, the data acquisition module is specifically used to:

[0052] Determine the root node of the dictionary tree, and obtain the first sample pinyin in the sample pinyin sequence to be selected;

[0053] Determine whether the first sample pinyin exists in the child nodes of the root node;

[0054] If yes, determine whether the second sample pinyin in the sample pinyin sequence to be selected exists in the next level node of the child node, and when it is determined that the second sample pinyin does not exist in the next level node, insert the second sample pinyin into the next level node;

[0055] If not, the first sample pinyin is inserted into the child node, wherein the first sample pinyin is earlier in sequence order than the second sample pinyin.

[0056] In a possible implementation, the data matching module is specifically used to:

[0057] Selecting a first pinyin sequence to be recognized in each of the pinyin sequences to be recognized, and determining a first pinyin to be recognized in the first pinyin sequence to be recognized;

[0058] Taking the root node of the dictionary tree as the starting node, determining whether the sample pinyin in the child nodes of the root node matches the first pinyin to be recognized;

[0059] If yes, then according to the sequence order of the respective pinyins to be recognized, the next first pinyin to be recognized in the first pinyin sequence to be recognized is selected for matching until each first pinyin to be recognized in the first pinyin sequence to be recognized is polled;

[0060] If not, then select the second pinyin sequence to be recognized in each of the pinyin sequences to be recognized, and perform matching based on the first pinyin to be recognized in the second pinyin sequence to be selected, until each of the pinyin sequences to be recognized is polled.

[0061] In a third aspect, the present application provides an electronic device, including:

[0062] Memory, used to store computer programs;

[0063] The processor is used to implement the steps of the above-mentioned data matching method when executing the computer program stored in the memory.

[0064] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned data matching method are implemented.

[0065] For each aspect from the second to the fourth aspect and the technical effects that may be achieved by each aspect, please refer to the above description of the technical effects that can be achieved by the first aspect or various possible schemes in the first aspect, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A flowchart of a data matching method provided in an embodiment of the present application;

[0067] Figure 2 A schematic diagram of the data matching system architecture provided in an embodiment of the present application;

[0068] Figure 3 An overall flow chart of data matching provided in an embodiment of the present application;

[0069] Figure 4 A schematic diagram of the structure of the dictionary tree provided in the embodiment of the present application;

[0070] Figure 5 A schematic diagram of the structure of a dictionary tree for example provided in an embodiment of the present application;

[0071] Figure 6 A schematic diagram of the structure of a data matching device provided in an embodiment of the present application;

[0072] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The specific operating methods in the method embodiments can also be applied to device embodiments or system embodiments. It should be noted that in the description of the present application, "multiple" is understood as "at least two". "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A is connected to B, which can represent: A is directly connected to B and A is connected to B through C. In addition, in the description of the present application, words such as "first" and "second" are only used to distinguish the purpose of description, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order.

[0074] The embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0075] Sensitive word recognition (detection) technology is usually applied to various scenarios where text content monitoring and filtering are required, such as online services, social media, news publishing platforms, e-books, etc.

[0076] By real-time monitoring of the text content generated by the target object, the platform can ensure that the generated text content complies with laws, regulations and social moral standards, and prevent the spread of inappropriate speech and information.

[0077] The traditional sensitive word identification method includes the following steps: organizing a sensitive dictionary; first constructing a sensitive word tree based on the sensitive dictionary; then using a deterministic finite automaton (DFA) algorithm to match sensitive words to the input text, and finally outputting the matching result.

[0078] However, the traditional sensitive word identification method has the following problems when using the DFA algorithm to match sensitive words in the input text:

[0079] 1. Problem of identifying homophonetic variations of sensitive words: Traditional sensitive word recognition methods are difficult to identify homophonetic variations of sensitive words.

[0080] 2. Word context ambiguity problem: When matching sensitive words, mainstream DFA algorithms only consider the meaning of the word itself and ignore the context of the word. This is not only easy to cause misjudgment, but may also mistakenly mark words that meet the standards as sensitive words in a specific context.

[0081] 3. Performance and efficiency issues in sensitive word recognition: Due to the huge size of the sensitive word library, existing technologies face performance bottlenecks when processing large-scale text content, affecting word matching efficiency and response time, which is not conducive to real-time monitoring and filtering of sensitive words.

[0082] In view of this, in order to enhance the recognition ability of deformations of similar-sounding characters and improve the accuracy and efficiency of sensitive word recognition, the present application provides a data matching method, which specifically includes: first obtaining a dictionary tree and receiving the text content to be recognized; then dividing the text content to be recognized into multiple words, and converting the multiple words according to the set conversion rules to obtain the corresponding pinyin sequence to be recognized for each word; finally, matching the pinyin to be recognized in each pinyin sequence to be recognized with the dictionary tree according to the sequence order of each pinyin to be recognized, and outputting the matching result.

[0083] The method provided in the present application is that the monitoring server constructs a Pinyin dictionary tree, and then converts the input text to be recognized into the Pinyin format after word segmentation, and directly uses the Pinyin form for matching, which effectively reduces the size of the sensitive word library and the size of the constructed dictionary tree, while enhancing the recognition ability of deformations of similar-sounding characters, reducing system misjudgments, and ensuring the reliability of information monitoring.

[0084] Reference Figure 1 As shown, it is a flow chart of a data matching method provided in an embodiment of the present application, the method comprising:

[0085] S1, obtain the dictionary tree and receive the text content to be recognized.

[0086] First of all, the method provided in this application can be applied to Figure 2 In the system architecture shown, the system architecture includes: a monitoring server, a terminal device, and the method provided in the present application can be executed in the monitoring server.

[0087] The monitoring server is used to monitor and filter the text content sent by the target object. For example, it can filter out the wrong speech and information of users on social media platforms or monitor the e-books provided by the server.

[0088] The terminal device includes a user terminal and a display terminal. The user can request corresponding services from the server through the user terminal, for example, posting relevant opinions on social software; the display terminal is used to display the monitoring information of the monitoring server to the system administrator.

[0089] In the embodiments of the present application, as shown in the attached Figure 3 As shown, before the monitoring server recognizes the text to be recognized, it first needs to build a dictionary tree, and the dictionary tree can be constructed by referring to steps 1 to 3:

[0090] Step 1: Obtain a sample word set;

[0091] Step 2: convert each sample word in the sample word set according to the set conversion rule to obtain a sample pinyin sequence corresponding to each sample word;

[0092] Step 3: Build a dictionary tree based on each sample pinyin sequence.

[0093] In step 1, the user can collect sensitive word information related to business requirements and input the sensitive word information into the monitoring server; the monitoring server uses the sensitive word information as sensitive word samples for subsequent sensitive word identification (matching). In this application, the sample word set refers to a sample set composed of sensitive word samples, which can also be called a sensitive word library. In addition, sensitive words can be composed of a single character (word) or multiple characters (words), and this application does not make specific restrictions on this.

[0094] In step two, in order to quickly identify homophonic variants of sensitive words and reduce the scale of maintaining the sensitive word library, each sample word in the sample word set can be transformed according to the set transformation rules to obtain the corresponding sample pinyin sequence for each sample word.

[0095] The monitoring server can convert each character of the sensitive word into the corresponding pinyin through a pinyin conversion algorithm or by using a toolkit (such as the pypinyin library). Specifically, the monitoring server can first obtain a sub-word pinyin table with a high frequency of occurrence in daily life, and establish a sub-pinyin table and a word pinyin table respectively, where the word pinyin table is used to identify polyphonic words.

[0096] Based on the word pinyin table, the monitoring server can map any candidate sample word in the sample word set from word to pinyin in a word mapping manner. When the mapping is successful, the corresponding sample pinyin sequence of the candidate sample word is obtained. For example, the sample "天真" in the sensitive word table is converted to "tian / zhen". If the mapping fails, the monitoring server can split the candidate sample word into multiple characters, and then map each character from word to pinyin in a character mapping manner to obtain the pinyin of each character; finally, the pinyins of each character are combined to obtain the corresponding sample pinyin sequence of the candidate sample word.

[0097] Similarly, according to the above method of transforming a candidate sample word, each sample word in the sample word set is transformed, and the corresponding sample pinyin sequence for each sample word can be obtained.

[0098] Through the above method, each sample word can be converted into the corresponding sample pinyin sequence for each sample word, and subsequent sensitive word matching can be performed based on the sample pinyin sequence.

[0099] In step three, after the monitoring server obtains the corresponding sample pinyin sequence for each sample word, it can construct a dictionary tree based on the corresponding sample pinyin sequence for each sample word.

[0100] For any candidate sample pinyin sequence in each sample pinyin sequence, the method of constructing a dictionary tree is described below by taking a candidate sample pinyin sequence as an example:

[0101] As shown in the appendix Figure 4 First, create a root node Root of the dictionary tree (the root node is the starting point of the dictionary tree, and the value in the root node can be empty or other values). Starting from the root node, each sample pinyin in the candidate sample pinyin sequence is added to the dictionary tree as a subsequent node in turn.

[0102] Determine the first sample Pinyin in the sample Pinyin sequence to be selected (e.g., pi in "pi / pei / guan / jian"), where the first sample Pinyin is the first letter of the sample Pinyin sequence to be selected; then determine whether the first sample Pinyin exists in the child node of the root node. If the first sample Pinyin already exists in the child node of the current node, move directly to the child node and continue to process the next sample Pinyin (i.e., determine whether the second sample Pinyin in the sample Pinyin sequence to be selected exists in the next-level node of the child node, and when it is determined that the second sample Pinyin does not exist in the next-level node, insert the second sample Pinyin into the next-level node); otherwise, create a new child node (i.e., insert the first sample Pinyin into the child node), where the first sample Pinyin is ahead in the sequence order relative to the second sample Pinyin.

[0103] When all the candidate sample pinyins in the candidate sample pinyin sequence have been inserted, the last candidate sample pinyin (e.g., Figure 4 The "jian" in the figure is marked as the end node, which is used to indicate that a complete sensitive word ends here.

[0104] Similarly, by repeating the above operation to process each sample pinyin sequence (eg, the sample pinyin sequence "pi / pei / suan / fa", "xin / xi"), a dictionary tree can be obtained.

[0105] After obtaining the dictionary tree, the monitoring server can receive the text content to be recognized sent by the target object, and monitor and filter the text content to be recognized.

[0106] Through the above method, the pinyin-based dictionary tree design allows the system to quickly locate potential sensitive words during the text matching process, which can optimize the matching speed and reduce the size of the dictionary tree and the consumption of computing resources.

[0107] S2, dividing the text content to be recognized into multiple words, and converting the multiple words according to the set conversion rules to obtain the pinyin sequence to be recognized corresponding to each word.

[0108] In an embodiment of the present application, after constructing a dictionary tree and receiving the text content to be recognized, the monitoring server can first divide the text content to be recognized into multiple words, and convert the multiple words according to the set conversion rules to obtain the pinyin sequence to be recognized corresponding to each word.

[0109] Specifically, the monitoring server can divide the text content to be recognized into multiple words. For example, the text content to be recognized "algorithm matching" is divided into "suan / fa / pi / pei" or "suanfa / pipei"; then the multiple words are converted into pinyin according to the above-mentioned method of converting each sample word in the sample word set to obtain the pinyin sequence to be recognized corresponding to each word, and the repetition is not repeated here.

[0110] In an embodiment of the present application, a deep learning mechanism can be used to segment the content of the text to be recognized, specifically including a data preparation stage, a data annotation stage, and a data training stage. In the data preparation stage, the monitoring server can collect a large-scale annotated Chinese corpus and convert the text content into a data format that the deep learning network can learn; in the data annotation stage, a suitable deep learning network (such as BiLSTM-CRF, BERT) is selected to convert the word segmentation (segmentation) problem into a sequence annotation task, and each word of the input text to be recognized is annotated, and the annotated labels include but are not limited to B (used to represent the starting word of the text to be recognized), I (used to represent the middle word of the text to be recognized), E (used to represent the ending word of the text to be recognized) and S (used to represent a single word); in the data training stage, a large-scale corpus is used to train the neural network model, and then the trained model is used to perform the task of word segmentation of the text to be recognized.

[0111] Through the above method, the text content to be recognized can be segmented, and the multiple words after segmentation can be converted according to the set conversion rules to obtain the pinyin sequence to be recognized corresponding to each word, providing a data basis for subsequent sensitive word matching and ensuring the accuracy and efficiency of the matching process.

[0112] S3, matching the pinyin to be recognized in each pinyin sequence to be recognized with the dictionary tree according to the sequence order of each pinyin to be recognized, and outputting the matching result.

[0113] In the embodiment of the present application, in order to improve the accuracy and efficiency of sensitive word recognition, the present application makes corresponding improvements to the traditional DFA algorithm to make it more suitable for Chinese pinyin matching scenarios. The core idea of ​​the algorithm is: no longer matching according to the character order of the original text, but based on the word segmentation results, matching starts from the pinyin corresponding to the first character of each word. In order to facilitate the description of the matching process, the first pinyin sequence to be identified in each pinyin sequence to be identified is selected, and the matching of the first pinyin sequence to be identified with the dictionary tree is used as an example for explanation:

[0114] The monitoring server starts matching from the first pinyin corresponding to each word in the text to be recognized after word segmentation (i.e., selects the first pinyin sequence to be recognized in each pinyin sequence to be recognized, and determines the first pinyin to be recognized in the first pinyin sequence to be recognized), and searches for the subsequent pinyins of each word in the dictionary tree (DFA tree) one by one (i.e., takes the root node of the dictionary tree as the starting node, and determines whether the sample pinyin in the child node of the root node matches the first pinyin to be recognized); if a matching node is found, move to the next pinyin to continue matching, and the matching does not end with the interval " / " of the word segmentation of the text to be recognized, until an end node is found, indicating that a match is found. sensitive words (that is, according to the sequence order of the pinyins to be recognized, the next first pinyin to be recognized in the first pinyin sequence to be recognized is selected for matching, until each first pinyin to be recognized in the first pinyin sequence to be recognized is polled); if a pinyin has no corresponding child node in the middle, it will fall back to the root node of the dictionary tree (DFA tree), and restart the matching from the first pinyin corresponding to the next word of the text segmentation to be recognized (that is, the second pinyin sequence to be recognized in each pinyin sequence to be recognized is selected, and matching is performed based on the first pinyin to be recognized in the second pinyin sequence to be recognized, until each pinyin sequence to be recognized is polled), and a matching result is obtained.

[0115] By matching the pinyin to be recognized in each pinyin sequence to be recognized through the improved DFA algorithm, the accuracy of the matching process can be improved; matching is performed according to the sequence order of the pinyin to be recognized, avoiding recognition errors caused by ignoring the context.

[0116] For ease of understanding, the following example illustrates the matching process of this application:

[0117] Assume that the sensitive words in the sensitive word list are: "Innocent" and "Harbin". The pinyin dictionary tree composed of sensitive words is shown in the attached Figure 5 As shown:

[0118] Example 1

[0119] Enter the text to be recognized: "It's so hot today!".

[0120] In the traditional sensitive word matching and filtering methods, these texts to be identified will be matched with the sensitive words "innocent" and "time" respectively;

[0121] However, in the matching and filtering method provided by this application, the text to be recognized will first be segmented: "Today / so hot / ah / !"; then the text to be recognized will be converted into pinyin "jin tian / zhen re / ya / !"; in addition, according to the DFA algorithm provided by this application, each initial match will only start from the first pinyin corresponding to each word after segmentation. Finally, for the match result "naive", no sensitive word will be hit. By the method of starting each match only from the first character of each word after segmentation provided by this application, misjudgment can be effectively avoided.

[0122] Example 2

[0123] Input the text content to be detected "Harbin is a tourist city.", the text to be recognized will first be segmented: "Ha / and / bin / is / a / tourist / city / ."; then the text content to be recognized will be converted into pinyin "ha / er / bin / shi / zuo / lv you / cheng shi / ." According to the DFA algorithm provided by this application, each initial match will only start from the first pinyin corresponding to each word after segmentation, and at the same time, the match will not end with the segmentation interval " / ", and the keyword "Harbin" can be successfully hit.

[0124] In daily life, the variants of sensitive words are strange. It is easy that one of the variants is not included in the sensitive word list. Therefore, some key sensitive information is likely to be missed, reducing the recall rate. However, in the method provided by this application, direct matching using pinyin can accurately identify the segmentation variants even if there are segmentation variants.

[0125] In summary, the method provided by this application has the following beneficial effects:

[0126] 1. Improve recognition accuracy: The present invention significantly improves the recognition accuracy of sensitive words through pinyin expansion and context-aware word segmentation technology, effectively reducing misjudgment and missed judgment, and ensuring the reliability of text information monitoring.

[0127] 2. Optimize system performance: The performance optimization module and the upgraded DFA algorithm enable the system to quickly respond to the processing requirements of large-scale text data to be recognized, improve the processing efficiency, and enhance the user experience.

[0128] 3. Reduce maintenance costs: By reducing the scale of the sensitive word library, the present invention simplifies the system maintenance work, reduces the time and resources required for updating and maintaining the sensitive word library, and reduces the operation cost.

[0129] Based on the method provided in the above embodiments, the embodiments of this application also provide a data matching device, as Figure 6The figure is a schematic diagram of the structure of a data matching device in an embodiment of the present application, the device comprising:

[0130] The data acquisition module 601 is used to acquire the dictionary tree and receive the text content to be recognized;

[0131] The data conversion module 602 is used to divide the text content to be recognized into multiple words, and convert the multiple words according to the set conversion rules to obtain the pinyin sequence to be recognized corresponding to each word;

[0132] The data matching module 603 is used to match the pinyin to be recognized in each pinyin sequence to be recognized with the dictionary tree according to the sequence order of each pinyin to be recognized, and output the matching result.

[0133] In a possible implementation, the data acquisition module 601 is specifically used for:

[0134] Obtain a sample word set;

[0135] According to the set conversion rule, each sample word in the sample word set is converted to obtain a sample pinyin sequence corresponding to each sample word;

[0136] Based on each of the sample pinyin sequences, the dictionary tree is constructed.

[0137] In a possible implementation, the data acquisition module 601 is specifically used for:

[0138] Mapping any sample word to be selected in the sample word set according to a word mapping method;

[0139] If the mapping is successful, the sample pinyin sequence corresponding to the sample word to be selected is obtained;

[0140] If the mapping fails, the sample word to be selected is split into multiple characters, and each of the multiple characters is mapped according to the character mapping method to obtain the pinyin of each character; the pinyin of each character is combined to obtain a sample pinyin sequence corresponding to the sample word to be selected;

[0141] After polling each sample word in the sample word set, a sample pinyin sequence corresponding to each sample word is obtained.

[0142] In a possible implementation, the data acquisition module 601 is specifically used for:

[0143] Determine the root node of the dictionary tree, and obtain the first sample pinyin in the sample pinyin sequence to be selected;

[0144] Determine whether the first sample pinyin exists in the child nodes of the root node;

[0145] If yes, determine whether the second sample pinyin in the sample pinyin sequence to be selected exists in the next level node of the child node, and when it is determined that the second sample pinyin does not exist in the next level node, insert the second sample pinyin into the next level node;

[0146] If not, the first sample pinyin is inserted into the child node, wherein the first sample pinyin is earlier in sequence order than the second sample pinyin.

[0147] In a possible implementation, the data matching module 603 is specifically used to:

[0148] Selecting a first pinyin sequence to be recognized in each of the pinyin sequences to be recognized, and determining a first pinyin to be recognized in the first pinyin sequence to be recognized;

[0149] Taking the root node of the dictionary tree as the starting node, determining whether the sample pinyin in the child nodes of the root node matches the first pinyin to be recognized;

[0150] If yes, then according to the sequence order of the respective pinyins to be recognized, the next first pinyin to be recognized in the first pinyin sequence to be recognized is selected for matching until each first pinyin to be recognized in the first pinyin sequence to be recognized is polled;

[0151] If not, then select the second pinyin sequence to be recognized in each of the pinyin sequences to be recognized, and perform matching based on the first pinyin to be recognized in the second pinyin sequence to be selected, until each of the pinyin sequences to be recognized is polled.

[0152] Based on the same inventive concept, an electronic device is also provided in the embodiment of the present application, and the electronic device can realize the functions of the aforementioned data matching device, referring to Figure 7 , the electronic device comprises:

[0153] At least one processor 701, and a memory 702 connected to the at least one processor 701. The specific connection medium between the processor 701 and the memory 702 is not limited in the embodiment of the present application. Figure 7 In the example, the processor 701 and the memory 702 are connected via a bus 700. Figure 7 The connections between other components are shown in bold lines, and are not intended to be limiting. The bus 700 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, Figure 7Only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. Alternatively, the processor 701 can also be called a controller, and there is no limitation on the name.

[0154] In the embodiment of the present application, the memory 702 stores instructions that can be executed by at least one processor 701. The at least one processor 701 can execute the data matching method discussed above by executing the instructions stored in the memory 702. The processor 701 can implement Figure 6 The functions of each module in the device shown.

[0155] Among them, the processor 701 is the control center of the device, and can use various interfaces and lines to connect the various parts of the entire control device. By running or executing instructions stored in the memory 702 and calling data stored in the memory 702, the various functions of the device and process data, the device can be monitored as a whole.

[0156] In one possible design, the processor 701 may include one or more processing units, and the processor 701 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor 701. In some embodiments, the processor 701 and the memory 702 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on separate chips.

[0157] Processor 701 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the data matching method disclosed in the embodiments of the present application can be directly embodied as a hardware processor to be executed, or a combination of hardware and software modules in the processor can be executed.

[0158] The memory 702 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 702 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 702 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 702 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0159] By programming the processor 701, the code corresponding to the data matching method described in the above embodiment can be fixed into the chip, so that the chip can execute the data matching method when running. Figure 1 The steps of the data matching method of the embodiment shown are as follows: How to design and program the processor 701 is a technique known to those skilled in the art and will not be described in detail here.

[0160] Based on the same inventive concept, an embodiment of the present application further provides a storage medium, which stores computer instructions. When the computer instructions are executed on a computer, the computer executes the data matching method discussed above.

[0161] In some possible implementations, various aspects of the data matching method provided by the present application may also be implemented in the form of a program product, which includes a program code. When the program product is run on an apparatus, the program code is used to enable the control device to execute the steps of the data matching method according to various exemplary implementations of the present application described above in this specification.

[0162] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0163] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0164] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0165] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0166] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A data matching method, characterized in that: include: Get the dictionary tree and receive the text content to be recognized; The text content to be recognized is divided into a plurality of words, and the plurality of words are converted according to a set conversion rule to obtain a pinyin sequence to be recognized corresponding to each word; The pinyin to be recognized in each pinyin sequence to be recognized is matched with the dictionary tree according to the sequence order of each pinyin to be recognized, and the matching result is output.

2. The method according to claim 1, characterized in that The obtaining of the dictionary tree comprises: Obtain a sample word set; According to the set conversion rule, each sample word in the sample word set is converted to obtain a sample pinyin sequence corresponding to each sample word; Based on each of the sample pinyin sequences, the dictionary tree is constructed.

3. The method according to claim 2, characterized in that The converting each sample word in the sample word set to obtain a sample pinyin sequence corresponding to each sample word includes: Mapping any sample word to be selected in the sample word set according to a word mapping method; If the mapping is successful, the sample pinyin sequence corresponding to the sample word to be selected is obtained; If the mapping fails, the sample word to be selected is split into multiple characters, and each of the multiple characters is mapped according to the character mapping method to obtain the pinyin of each character; the pinyin of each character is combined to obtain a sample pinyin sequence corresponding to the sample word to be selected; After polling each sample word in the sample word set, a sample pinyin sequence corresponding to each sample word is obtained.

4. The method according to claim 2, characterized in that For any one of the candidate sample pinyin sequences in each of the sample pinyin sequences, repeatedly perform the following operations to construct the dictionary tree: Determine the root node of the dictionary tree, and obtain the first sample pinyin in the sample pinyin sequence to be selected; Determine whether the first sample pinyin exists in the child nodes of the root node; If yes, determine whether the second sample pinyin in the sample pinyin sequence to be selected exists in the next level node of the child node, and when it is determined that the second sample pinyin does not exist in the next level node, insert the second sample pinyin into the next level node; If not, the first sample pinyin is inserted into the child node, wherein the first sample pinyin is earlier in sequence order than the second sample pinyin.

5. The method according to claim 1, characterized in that The step of matching the pinyin to be recognized in each pinyin sequence to be recognized with the dictionary tree according to the sequence order of each pinyin to be recognized includes: Selecting a first pinyin sequence to be recognized in each of the pinyin sequences to be recognized, and determining a first pinyin to be recognized in the first pinyin sequence to be recognized; Taking the root node of the dictionary tree as the starting node, determining whether the sample pinyin in the child nodes of the root node matches the first pinyin to be recognized; If yes, then according to the sequence order of the respective pinyins to be recognized, the next first pinyin to be recognized in the first pinyin sequence to be recognized is selected for matching until each first pinyin to be recognized in the first pinyin sequence to be recognized is polled; If not, then select the second pinyin sequence to be recognized in each of the pinyin sequences to be recognized, and perform matching based on the first pinyin to be recognized in the second pinyin sequence to be selected, until each of the pinyin sequences to be recognized is polled.

6. A data matching device, characterized in that: include: A data acquisition module is used to obtain a dictionary tree and receive text content to be recognized; A data conversion module is used to divide the text content to be recognized into multiple words, and convert the multiple words according to the set conversion rules to obtain the pinyin sequence to be recognized corresponding to each word; The data matching module is used to match the pinyin to be recognized in each pinyin sequence to be recognized with the dictionary tree according to the sequence order of each pinyin to be recognized, and output the matching result.

7. The device according to claim 6, characterized in that The data acquisition module is specifically used for: Obtain a sample word set; According to the set conversion rule, each sample word in the sample word set is converted to obtain a sample pinyin sequence corresponding to each sample word; Based on each of the sample pinyin sequences, the dictionary tree is constructed.

8. The device according to claim 7, characterized in that The data acquisition module is specifically used for: Mapping any sample word to be selected in the sample word set according to a word mapping method; If the mapping is successful, the sample pinyin sequence corresponding to the sample word to be selected is obtained; If the mapping fails, the sample word to be selected is split into multiple characters, and each of the multiple characters is mapped according to the character mapping method to obtain the pinyin of each character; Combining the pinyin of each character to obtain a sample pinyin sequence corresponding to the sample word to be selected; After polling each sample word in the sample word set, a sample pinyin sequence corresponding to each sample word is obtained.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to implement the method steps of any one of claims 1 to 5 when executing the computer program stored in the memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 5 are implemented.

Images