A combined word matching detection method, system, electronic device and storage medium

Abstract

The present invention discloses a combined word matching detection method, system and medium, including: obtaining a preset first combined word, disassembling the first combined word to obtain a plurality of first word elements, and constructing a first relationship mapping table according to the first word elements and the first combined word; generating a word search tree according to the first relationship mapping table; obtaining a text to be detected, performing word element matching on the text to be detected according to the word search tree to obtain a matching record list, and then performing aggregation processing on the matching record list to obtain a combined word index dictionary; determining the minimum interval of word elements in the combined word index dictionary, and calculating the matching scores of each combined word in the combined word index dictionary according to the minimum interval of word elements to obtain a matching score list; performing normalization processing on the matching score list according to the text length and the combined word length to obtain the combined word matching result of the text to be detected. The present invention improves the efficiency and accuracy of combined word matching detection and can be widely applied to the field of text processing technology.

Description

Technical Field

[0001] The present invention relates to the technical field of text processing, and in particular to a combination word matching detection method, system, electronic device and storage medium. Background Art

[0002] Traditional text keyword matching detection methods mostly use a word list to construct a Trie tree and perform character-by-character matching. While this method is highly efficient, it cannot match compound words. Compound word matching methods based on sets or bitwise operations are not only inefficient but also suffer from the following drawbacks, leading to poor accuracy: 1) they cannot handle character deformations (such as pinyin deformation, near-phonetic deformation, and insertion deformation); 2) they cannot handle the distance relationship between tokens in long texts; and 3) they cannot eliminate the influence of token number or length on matching.

[0003] Explanation of terms:

[0004] Trie tree: also known as word search tree, is a tree structure and a variant of hash tree. It is characterized by low memory consumption and super high matching performance.

[0005] Unicode encoding: Unicode has only one character set. Chinese, Japanese, and Korean occupy the Unicode range 0x3000 to 0x9FFF. Unicode currently uses the UCS-2 standard, which uses two bytes to encode a character. For example, the Chinese character for "经" (jing) is encoded as 0x7ECF.

[0006] Unicode Private Use Area: In Unicode, a Private Use Area (PUA) is a range of code points that, by definition, the Unicode Consortium does not assign characters to, so that third parties can define their own characters without conflicting with Unicode Consortium assignments. PUA defines three private use areas, one of which is in the Basic Multilingual Plane (U+E000–U+F8FF). Code points in these areas cannot be considered standardized characters in Unicode itself.

[0007] Sigmoid function: named after its image resembling S, it is usually used to map real numbers to the interval (0,1). Its characteristic is that it grows rapidly within a certain interval and tends to saturation outside the interval.

[0008] Compound words: a word sequence consisting of multiple words, such as "online|poker".

[0009] Compound word lemma: a word in a compound word. For example, if the compound word is “在线|扑克”, then the lemma is “在线” and “扑克”. Summary of the Invention

[0010] An object of the present invention is to solve at least to some extent one of the technical problems existing in the prior art.

[0011] To this end, an object of an embodiment of the present invention is to provide a combined word matching detection method, which realizes the matching detection of combined words with complex structures in long texts and improves the efficiency and accuracy of combined word matching detection.

[0012] Another object of an embodiment of the present invention is to provide a combined word matching detection system.

[0013] In order to achieve the above technical object, the technical solutions adopted in the embodiments of the present invention include:

[0014] On the one hand, an embodiment of the present invention provides a combined word matching detection method, including the following steps:

[0015] Obtain a preset first combined word, disassemble the first combined word to obtain a plurality of first word elements, and construct a first relationship mapping table according to the first word elements and the first combined word;

[0016] Generate a word search tree according to the first relationship mapping table;

[0017] Obtain the text to be detected, perform word element matching on the text to be detected according to the word search tree to obtain a matching record list, and then perform aggregation processing on the matching record list to obtain a combined word index dictionary;

[0018] Determine the minimum interval of the word elements in the combined word index dictionary, and calculate the matching scores of each combined word in the combined word index dictionary according to the minimum interval of the word elements to obtain a matching score list;

[0019] Perform normalization processing on the matching score list according to the text length and the combined word length to obtain the combined word matching result of the text to be detected.

[0020] Further, in an embodiment of the present invention, the step of obtaining a preset first combined word, disassembling the first combined word to obtain a plurality of first word elements, and constructing a first relationship mapping table according to the first word elements and the first combined word specifically includes:

[0021] Obtain a plurality of the first combined words from a preset combined word table, and determine the combined word IDs of each of the first combined words;

[0022] Disassemble the first combined word according to a combined separator to obtain a plurality of the first word elements, and determine the combined word IDs corresponding to each of the first word elements;

[0023] Generate a first key-value pair based on the first token and the combined word ID, and construct the first relationship mapping table according to the first key-value pair.

[0024] Further, in an embodiment of the present invention, the step of generating a word search tree according to the first relationship mapping table specifically includes:

[0025] Obtain the first key-value pair according to the first relationship mapping table, perform a difference on the first token of the first key-value pair according to characters and convert it into a Unicode encoding sequence;

[0026] Determine the root node and child nodes according to the Unicode encoding sequence and the combined word ID of the first key-value pair;

[0027] Generate the word search tree according to the root node and the child nodes.

[0028] Further, in an embodiment of the present invention, the step of performing token matching on the text to be detected according to the word search tree to obtain a matching record list specifically includes:

[0029] Perform character encoding on the text to be detected to obtain a Unicode encoding array;

[0030] Traverse the Unicode encoding array to obtain multiple token encoding to be matched, and determine the combined word ID corresponding to the token encoding to be matched according to the word search tree;

[0031] Determine the corresponding second token and matching index value according to the token encoding to be matched, and then generate the matching record list according to the second token, the matching index value, and the corresponding combined word ID.

[0032] Further, in an embodiment of the present invention, the step of aggregating and processing the matching record list to obtain a combined word index dictionary specifically includes:

[0033] Obtain target tokens from the matching record list, and determine a plurality of matching index values corresponding to each target token;

[0034] Generate a second key-value pair according to the target token and the corresponding plurality of matching index values;

[0035] Determine a plurality of target tokens corresponding to each combined word ID, and construct a sub-dictionary according to the second key-value pairs generated by the plurality of target tokens corresponding to the same combined word ID;

[0036] Generate a third key-value pair according to the combined word ID and the corresponding sub-dictionary, and construct the combined word index dictionary according to the third key-value pair.

[0037] Further, in an embodiment of the present invention, the step of determining the minimum interval of the word elements of the combined word index dictionary and calculating the matching scores of each combined word in the combined word index dictionary according to the minimum interval of the word elements to obtain a list of matching scores specifically includes:

[0038] Determine the number of word elements of the combined word corresponding to each combined word ID, and determine the number of elements of the corresponding sub-dictionary for each combined word ID;

[0039] If the number of elements is less than the number of word elements, remove the corresponding third key-value pair to obtain the filtered combined word index dictionary;

[0040] Determine the minimum interval of the word elements of the filtered combined word index dictionary, and determine the combined word length of the combined word corresponding to each combined word ID;

[0041] Calculate the matching scores of the combined words corresponding to each combined word ID according to the combined word length, the number of word elements, and the minimum interval of the word elements to obtain the list of matching scores.

[0042] Further, in an embodiment of the present invention, the step of normalizing the list of matching scores according to the text length and the combined word length to obtain the combined word matching result of the text to be detected specifically includes:

[0043] Determine the text length of the text to be detected, and determine the first difference between the text length and the combined word length corresponding to each combined word ID;

[0044] Determine each combined word ID and the corresponding matching score according to the list of matching scores, and determine the corresponding normalized matching score according to the quotient of the matching score and the first difference;

[0045] Compare each normalized matching score with a preset first threshold. If the normalized matching score is less than the first threshold, remove the normalized matching score and the corresponding combined word ID to obtain the filtered normalized matching score;

[0046] Determine the combined word matching result of the text to be detected according to the filtered normalized matching score and the corresponding combined word ID.

[0047] On the other hand, an embodiment of the present invention provides a combined word matching detection system, including:

[0048] A relationship mapping table construction module, configured to obtain a preset first combined word, disassemble the first combined word to obtain a plurality of first word elements, and construct a first relationship mapping table according to the first word elements and the first combined word;

[0049] A word search tree generation module, configured to generate a word search tree according to the first relationship mapping table;

[0050] A word element matching module, configured to obtain a text to be detected, perform word element matching on the text to be detected according to the word search tree to obtain a list of matching records, and then perform aggregation processing on the list of matching records to obtain a combined word index dictionary;

[0051] A matching score calculation module, configured to determine the minimum interval of word elements in the combined word index dictionary, and calculate the matching scores of each combined word in the combined word index dictionary according to the minimum interval of word elements to obtain a list of matching scores;

[0052] A normalization processing module, configured to perform normalization processing on the list of matching scores according to the text length and the combined word length to obtain the combined word matching result of the text to be detected.

[0053] On the other hand, an embodiment of the present invention provides an electronic device, which includes a memory, a processor, a program stored on the memory and executable on the processor, and a data bus for realizing connection communication between the processor and the memory. When the program is executed by the processor, it implements the combined word matching detection method described above.

[0054] On the other hand, an embodiment of the present invention further provides a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the combined word matching detection method described above.

[0055] The advantages and beneficial effects of the present invention will be partially given in the following description, partially become obvious from the following description, or be understood through the practice of the present invention:

[0056] In an embodiment of the present invention, a preset first combined word is obtained, the first combined word is disassembled to obtain a plurality of first word elements, and a first relationship mapping table is constructed based on the first word elements and the first combined word. A word search tree is generated according to the first relationship mapping table. A text to be detected is obtained, and word element matching is performed on the text to be detected according to the word search tree to obtain a list of matching records. Then, the list of matching records is aggregated to obtain a combined word index dictionary. The minimum interval of word elements in the combined word index dictionary is determined, and the matching scores of each combined word in the combined word index dictionary are calculated according to the minimum interval of word elements to obtain a list of matching scores. The list of matching scores is normalized according to the text length and the combined word length to obtain the combined word matching result of the text to be detected. In the embodiment of the present invention, word element matching is performed on the text to be detected through a word search tree, and a combined word index dictionary is obtained through aggregation processing. Furthermore, the matching scores of each combined word are calculated based on the minimum interval of word elements, realizing the matching detection of combined words with complex structures in long texts and improving the efficiency of combined word matching detection. The list of matching scores is normalized according to the text length and the combined word length, reducing the influence of the combined word length and the number of combined words on the matching scores and improving the accuracy of combined word matching detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following introduces the drawings required to be used in the embodiments of the present invention. It should be understood that the drawings introduced below only conveniently and clearly illustrate some embodiments of the technical solutions in the present invention. For those skilled in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0058] Figure 1 It is a flowchart of steps of a combined word matching detection method provided by an embodiment of the present invention;

[0059] Figure 2 It is a flowchart of steps of step S101 provided by an embodiment of the present invention;

[0060] Figure 3 It is a flowchart of steps of step S102 provided by an embodiment of the present invention;

[0061] Figure 4 It is a flowchart of steps of step S103 provided by an embodiment of the present invention;

[0062] Figure 5 It is another flowchart of steps of step S103 provided by an embodiment of the present invention;

[0063] Figure 6 It is a flowchart of steps of step S104 provided by an embodiment of the present invention;

[0064] Figure 7 It is a flowchart of step S105 provided by an embodiment of the present invention;

[0065] Figure 8 It is a schematic structural diagram of a combined word matching detection system provided by an embodiment of the present invention;

[0066] Figure 9 It is a schematic hardware structure diagram of an electronic device provided by an embodiment of the present invention;

[0067] Figure 10 It is a schematic structural diagram of a storage medium provided by an embodiment of the present invention. Detailed implementation manners

[0068] The embodiments of the present invention will be described in detail below. Examples of the embodiments are shown in the drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the drawings are exemplary and are only used to explain the present application and should not be construed as limiting the present application. It should be noted that although the functional modules are divided in the system schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different module division from that in the system schematic diagram or a different order from that in the flowchart. For the step numbers in the following embodiments, they are only set for the convenience of explanation and do not limit the order between the steps at all. The execution order of each step in the embodiments can be adjusted adaptively according to the understanding of those skilled in the art.

[0069] In the description of the present invention, "a plurality" means two or more. If the first and second are described, it is only for the purpose of distinguishing technical features and should not be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features or implicitly indicating the sequence of the indicated technical features. In addition, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0070] The combined word matching detection method provided by the embodiments of the present application can be applied to a terminal, or to a server side, or can also be software running on a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a set-top box, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, or can also be configured as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the combined word matching detection method, etc., but is not limited to the above forms.

[0071] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet-type devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and so on. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0072] It should be noted that in each specific embodiment of the present application, when it comes to relevant processing that needs to be carried out according to data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first. Moreover, the collection, use, and processing of these data will comply with the relevant laws, regulations, and standards of the relevant countries and regions. In addition, when the embodiments of the present application need to obtain the user's sensitive personal information, the user's separate permission or separate consent will be obtained through methods such as pop-up windows or redirecting to a confirmation page. After clearly obtaining the user's separate permission or separate consent, the necessary user-related data for the normal operation of the embodiments of the present application will be obtained.

[0073] The method of the embodiment of the present invention is based on word matching. At the cost of adding extremely little memory and CPU consumption, it supports complex compound word matching detection. By using a lemma matching queue and a sigmoid-like function to calculate scores, the embodiment of the present invention realizes efficient matching of compound words with complex structures in the text. This innovative method not only improves the accuracy and speed of matching, but also achieves remarkable improvements in resource consumption, bringing a new solution approach to the field of compound word matching. Finally, during the process of constructing a word search tree (Trie tree), the embodiment of the present invention uses the Private Use Area of Unicode for Chinese pinyin mapping, supporting variant searches of pinyin, demonstrating the powerful expansion ability based on the present invention.

[0074] As Figure 1 shown in the following is a flowchart of steps of a compound word matching detection method provided by an embodiment of the present invention. Referring to Figure 1 , the embodiment of the present invention provides a compound word matching detection method, which specifically includes the following steps:

[0075] S101. Obtain a preset first compound word, disassemble the first compound word into multiple first lemmas, and construct a first relationship mapping table according to the first lemmas and the first compound word.

[0076] As Figure 2 shown in the following is a flowchart of steps of step S101 provided by an embodiment of the present invention. Referring to Figure 2 , further as an optional implementation manner, the step of disassembling the first compound word into multiple first lemmas and constructing a first relationship mapping table according to the first lemmas and the first compound word specifically includes:

[0077] S1011. Obtain multiple first compound words from a preset compound word table, and determine the compound word IDs of each first compound word;

[0078] S1012. Disassemble the first compound word into multiple first lemmas according to a combination separator, and determine the compound word IDs corresponding to each first lemma;

[0079] S1013. Generate first key-value pairs according to the first lemmas and the compound word IDs, and construct a first relationship mapping table according to the first key-value pairs.

[0080] Specifically, obtain the compound word table from a preset file system or database, such as "online|poker", "online|loan", "organ|trafficking", "human|trafficking", where the "|" symbol is the combination separator. Through the combination separator, the first compound word can be disassembled into multiple first lemmas.

[0081] Construct a "pinyin - Unicode" relationship mapping table Y:

[0082] 1) Create a dictionary with pinyin character sequence as key and Unicode code as value;

[0083] 2) Using the pinyin string itself as the key and E000+size (dictionary) as the value, add the "pinyin-Unicode" relationship mapping to the dictionary;

[0084] 3) Repeat 2) until all Chinese pinyin are traversed.

[0085] Construct a "word unit (including pinyin deformation) - combination word id" relationship mapping table (i.e., the first relationship mapping table):

[0086] 1) Create a dictionary with word element as key and combination word ID list as value;

[0087] 2) Split the compound word (assuming id = x) into tokens by the separator "|", traverse these tokens to search the dictionary in 1, and append the current id = x to the value of the dictionary;

[0088] 3) Convert the Chinese characters in the word into pinyin and repeat the previous step 2);

[0089] 4) Repeat steps 2) and 3) until all combinations are traversed. The final dictionary in step 1 is a "word unit - combination word id" relationship mapping table. For example, if there are two combinations "online|poker" (id=1) and "online|loan" (id=2), the final relationship mapping table will be: "{online: [1, 2], poker: [1], loan: [2]}".

[0090] S102: Generate a word search tree according to the first relationship mapping table.

[0091] like Figure 3 FIG. 1 is a flowchart of step S102 provided in an embodiment of the present invention, referring to FIG. Figure 3 As an optional implementation, the step of generating a word search tree according to the first relationship mapping table specifically includes:

[0092] S1021. Obtain a first key-value pair according to the first relationship mapping table, perform character differentiation on a first word of the first key-value pair, and convert the result into a Unicode encoding sequence;

[0093] S1022, determining a tree root node and child nodes according to the Unicode encoding sequence and the combined word ID of the first key-value pair;

[0094] S1023. Generate a word search tree based on the root node and child nodes.

[0095] Specifically, convert the first relationship mapping table obtained in the foregoing steps into a word search tree (Trie tree):

[0096] 1) Create a tree node data structure - a binary tuple (value, children), where: value is the same as the value in the first relationship mapping table and is a list of combined word IDs; children is a dictionary with Unicode encoding as the key and the child node as the value;

[0097] 2) Create the root node Root(Trie) of the Trie tree as the starting search node - (value = empty, children = {});

[0098] 3) Take out the key-value pairs of the "lexical unit - combined word ID" relationship mapping table obtained in the foregoing steps (assuming key = online, id = [1, 2]), split the key by single characters and convert it into a Unicode encoding sequence U (for example, [7ebf (line), 4e0a (up)]);

[0099] 4) Set the node pointer P to point to the root node Root(Trie), and the character index is C (initially 1). Check whether P.children[U[C]] exists. If it does not exist, create a tree node and add it to P.children. If it exists, change P to point to the sub-node (i.e., P = P.children[U[C]]);

[0100] 5) Advance C by one index (i.e., C = C + 1) and repeat operation 4) until C is greater than the length of the lexical unit characters. At this time, append the combined word ID array corresponding to the lexical unit to P.value;

[0101] 6) Repeat operations 3), 4), and 5) until all key-value pairs in the relationship mapping table are processed. Finally, the root node Root(Trie) in 2) represents the word search tree (Trie tree).

[0102] S103. Obtain the text to be detected, perform lexical unit matching on the text to be detected according to the word search tree, obtain a list of matching records, and then perform aggregation processing on the list of matching records to obtain a combined word index dictionary.

[0103] As Figure 4 shown is a flowchart of the steps of step S103 provided by an embodiment of the present invention. Referring to Figure 4 , further as an optional implementation manner, the step of performing lexical unit matching on the text to be detected according to the word search tree to obtain a list of matching records specifically includes:

[0104] S1031. Perform character encoding on the text to be detected to obtain a Unicode encoding array;

[0105] S1032. Traverse the Unicode encoding array to obtain multiple to-be-matched token encodings, and determine the combined word ID corresponding to the to-be-matched token encoding according to the word search tree;

[0106] S1033. Determine the corresponding second token and matching index value according to the to-be-matched token encoding, and then generate a matching record list according to the second token, the matching index value, and the corresponding combined word ID.

[0107] Specifically, use the word search tree (Trie tree) obtained in the foregoing steps to match the long text, and obtain the matching record list M of tokens:

[0108] 1) Create a matching record list M of tokens, where each element is a triple (token, matching index, combined word id list), and the matching index here indicates the position where the token appears in the long text;

[0109] 2) Assume that the characters in the long text have been encoded as Unicode, then the entire long text can be regarded as a Unicode encoding array U. At this time, the text cursor C points to the i-th (initialized to 1) encoding of the text, and the Trie tree node pointer P points to the root node Root(Trie);

[0110] 3) Check whether P.children contains an element with the key U[C]. If it contains, then point the pointer P to it (i.e., P = P.children[U[C]]). If P.value is not empty, it means that the current encoding sequence has matched a token, and it is necessary to construct a triple (U[i..C], i, P.value) and add it to the matching record list M at the same time;

[0111] 4) If U[C] belongs to the English character encoding range, record the pinyin start cursor K = C, and in each subsequent match, determine whether U[C] belongs to the English character encoding category, and try to find the pinyin Unicode encoding y = Y[U[K..C]] in the mapping table obtained in step 1. If it exists, regard y as a single character encoding and perform matching in the method of 3);

[0112] 5) Advance C by one index (i.e., C = C + 1) and repeat operation 2) until there is no corresponding Unicode encoding in the children of a certain node in the Trie tree, that is, P.chilren[U[C]] = null;

[0113] 6) Advance i by one index (i.e., i = i + 1), that is, point to the next character of the long text. Reset the text cursor C to point to i, P to point to the root node, and repeat operations 2), 3), 4), and 5) until i is greater than the text length.

[0114] Such asFigure 5 The following shows another flowchart of step S103 provided by an embodiment of the present invention. Referring to Figure 5 , as a further optional implementation, the step of aggregating the matching record list to obtain a combined word index dictionary specifically includes:

[0115] S1034. Obtain target word elements from the matching record list and determine a plurality of matching index values corresponding to each target word element;

[0116] S1035. Generate a second key-value pair according to the target word element and the corresponding plurality of matching index values;

[0117] S1036. Determine a plurality of target word elements corresponding to each combined word ID, and construct a sub-dictionary according to the second key-value pairs generated by the plurality of target word elements corresponding to the same combined word ID;

[0118] S1037. Generate a third key-value pair according to the combined word ID and the corresponding sub-dictionary, and construct a combined word index dictionary according to the third key-value pair.

[0119] Specifically, aggregate the obtained matching record list M by combined word id to obtain a combined word index dictionary G:

[0120] 1) Create a combined word index dictionary G, which is a two-level dictionary structure, where the key is the combined word id and the value is the sub-dictionary g. The key of g is the word element and the value is the matching index queue. The structure description can be referred to Figure 3 , for example, G = "{1: {online: [4, 13], poker: [6]}}", indicating that in the group with combined word id = 1, the word "online" appears at indexes 4 and 13, and the word "poker" appears at index 6;

[0121] 2) Take out the triple Mi in the matching record list M in the previous step. Traverse the combined word id list in Mi, search for the sub-dictionary g in the combined word index dictionary G, and further search for the matching index queue by using the word element recorded in Mi in g, and finally append the matching index recorded in Mi to it. It should be noted that since the matching indexes in the matching record list M must be increasing, any index queue in g must also be increasing;

[0122] 3) Repeat step 2) until all matching records are taken out.

[0123] S104. Determine the minimum interval of the word elements in the combined word index dictionary, and calculate the matching scores of each combined word in the combined word index dictionary according to the minimum interval of the word elements to obtain a matching score list.

[0124] Such as Figure 6The following is a flowchart of the steps of step S104 provided by an embodiment of the present invention. Referring to Figure 6 , further as an optional implementation, the step of determining the minimum interval of the word elements of the combined word index dictionary and calculating the matching scores of each combined word in the combined word index dictionary to obtain a list of matching scores specifically includes:

[0125] S1041. Determine the number of word elements of the combined word corresponding to each combined word ID, and determine the number of elements in the sub-dictionary corresponding to each combined word ID;

[0126] S1042. If the number of elements is less than the number of word elements, remove the corresponding third key-value pair to obtain a filtered combined word index dictionary;

[0127] S1043. Determine the minimum interval of the word elements of the filtered combined word index dictionary, and determine the combined word length of the combined word corresponding to each combined word ID;

[0128] S1044. Calculate the matching scores of the combined words corresponding to each combined word ID according to the combined word length, the number of word elements, and the minimum interval of the word elements to obtain a list of matching scores.

[0129] Specifically, calculate the matching scores of each combined word in G to obtain a list of matching scores S:

[0130] 1) Create a score list S, where each element is a binary tuple (combined word id, score s);

[0131] 2) Take out the key-value pairs of the dictionary G, where the key is the combined word id and the value is the dictionary g of "word element - matching queue". If the number of elements in the dictionary g is less than the number of word elements of the combined word itself, it means that the combined word does not appear completely in the long text, and the dictionary g is discarded;

[0132] 3) Use two cursors to represent the distance between two word elements in the combined word, continuously move and narrow the interval of the two cursors to find the minimum distance x of any word element in the combined word;

[0133] 4) Substitute x into the following sigmoid function to obtain the matching score s, and construct the binary tuple (combined word id, score s) and append it to the score list S;

[0134]

[0135] In the formula:

[0136] len(g) - the length of the combined word, for example, len("online|poker") = 4;

[0137] count(g) - the number of word elements of the combined word, for example, count("online|poker") = 2;

[0138] α - A preset parameter used to eliminate the influence of the number of tokens on the score.

[0139] 5) Continuously operate steps 2), 3) and 4) until all key-value pairs in the aggregation dictionary G are taken out, and the calculation of the combined word matching score is completed.

[0140] S105. Normalize the matching score list according to the text length and the combined word length to obtain the combined word matching result of the text to be detected.

[0141] Such as Figure 7 shown is another step flow chart of step S105 provided by the embodiment of the present invention. Referring to Figure 7 further as an optional implementation manner, the step of normalizing the matching score list according to the text length and the combined word length to obtain the combined word matching result of the text to be detected specifically includes:

[0142] S1051. Determine the text length of the text to be detected, and determine the first difference between the text length and the combined word length corresponding to each combined word ID;

[0143] S1052. Determine each combined word ID and the corresponding matching score according to the matching score list, and determine the corresponding normalized matching score according to the quotient of the matching score and the first difference;

[0144] S1053. Compare each normalized matching score with a preset first threshold. If the normalized matching score is less than the first threshold, remove the normalized matching score and the corresponding combined word ID to obtain the filtered normalized matching score;

[0145] S1054. Determine the combined word matching result of the text to be detected according to the filtered normalized matching score and the corresponding combined word ID.

[0146] Specifically, the normalization method is used to remove the interference of dynamic factors (text length, combined word length) on the matching result:

[0147] 1) Take out the binary group (combined word id, matching score s) from S and substitute it into the following formula to obtain the normalized score Norm(s);

[0148]

[0149] In the formula:

[0150] len(U) - The length of the text to be matched;

[0151] len(GroupText) - The sum of the lengths of all tokens of the combined word (combined word length).

[0152] 2) Compare Norm(s) with a preset threshold. If it is less than the threshold, determine that this compound word is a false positive match and remove it from S. Note that the preset threshold needs to be obtained according to different services and statistical results.

[0153] 3) According to the filtered normalized matching scores and the corresponding compound word IDs, the matching results of the compound words in the text to be detected can be output, and the matching scores of each matched compound word are marked.

[0154] The method steps of the embodiments of the present invention are described above. It can be understood that the embodiments of the present invention perform word element matching on the text to be detected through a word search tree, obtain a compound word index dictionary through aggregation processing, and then calculate the matching scores of each compound word based on the minimum interval of word elements, realizing the matching detection of compound words with complex structures in long texts and improving the efficiency of compound word matching detection; the matching score list is normalized based on the text length and the compound word length, reducing the influence of the compound word length and the number of compound words on the matching score and improving the accuracy of compound word matching detection.

[0155] Compared with the prior art, the embodiments of the present invention also have the following advantages:

[0156] 1) Add compound word information to the nodes of the traditional Trie matching tree, maximize the reuse of the word matching method, and with its excellent algorithm, high matching efficiency can be achieved, and at the same time, various heuristic methods for processing deformed words are reused;

[0157] 2) By using the Unicode Private Area encoding to encode pinyin as a single unicode character, an efficient method for combating pinyin deformation is supported, demonstrating the powerful expansion ability of the data structure in the present invention;

[0158] 3) Use the matching index queue to record all possible index combinations, and use an efficient cursor traversal method to find the minimum compound word interval, avoiding false positive matching results caused by simple "discovery and hit";

[0159] 4) The score calculation formula effectively reduces the influence of the compound word length and the number of compound words on the matching score, and uses the sigmoid function to fit the matching situation of the compound word interval in the real world;

[0160] 5) All steps are linear algorithm complexities, capable of supporting application scenarios with ultra-large scale and ultra-high concurrency.

[0161] As Figure 8 shown is the structural schematic diagram of the compound word matching detection system provided by the embodiments of the present invention. Referring to Figure 8 , the embodiments of the present invention provide a compound word matching detection system, including:

[0162] A relationship mapping table construction module, configured to obtain a preset first combined word, disassemble the first combined word to obtain a plurality of first word elements, and construct a first relationship mapping table according to the first word elements and the first combined word;

[0163] A word search tree generation module, configured to generate a word search tree according to the first relationship mapping table;

[0164] A word element matching module, configured to obtain a text to be detected, perform word element matching on the text to be detected according to the word search tree to obtain a list of matching records, and then perform aggregation processing on the list of matching records to obtain a combined word index dictionary;

[0165] A matching score calculation module, configured to determine the minimum interval of word elements in the combined word index dictionary, and calculate the matching scores of each combined word in the combined word index dictionary according to the minimum interval of word elements to obtain a list of matching scores;

[0166] A normalization processing module, configured to perform normalization processing on the list of matching scores according to the text length and the combined word length to obtain a combined word matching result of the text to be detected.

[0167] The content in the above method embodiments is applicable to the system embodiments of the present invention. The functions specifically implemented by the system embodiments of the present invention are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those of the above method embodiments.

[0168] An embodiment of the present invention further provides an electronic device, including: a memory, a processor, a program stored on the memory and executable on the processor, and a data bus for realizing connection communication between the processor and the memory. When the program is executed by the processor, the above combined word matching detection method is realized. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0169] As Figure 9 shown is a schematic hardware structure diagram of the electronic device provided by the embodiment of the present invention. Referring to Figure 9 , an embodiment of the present invention provides an electronic device, including:

[0170] A processor 901, which can be implemented by using a general-purpose CPU (Central Processing Unit, central processor), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, etc., and is configured to execute relevant programs to implement the technical solution provided by the embodiment of the present invention;

[0171] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), etc. The memory 902 can store an operating system and other application programs. When implementing the technical solutions provided in the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 902 and are called by the processor 901 to execute the combined word matching detection method of the embodiments of the present invention;

[0172] The input / output interface 903 is used to implement information input and output;

[0173] The communication interface 904 is used to implement communication interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or through wireless means (such as mobile network, WIFI, Bluetooth, etc.);

[0174] The bus 905 transmits information between various components of the device (such as the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0175] Among them, the processor 901, the memory 902, the input / output interface 903, and the communication interface 904 are communicatively connected to each other inside the device through the bus 905.

[0176] As Figure 10 shown is the structural schematic diagram of the storage medium provided by the embodiments of the present invention. Referring to Figure 10 , the embodiments of the present invention also provide a storage medium. The storage medium is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs 1001, and the one or more programs 1001 can be executed by one or more processors to implement the above-mentioned combined word matching detection method.

[0177] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include high-speed random access memory, and can also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory optionally includes a memory remotely disposed relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above networks include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0178] An embodiment of the present invention also discloses a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes Figure 1 the method shown.

[0179] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order mentioned in the operation diagrams. For example, depending on the functions / operations involved, two consecutive blocks shown may actually be executed substantially simultaneously or the above-mentioned blocks can sometimes be executed in the reverse order. In addition, the embodiments presented and described in the flowcharts of the present invention are provided by way of example for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operations and logical flows presented herein. Alternative embodiments are contemplated, in which the order of various operations is changed and the sub-operations described as part of a larger operation are executed independently.

[0180] Furthermore, although the present invention has been described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the above functions and / or features may be integrated in a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It can also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. More precisely, considering the attributes, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the modules will be understood within the ordinary skills of an engineer. Therefore, those skilled in the art can implement the present invention as set forth in the claims without undue experimentation. It can also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0181] If the above functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the above methods in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs.

[0182] The logic and / or steps represented in the flowchart or described in other ways herein, for example, can be considered as a definite sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch instructions from the instruction execution system, apparatus, or device and execute the instructions), or in combination with these instruction execution systems, apparatuses, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in combination with an instruction execution system, apparatus, or device.

[0183] More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection part (electronic device) having one or more wirings, a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, a computer-readable medium can even be paper or other suitable media on which the above program can be printed, because the above program can be obtained electronically, for example, by optically scanning the paper or other media, then editing, interpreting, or processing it in other suitable ways as necessary, and then storing it in a computer memory.

[0184] It should be understood that various parts of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits with logic gate circuits for implementing logic functions on data signals, application specific integrated circuits with suitable combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0185] In the foregoing description of this specification, descriptions with reference to the terms "one embodiment / example", "another embodiment / example", or "certain embodiments / examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0186] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the claims and their equivalents.

[0187] The above has specifically described the preferred embodiments of the present invention, but the present invention is not limited to the above embodiments. Those skilled in the art can also make various equivalent deformations or substitutions without departing from the spirit of the present invention, and these equivalent deformations or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A combined word matching detection method, characterized in that, It includes the following steps: Obtain a preset first combined word, disassemble the first combined word to obtain a plurality of first word elements, and construct a first relationship mapping table according to the first word elements and the first combined word; Generate a word search tree according to the first relationship mapping table; Obtain the text to be detected, perform word element matching on the text to be detected according to the word search tree to obtain a list of matching records, and then perform aggregation processing on the list of matching records to obtain a combined word index dictionary; Determine the minimum interval of word elements in the combined word index dictionary, and calculate the matching scores of each combined word in the combined word index dictionary according to the minimum interval of word elements to obtain a list of matching scores; Perform normalization processing on the list of matching scores according to the text length and the combined word length to obtain the combined word matching result of the text to be detected.

2. The combined word matching detection method according to claim 1, characterized in that The step of obtaining a preset first combined word, disassembling the first combined word to obtain a plurality of first word elements, and constructing a first relationship mapping table according to the first word elements and the first combined word specifically includes: Obtain a plurality of the first combined words from a preset combined word table, and determine the combined word IDs of each of the first combined words; Disassemble the first combined word according to a combined separator to obtain a plurality of the first word elements, and determine the combined word ID corresponding to each of the first word elements; Generate a first key-value pair according to the first word element and the combined word ID, and construct the first relationship mapping table according to the first key-value pair.

3. The method for detecting combined word matching according to claim 2, characterized in that, The step of generating a word search tree according to the first relationship mapping table specifically includes: Obtain the first key-value pair according to the first relationship mapping table, perform difference on the first word element of the first key-value pair according to characters and convert it into a Unicode encoding sequence; Determine a root node and child nodes according to the Unicode encoding sequence and the combined word ID of the first key-value pair; Generate the word search tree according to the root node and the child nodes.

4. A combined word matching detection method according to claim 3, characterized in that The step of performing word element matching on the text to be detected according to the word search tree to obtain a list of matching records specifically includes: Perform character encoding on the text to be detected to obtain a Unicode encoding array; Traverse the Unicode encoding array to obtain a plurality of word element encodings to be matched, and determine the combined word ID corresponding to the word element encoding to be matched according to the word search tree; Determine the corresponding second word element and matching index value according to the word element encoding to be matched, and then generate the list of matching records according to the second word element, the matching index value, and the corresponding combined word ID.

5. A method for detecting combined word matching according to claim 4, characterized in that The step of performing aggregation processing on the list of matching records to obtain a combined word index dictionary specifically includes: Obtain target word elements from the list of matching records, and determine a plurality of matching index values corresponding to each of the target word elements; Generate a second key-value pair according to the target word element and the corresponding plurality of matching index values; Determine a plurality of target word elements corresponding to each combined word ID, and construct a sub-dictionary according to the second key-value pairs generated by the plurality of target word elements corresponding to the same combined word ID; Generate a third key-value pair according to the combined word ID and the corresponding sub-dictionary, and construct the combined word index dictionary according to the third key-value pair.

6. The combination word matching detection method according to claim 5, wherein The step of determining the minimum interval of the word elements of the combined word index dictionary and calculating the matching scores of each combined word in the combined word index dictionary according to the minimum interval of the word elements to obtain a list of matching scores specifically includes: Determine the number of word elements of the combined word corresponding to each combined word ID, and determine the number of elements of the corresponding sub-dictionary for each combined word ID; If the number of elements is less than the number of word elements, remove the corresponding third key-value pair to obtain the filtered combined word index dictionary; Determine the minimum interval of the word elements of the filtered combined word index dictionary, and determine the length of the combined word corresponding to each combined word ID; Calculate the matching scores of the combined words corresponding to each combined word ID according to the combined word length, the number of word elements, and the minimum interval of the word elements to obtain the list of matching scores.

7. A method for detecting combined word matching according to claim 6, characterized in that The step of normalizing the list of matching scores according to the text length and the combined word length to obtain the combined word matching result of the text to be detected specifically includes: Determine the text length of the text to be detected, and determine the first difference between the text length and the combined word length corresponding to each combined word ID; Determine each combined word ID and the corresponding matching score according to the list of matching scores, and determine the corresponding normalized matching score according to the quotient of the matching score and the first difference; Compare each normalized matching score with a preset first threshold. If the normalized matching score is less than the first threshold, remove the normalized matching score and the corresponding combined word ID to obtain the filtered normalized matching score; Determine the combined word matching result of the text to be detected according to the filtered normalized matching score and the corresponding combined word ID.

8. A combined word matching detection system, characterized in that Including: A relationship mapping table construction module, configured to obtain a preset first combined word, disassemble the first combined word into multiple first word elements, and construct a first relationship mapping table according to the first word elements and the first combined word; A word search tree generation module, configured to generate a word search tree according to the first relationship mapping table; A word element matching module, configured to obtain the text to be detected, perform word element matching on the text to be detected according to the word search tree to obtain a list of matching records, and then perform aggregation processing on the list of matching records to obtain a combined word index dictionary; A matching score calculation module, configured to determine the minimum interval of the word elements of the combined word index dictionary, and calculate the matching scores of each combined word in the combined word index dictionary according to the minimum interval of the word elements to obtain a list of matching scores; A normalization processing module, configured to normalize the list of matching scores according to the text length and the combined word length to obtain the combined word matching result of the text to be detected.

9. An electronic device, characterized in that, The electronic device includes a memory, a processor, a program stored on the memory and executable on the processor, and a data bus for implementing connection communication between the processor and the memory. When the program is executed by the processor, the steps of the combined word matching detection method according to any one of claims 1 to 7 are implemented.

10. A storage medium, the storage medium being a computer-readable storage medium for computer-readable storage, characterized in that The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the combined word matching detection method according to any one of claims 1 to 7.

Images