Combined word matching detection method and system, electronic device and storage medium

Abstract

The present application relates to a combined word matching detection method and system and a medium. The method comprises: acquiring a preset first combined word, splitting the first combined word to obtain a plurality of first word elements, and, according to the first word elements and the first combined word, constructing a first relationship mapping table; according to the first relationship mapping table, generating a trie; acquiring a text to be subjected to detection, performing, according to the trie, word element matching on said text to obtain a matching record list, and performing aggregation processing on the matching record list to obtain a combined word index dictionary; determining a word element minimum interval of the combined word index dictionary, and, according to the word element minimum interval, calculating a matching score of each combined word in the combined word index dictionary, so as to obtain a matching score list; and, according to the text length and the combined word length, performing normalization processing on the matching score list, so as to obtain a combined word matching result of said text.

Description

Combination word matching detection method, system, electronic device and storage medium

[0001] Related applications

[0002] The present application claims priority to the Chinese patent application No. 202410431160.5, filed on April 10, 2024, and entitled "Combination word matching detection method, system, electronic device and storage medium", the contents of which are hereby incorporated by reference in its entirety. TECHNICAL FIELD

[0003] The present application relates to the technical field of text processing, and in particular to a combination word matching detection method, system, electronic device and storage medium. BACKGROUND

[0004] In the traditional text keyword matching detection method, a word table is usually used to construct a Trie tree and the matching is realized by word-by-word matching. Although this method performs well in efficiency, it cannot complete the matching of combination words. The method of realizing combination word matching based on set or bit operation is not only inefficient, but also has the following shortcomings, which further leads to poor accuracy: 1) unable to handle word deformation (such as Chinese character pinyin deformation, near sound deformation and insertion deformation); 2) unable to handle the distance relationship between word units in long text; 3) unable to eliminate the influence of the number or length of word units on matching.

[0005] SUMMARY

[0006] The present application provides a combination word matching detection method, system, electronic device and storage medium.

[0007] In one aspect, the present application provides a combination word matching detection method, comprising the following steps:

[0008] Obtaining a first combination word, disassembling the first combination word to obtain a plurality of first word units, and constructing a first relationship mapping table according to the first word units and the first combination word;

[0009] Generating a word lookup tree according to the first relationship mapping table;

[0010] Obtaining a to-be-detected text, performing word unit matching on the to-be-detected text according to the word lookup tree to obtain a matching record list, and then performing aggregation processing on the matching record list to obtain a combination word index dictionary;

[0011] Determining the minimum interval of word units of the combination word index dictionary, and calculating the matching scores of each combination word in the combination word index dictionary according to the minimum interval of word units to obtain a matching score list;

[0012] The matching score list is normalized according to the text length and the combined word length, to obtain a combined word matching result of the to-be-detected text.

[0013] In one embodiment, the first combined word is obtained, the first combined word is disassembled to obtain a plurality of first word units, and a first relationship mapping table is constructed according to the first word units and the first combined word, including:

[0014] A plurality of first combined words are obtained from a preset combined word table, and a combined word ID of each first combined word is determined;

[0015] The first combined word is disassembled according to a combined separator to obtain a plurality of first word units, and the combined word ID corresponding to each first word unit is determined;

[0016] A first key-value pair is generated according to the first word unit and the combined word ID, and the first relationship mapping table is constructed according to the first key-value pair.

[0017] In one embodiment, the word lookup tree is generated according to the first relationship mapping table, including:

[0018] The first key-value pair is obtained according to the first relationship mapping table, the first word unit of the first key-value pair is differentially processed according to characters and converted into a Unicode encoding sequence;

[0019] The tree root node and the child node are determined according to the Unicode encoding sequence and the combined word ID of the first key-value pair;

[0020] The word lookup tree is generated according to the tree root node and the child node.

[0021] In one embodiment, the word unit matching is performed on the to-be-detected text according to the word lookup tree, to obtain a matching record list, including:

[0022] The to-be-detected text is character-encoded to obtain a Unicode encoding array;

[0023] A plurality of to-be-matched word unit encodings are obtained by traversing the Unicode encoding array, and the combined word ID corresponding to the to-be-matched word unit encoding is determined according to the word lookup tree;

[0024] The second word unit and the matching index value corresponding to the to-be-matched word unit encoding are determined, and the matching record list is generated according to the second word unit, the matching index value, and the corresponding combined word ID.

[0025] In one of the embodiments, the aggregating the matching record list to obtain a combined word index dictionary comprises:

[0026] obtaining target word elements from the matching record list, and determining a plurality of matching index values corresponding to each of the target word elements;

[0027] generating a second key-value pair according to the target word element and the plurality of matching index values corresponding to the target word element;

[0028] determining a plurality of target word elements corresponding to each of the combined word IDs, and constructing a sub-dictionary according to the second key-value pairs generated by the plurality of target word elements corresponding to the same combined word ID;

[0029] generating a third key-value pair according to the combined word ID and the corresponding sub-dictionary, and constructing the combined word index dictionary according to the third key-value pair.

[0030] In one of the embodiments, the determining a word element minimum interval of the combined word index dictionary, and calculating a matching score of each combined word in the combined word index dictionary according to the word element minimum interval to obtain a matching score list comprises:

[0031] determining a number of word elements of a combined word corresponding to each of the combined word IDs, and determining a number of elements of the sub-dictionary corresponding to each of the combined word IDs;

[0032] if the number of elements is less than the number of word elements, removing the corresponding third key-value pair to obtain a filtered combined word index dictionary;

[0033] determining a word element minimum interval of the filtered combined word index dictionary, and determining a combined word length of a combined word corresponding to each of the combined word IDs;

[0034] calculating a matching score of a combined word corresponding to each of the combined word IDs according to the combined word length, the number of word elements, and the word element minimum interval to obtain the matching score list.

[0035] In one of the embodiments, the normalizing the matching score list according to the text length and the combined word length to obtain a combined word matching result of the to-be-detected text comprises:

[0036] determining a text length of the to-be-detected text, and determining a first difference value between the text length and a combined word length corresponding to each of the combined word IDs;

[0037] determining each combination word ID and the corresponding matching score according to the matching score list, and determining the corresponding normalized matching score according to the quotient of the matching score and the first difference value;

[0038] comparing each normalized matching score with a preset first threshold value, removing the normalized matching score and the corresponding combination word ID if the normalized matching score is less than the first threshold value, and obtaining the screened normalized matching score;

[0039] determining the combination word matching result of the text to be detected according to the screened normalized matching score and the corresponding combination word ID.

[0040] In another aspect, an embodiment of the present application provides a combination word matching detection system, comprising:

[0041] a relationship mapping table construction module configured to obtain a preset first combination word, disassemble the first combination word to obtain a plurality of first word units, and construct a first relationship mapping table according to the first word units and the first combination word;

[0042] a word lookup tree generation module configured to generate a word lookup tree according to the first relationship mapping table;

[0043] a word unit matching module configured to obtain a text to be detected, perform word unit matching on the text to be detected according to the word lookup tree, obtain a matching record list, and further perform aggregation processing on the matching record list to obtain a combination word index dictionary;

[0044] a matching score calculation module configured to determine a word unit minimum interval of the combination word index dictionary, and calculate a matching score of each combination word in the combination word index dictionary according to the word unit minimum interval to obtain a matching score list;

[0045] a normalization processing module configured to perform normalization processing on the matching score list according to a text length and a combination word length to obtain a combination word matching result of the text to be detected.

[0046] In another aspect, an embodiment of the present application provides an electronic device, which comprises a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection communication between the processor and the memory, and the program is executed by the processor to realize the combination word matching detection method as described above.

[0047] In another aspect, an embodiment of the present application also provides a storage medium, which is a computer readable storage medium, for computer readable storage, and the storage medium stores one or more programs, which can be executed by one or more processors to implement the combined word matching detection method as described above.

[0048] The details of one or more embodiments of the application are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the application will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following introduces the drawings needed to be used in the embodiments of the present application as follows. It should be understood that the drawings introduced in the following merely for the convenience of clearly describing some embodiments in the technical solutions of the present application, and for those skilled in the art, other drawings can also be obtained without paying creative labor on the premise of the drawings.

[0050] FIG. 1 is a step flow chart of a combined word matching detection method provided by an embodiment of the present application;

[0051] FIG. 2 is a step flow chart of step S101 provided by an embodiment of the present application;

[0052] FIG. 3 is a step flow chart of step S102 provided by an embodiment of the present application;

[0053] FIG. 4 is a step flow chart of step S103 provided by an embodiment of the present application;

[0054] FIG. 5 is another step flow chart of step S103 provided by an embodiment of the present application;

[0055] FIG. 6 is a step flow chart of step S104 provided by an embodiment of the present application;

[0056] FIG. 7 is a step flow chart of step S105 provided by an embodiment of the present application;

[0057] FIG. 8 is a structural schematic diagram of a combined word matching detection system provided by an embodiment of the present application;

[0058] FIG. 9 is a hardware structural schematic diagram of an electronic device provided by an embodiment of the present application;

[0059] FIG. 10 is a structural schematic diagram of a storage medium provided by an embodiment of the present application. DETAILED DESCRIPTION

[0060] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and are not to be construed as limitations on the present application. It should be noted that, although the functional modules are divided in the system schematic and the logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than the module division in the system schematic or the order in the flow chart. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0061] In the description of this application, the meaning of "a plurality" is two or more. If there is a description of "first" or "second", it is only for the purpose of distinguishing technical features and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features. In addition, unless otherwise defined, all technical and scientific terms used in this document have the same meaning as commonly understood by those skilled in the art to which this application belongs. The terms used in this document are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0062] Explanation of terms:

[0063] Trie tree: also known as word search tree, is a tree structure and a variant of hash tree. It is characterized by low memory consumption and super high matching performance.

[0064] Unicode encoding: Unicode has only one character set. Chinese, Japanese, and Korean occupy the Unicode range 0x3000 to 0x9FFF. Unicode currently uses the UCS-2 standard, which uses two bytes to encode a character. For example, the Chinese character for "经" (jing) is encoded as 0x7ECF.

[0065] Unicode Private Use Area: In Unicode, a Private Use Area (PUA) is a range of code points that, by definition, the Unicode Consortium does not assign characters to, so that third parties can define their own characters without conflicting with Unicode Consortium assignments. PUA defines three private use areas, one of which is in the Basic Multilingual Plane (U+E000–U+F8FF). Code points in these areas cannot be considered standardized characters in Unicode itself.

[0066] Sigmoid function: named after its graph resembling the letter S, often used to map real numbers to the interval (0, 1), characterized by rapid growth in a certain interval and saturation outside the interval.

[0067] Combination word: a sequence of words, for example, "online poker".

[0068] Combination word token: a word in a combination word, for example, the combination word is "online poker", and the token is "online" and "poker".

[0069] The combination word matching detection method provided by the embodiments of the present application can be applied in a terminal, can also be applied in a server, and can also be software running in the terminal or the server. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a set-top box, etc.; the server can be configured as a standalone physical server, can also be configured as a server cluster or a distributed system composed of multiple physical servers, can also be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform; and the software can be an application implementing the combination word matching detection method, but is not limited to the above forms.

[0070] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0071] It should be noted that in various specific embodiments of the present application, when relevant processing needs to be performed on data related to the identity or characteristics of the user, such as user information, user behavior data, user history data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards in the relevant countries and regions. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the separate permission or separate consent of the user will be obtained through a pop-up window or a jump to a confirmation page, and after obtaining the separate permission or separate consent of the user, the necessary user-related data for the normal operation of the embodiments of the present application will be obtained.

[0072] The method of the embodiments of the present application is based on word matching, and supports complex combined word matching detection at the cost of increasing very little memory and CPU consumption. By utilizing the word element matching queue and the sigmoid function statistical score, the embodiments of the present application realize efficient matching of complex structured combined words in text, and this innovative method not only improves the accuracy and speed of matching, but also significantly improves resource consumption, bringing a new solution to the field of combined word matching. Finally, in the process of constructing the word lookup tree (Trie tree), the embodiments of the present application use the Private Use Area of Unicode for Chinese pinyin mapping, support pinyin deformation lookup, and demonstrate the powerful extension capability based on the present application.

[0073] As shown in FIG. 1, it is a step flow chart of the combined word matching detection method provided by the embodiments of the present application. Referring to FIG. 1, the embodiments of the present application provide a combined word matching detection method, which specifically includes the following steps:

[0074] S101, obtaining a first set of combined words, disassembling the first set of combined words to obtain a plurality of first word elements, and constructing a first relationship mapping table according to the first word elements and the first set of combined words.

[0075] As shown in FIG. 2, it is a step flow chart of step S101 provided by the embodiments of the present application. Referring to FIG. 2, as a further optional implementation, the step of disassembling the first set of combined words to obtain a plurality of first word elements and constructing a first relationship mapping table according to the first word elements and the first set of combined words specifically includes:

[0076] S1011, obtaining a plurality of first combined words from a preset combined word table, and determining the combined word IDs of the first combined words;

[0077] S1012, disassembling the first combined words according to the combined separator to obtain a plurality of first word elements, and determining the combined word IDs corresponding to the first word elements;

[0078] S1013, generating a first key-value pair according to the first word item and the combined word ID, and constructing a first relationship mapping table according to the first key-value pair.

[0079] Specifically, a combined word table is obtained from a preset file system or database, for example, "online | poker", "online | loan", "organ | trafficking", "human | trafficking", wherein the symbol "|" is a combined separator, and the first combined word can be disassembled into multiple first word items through the combined separator.

[0080] A "pinyin-Unicode" relationship mapping table Y is constructed:

[0081] 1) A dictionary is created with a pinyin character sequence as the key and Unicode encoding as the value;

[0082] 2) The pinyin string itself is taken as the key, and E000+size (dictionary) is taken as the value, and the "pinyin-Unicode" relationship mapping is added to the dictionary;

[0083] 3) Repeat 2) until all Chinese pinyins are traversed.

[0084] A "word item (including pinyin deformation)-combined word id" relationship mapping table (i.e., the first relationship mapping table) is constructed:

[0085] 1) A dictionary is created with a word item as the key and a combined word id list as the value;

[0086] 2) The combined word (assuming id=x) is disassembled into word items according to the separator "|", the dictionary in 1 is traversed, and the current id=x is appended to the value of the dictionary;

[0087] 3) The Chinese characters in the word item are converted into pinyin, and then the above step 2) is repeated;

[0088] 4) Continuously repeat operations 2) and 3) until all combined words are traversed, and finally the dictionary in 1 is the "word item-combined word id" relationship mapping table, for example, there are two combined words "online | poker" (id=1) and "online | loan" (id=2), and finally the relationship mapping table "{online: [1, 2], poker: [1], loan: [2]}" is obtained.

[0089] S102, generating a word lookup tree according to the first relationship mapping table.

[0090] As shown in FIG. 3 is a step flow chart of step S102 provided by the embodiments of the present application, with reference to FIG. 3, further as an optional implementation manner, according to the first relationship mapping table to generate a word lookup tree, which specifically includes:

[0091] S1021, obtain a first key-value pair according to the first relationship mapping table, and differentially process the first word element of the first key-value pair according to characters and convert the first word element into a Unicode encoding sequence;

[0092] S1022, determine a tree root node and a child node according to the Unicode encoding sequence and a combination word ID of the first key-value pair;

[0093] S1023, generate a word lookup tree according to the tree root node and the child node.

[0094] Specifically, the first relationship mapping table obtained in the foregoing steps is converted into a word lookup tree (Trie tree):

[0095] 1) create a tree node data structure - a binary tuple (value, children), wherein: the value is consistent with the value of the first relationship mapping table, and is a combination word id list; the children is a dictionary with Unicode encoding as the key and a child node as the value;

[0096] 2) create a Trie tree root node Root (Trie) as a starting lookup node - (value = empty, children = {});

[0097] 3) take the key-value pair (assuming key = online, id = [1, 2]) of the "word element - combination word id" relationship mapping table obtained in the foregoing steps, split the key according to single characters and convert it into a Unicode encoding sequence U (for example, [7ebf (line), 4e0a (up)]);

[0098] 4) set a node pointer P to point to the root node Root (Trie), a character index is C (initially 1), and it is determined whether P.children[U[C]] exists, if not, a tree node is created and added to P.children, if yes, P is changed to point to the child node (i.e. P = P.children[U[C]]);

[0099] 5) advance C by one index (i.e. C = C + 1) and repeat operation 4), until C is greater than the word element character length, at which time the combination word id array corresponding to the word element is appended to P.value;

[0100] 6) repeat operations 3), 4) and 5) until all key-value pairs of the relationship mapping table are processed, and finally the root node Root (Trie) in 2) represents the word lookup tree (Trie tree).

[0101] S103, obtain a to-be-detected text, perform word element matching on the to-be-detected text according to the word lookup tree, obtain a matching record list, and then perform aggregation processing on the matching record list to obtain a combination word index dictionary.

[0102] As shown in Figure 4 is the step S103 of the embodiment of the application provides a step flow chart, referring to Figure 4, further as an optional implementation, according to the word lookup tree for text to be detected word matching, get matching record list this step, it specifically includes:

[0103] S1031, the text to be detected is character coded, and a Unicode encoding array is obtained;

[0104] S1032, a plurality of to-be-matched word element encodings are obtained by traversing the Unicode encoding array, and the to-be-matched word element encodings corresponding combination word IDs are determined according to the word lookup tree;

[0105] S1033, the corresponding second word element and matching index value are determined according to the to-be-matched word element encoding, and then the matching record list is generated according to the second word element, the matching index value and the corresponding combination word ID.

[0106] Specifically, the word lookup tree (Trie tree) obtained by using the foregoing steps is used to match the long text, and the matching record list M of the word element is obtained:

[0107] 1) Create a word element matching record list M, each element of which is a triple (word element, matching index, combination word id list), wherein the matching index indicates the position of the word element in the long text;

[0108] 2) Assuming that the characters in the long text have been encoded as Unicode, the entire long text can be regarded as a Unicode encoding array U, and at this time, the text cursor C points to the i-th (initialized to 1) encoding of the text, and the Trie tree node pointer P points to the root node Root (Trie);

[0109] 3) Find whether P.children contains an element with key U[C], if yes, point the pointer P to it (i.e. P = P.children[U[C]]). If P.value is not empty, it indicates that the current encoding sequence has matched a word element, and a triple (U[i..C], i, P.value) needs to be constructed and added to the matching record list M;

[0110] 4) If U[C] belongs to the English character encoding range, record the pinyin starting cursor K = C, and in each subsequent matching, judge whether U[C] belongs to the English character encoding category, and try to find the pinyin Unicode encoding y = Y[U[K..C]] in the mapping table obtained in step 1, if it exists, regard y as a single character encoding and match it in the method of 3);

[0111] 5) advance C by one index (i.e. C=C+1) and repeat operation 2) until there is no corresponding Unicode encoding for a node's children in the Trie tree, i.e. P.chilren[U[C]]=empty;

[0112] 6) advance i by one index (i.e. i=i+1), i.e. point to the next character of the long text. Reset the text cursor C to point to i, P to point to the root node, and repeat operations 2), 3), 4) and 5) until i is greater than the length of the text.

[0113] As shown in Figure 5 is another step flow chart of step S103 provided by the embodiment of the application, referring to Figure 5, further as an optional implementation, the step of aggregating the matching record list to obtain the combined word index dictionary, which specifically includes:

[0114] S1034, obtaining target word elements from the matching record list, and determining a plurality of matching index values corresponding to each target word element;

[0115] S1035, generating a second key-value pair according to the target word element and the corresponding plurality of matching index values;

[0116] S1036, determining a plurality of target word elements corresponding to each combined word ID, and constructing a sub-dictionary according to the second key-value pairs generated by the plurality of target word elements corresponding to the same combined word ID;

[0117] S1037, generating a third key-value pair according to the combined word ID and the corresponding sub-dictionary, and constructing a combined word index dictionary according to the third key-value pair.

[0118] Specifically, the matching record list M obtained by the foregoing steps is aggregated according to the combined word id to obtain the combined word index dictionary G:

[0119] 1) Create a combined word index dictionary G, which is a two-level dictionary structure, where the key is the combined word id, and the value is the sub-dictionary g. The key of g is the word element, and the value is the matching index queue. The structure can be referred to in Figure 3, for example, G=" {1:{online:[4,13], poker:[6]}} ", which means that in the combined word id=1 group, the word "online" appears at indexes 4 and 13, and the word "poker" appears at index 6;

[0120] 2) Take out the triplets Mi of the matching record list M in the foregoing steps. Traverse the combined word id list in Mi, find the sub-dictionary g in the combined word index dictionary G, and further find the matching index queue using the word element recorded in Mi to obtain g, and finally append the matching index recorded in Mi to it. It should be noted that since the matching index in the matching record list M is definitely increasing, any index queue in g is also definitely increasing;

[0121] 3) Repeat 2) until all matching records are retrieved.

[0122] S104, determine the minimum interval of the word units of the combined word index dictionary, and calculate the matching scores of each combined word in the combined word index dictionary according to the minimum interval of the word units, to obtain a matching score list.

[0123] As shown in FIG. 6 is a step flow chart of step S104 provided by the embodiment of the present application, with reference to FIG. 6, further as an optional implementation, the step of determining the minimum interval of the word units of the combined word index dictionary, and calculating the matching scores of each combined word in the combined word index dictionary according to the minimum interval of the word units, to obtain a matching score list, specifically includes:

[0124] S1041, determine the number of word units of the combined word corresponding to each combined word ID, and determine the number of elements of the sub-dictionary corresponding to each combined word ID;

[0125] S1042, if the number of elements is less than the number of word units, remove the corresponding third key-value pair to obtain a screened combined word index dictionary;

[0126] S1043, determine the minimum interval of the word units of the screened combined word index dictionary, and determine the length of the combined word of the combined word corresponding to each combined word ID;

[0127] S1044, calculate the matching score of the combined word corresponding to each combined word ID according to the length of the combined word, the number of word units and the minimum interval of the word units, to obtain a matching score list.

[0128] Specifically, calculate the matching score of each combined word in G to obtain a matching score list S:

[0129] 1) Create a score list S, where each element is a two-tuple (combined word id, score s);

[0130] 2) Take out the key-value pair of the dictionary G, whose key is the combined word id, and the value is the "word unit-matching queue" dictionary g, if the number of elements of the dictionary g is less than the number of word units of the combined word itself, it means that the combined word does not appear completely in the long text, and the dictionary g is discarded;

[0131] 3) Use a double cursor to represent the distance between two word units in the combined word, and constantly move and narrow the interval of the double cursor to find the minimum distance x of any word unit in the combined word;

[0132] 4) Substitute x into the following sigmoid function to obtain the matching score s, and construct a two-tuple (combined word id, score s) to append to the score list S;

[0133] In the formula:

[0134] len(g) - length of the combination word, e.g., len("online | poker") = 4;

[0135] count(g) - number of word units of the combination word, e.g., count("online | poker") = 2;

[0136] a - a preset parameter used to eliminate the influence of the number of word units on the score.

[0137] 5) continuously operate 2), 3) and 4) until all key-value pairs in the aggregated dictionary G are taken out, and the combination word matching score calculation is completed.

[0138] S105, normalizing the matching score list according to the text length and the combination word length to obtain the combination word matching result of the text to be detected.

[0139] As shown in FIG. 7 is another step flow chart of step S105 provided by the embodiment of the application, referring to FIG. 7, further as an optional implementation manner, the step of normalizing the matching score list according to the text length and the combination word length to obtain the combination word matching result of the text to be detected, specifically includes:

[0140] S1051, determining the text length of the text to be detected, and determining a first difference between the text length and the combination word length corresponding to each combination word ID;

[0141] S1052, determining each combination word ID and the corresponding matching score according to the matching score list, and determining the corresponding normalized matching score according to the quotient of the matching score and the first difference;

[0142] S1053, comparing each normalized matching score with a preset first threshold value, if the normalized matching score is less than the first threshold value, removing the normalized matching score and the corresponding combination word ID to obtain the screened normalized matching score;

[0143] S1054, determining the combination word matching result of the text to be detected according to the screened normalized matching score and the corresponding combination word ID.

[0144] Specifically, the dynamic factors (text length, combination word length) are removed by the normalization method to interfere with the matching result:

[0145] 1) taking the binary tuple (combination word id, matching score s) from S, and substituting it into the following formula to obtain the normalized score Norm(s);

[0146] In the formula:

[0147] len(U) - length of the matched text;

[0148] len(GroupText) - sum of all wordpiece lengths in the group word (group word length).

[0149] 2) Compare Norm(s) with a preset threshold value, if less than the threshold value, determine that the group word is a false positive match, remove it from S, note that the preset threshold value needs to be obtained according to different services and statistical results.

[0150] 3) According to the normalized matching score and the corresponding group word ID after screening, the group word matching result of the text to be detected can be output, and the matching score of each group word matched is marked.

[0151] The method steps of the embodiments of the present application are described above. It can be understood that the embodiments of the present application perform wordpiece matching on the text to be detected through the word search tree, and obtain the group word index dictionary through aggregation processing, and then calculate the matching score of each group word based on the minimum interval of the wordpiece, realize the matching detection of the complex structure group word in the long text, and improve the efficiency of the group word matching detection; The matching score list is normalized based on the text length and the group word length, which reduces the influence of the group word length and the group word number on the matching score, and improves the accuracy of the group word matching detection.

[0152] Compared with the traditional technology, the embodiments of the present application also have the following advantages:

[0153] 1) Increase the group word information in the node of the traditional Trie matching tree, maximize the reuse of the word matching method, and use the excellent algorithm to realize high matching efficiency, and reuse various heuristic methods for processing deformation words;

[0154] 2) By using Unicode Private Area encoding pinyin as a single unicode character, the pinyin deformation confrontation method is efficiently supported, and the strong expansion ability of the data structure in the present application is displayed;

[0155] 3) Use the matching index queue to record all possible index combinations, and use the efficient cursor traversal method to find the smallest group word interval, which avoids the false positive matching results caused by the simple "discovery and hit";

[0156] 4) The score calculation formula effectively reduces the influence of the group word length and the group word number on the matching score, and uses the sigmoid function to fit the matching situation of the group word interval in the real world;

[0157] 5) All steps are linear level algorithm complexity, which can support super large scale and super high concurrency application scenarios.

[0158] As shown in Figure 8 is the structure schematic diagram of the combined word matching detection system provided by the embodiment of the application, referring to Figure 8, the embodiment of the application provides a combined word matching detection system, comprising:

[0159] The relationship mapping table construction module is configured to obtain a preset first combined word, disassemble the first combined word to obtain a plurality of first word units, and construct a first relationship mapping table according to the first word units and the first combined word;

[0160] The word lookup tree generation module is configured to generate a word lookup tree according to the first relationship mapping table;

[0161] The word unit matching module is configured to obtain a to-be-detected text, perform word unit matching on the to-be-detected text according to the word lookup tree, obtain a matching record list, and then perform aggregation processing on the matching record list to obtain a combined word index dictionary;

[0162] The matching score calculation module is configured to determine a word unit minimum interval of the combined word index dictionary, and calculate matching scores of each combined word in the combined word index dictionary according to the word unit minimum interval to obtain a matching score list;

[0163] The normalization processing module is configured to perform normalization processing on the matching score list according to the text length and the combined word length to obtain a combined word matching result of the to-be-detected text.

[0164] The content in the above method embodiment is applicable to the system embodiment, the system embodiment specifically implements the same functions as the above method embodiment, and achieves the same beneficial effects as the above method embodiment.

[0165] The embodiment of the application further provides an electronic device, which comprises a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the combined word matching detection method. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0166] As shown in Figure 9 is the hardware structure schematic diagram of the electronic device provided by the embodiment of the application, referring to Figure 9, the embodiment of the application provides an electronic device, comprising:

[0167] The processor 901 can be implemented in a general-purpose CPU (Central Processing Unit, central processor), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is used to execute a related program to realize the technical solutions provided by the embodiment of the application;

[0168] The memory 902 can be implemented in the form of a Read Only Memory (ROM), a static storage device, a dynamic storage device, or a Random Access Memory (RAM), etc. The memory 902 can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present specification are implemented by software or firmware, the related program codes are stored in the memory 902 and are called and executed by the processor 901 to perform the combined word matching detection method of the embodiments of the present application.

[0169] The input / output interface 903 is configured to realize information input and output.

[0170] The communication interface 904 is configured to realize the communication interaction between the device and other devices, and the communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0171] The bus 905 is configured to transmit information between various components (for example, the processor 901, the memory 902, the input / output interface 903, and the communication interface 904) of the device.

[0172] The processor 901, the memory 902, the input / output interface 903, and the communication interface 904 are connected to each other through the bus 905 to realize the communication connection between the device.

[0173] As shown in FIG. 10, the present embodiment provides a structure diagram of a storage medium. Referring to FIG. 10, the present embodiment further provides a storage medium, which is a computer readable storage medium, for computer readable storage. The storage medium stores one or more programs 1001, and the one or more programs 1001 can be executed by one or more processors to implement the combined word matching detection method.

[0174] The memory is a non-transitory computer readable storage medium, which can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0175] The embodiment of the present application further discloses a computer program product or computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method shown in FIG. 1.

[0176] In some alternative embodiments, the functions / operations mentioned in the block diagrams can not occur in the order mentioned in the operation diagram. For example, depending on the functions / operations involved, two blocks shown in succession can actually be executed substantially concurrently or the blocks mentioned above can be executed in reverse order at times. In addition, the embodiments presented and described in the flowcharts of the present application are provided by way of example, and the purpose is to provide a more comprehensive understanding of the technology. The disclosed method is not limited to the operations and logical flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and in which sub-operations described as part of larger operations are independently executed.

[0177] In addition, although the present application is described in the context of functional modules, it should be understood that one or more of the functions and / or features described above can be integrated in a single physical device and / or software module, or one or more functions and / or features can be implemented in separate physical devices or software modules, unless otherwise specified. It can also be understood that detailed discussion of the actual implementation of each module is unnecessary for understanding the present application. Rather, given the properties, functions and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the routine skill of an engineer. Therefore, those skilled in the art can implement the present application as set forth in the claims without undue experimentation using ordinary skill. It can also be understood that the disclosed specific concepts are merely illustrative and are not intended to limit the scope of the present application, which is determined by the full scope of the appended claims and their equivalents.

[0178] If the above functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods according to the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0179] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a list of executable instructions for implementing logic functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device, such as a computer-based system, a system including a processor, or other system that can fetch instructions from the instruction execution system, apparatus or device and execute the instructions, or in conjunction with these instructions. For the purpose of the present specification, the "computer-readable medium" can be any device that can contain, store, communicate, propagate or transport programs for use by an instruction execution system, apparatus or device, or in conjunction with these instruction execution systems, apparatus or devices.

[0180] More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection having one or more wires (electrical devices), a portable computer diskette (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium can even be paper or other suitable medium on which the aforementioned programs can be printed, because the programs can be obtained electronically, for example, by optical scanning of the paper or other medium, followed by editing, interpreting or otherwise processing, if necessary, in other suitable ways, to be stored in the computer memory.

[0181] It should be understood that portions of the application can be realized with a hardware, software, firmware or a combination thereof. In the above-described embodiments, a plurality of steps or methods can be realized with software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if realized with hardware, and as in another embodiment, any one or a combination of the following technologies known in the art can be used: discrete logic circuitry having logic gates for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and the like.

[0182] In the above description of the present specification, the description of the terms "one embodiment", "another embodiment" or "certain embodiments" or the like means that the specific features, structures, materials or characteristics described in connection with the embodiments or examples are included in at least one embodiment or example of the present application. In the present specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0183] The technical features of the above-described embodiments can be combined in any manner. In order to make the description concise, all possible combinations of the technical features in the above-described embodiments are not described, however, as long as the combinations of the technical features do not contradict each other, they should be considered within the scope of the present specification.

[0184] The above-described embodiments only express several embodiments of the present application, and the description is specific and detailed, but it should not be understood as limiting the scope of the patent application. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application. Therefore, the scope of the patent of the present application should be subject to the appended claims.

Claims

1. A combination word matching detection method, comprising: Obtaining a preset first combination word, decomposing the first combination word to obtain a plurality of first word-units, and constructing a first relationship mapping table based on the first word-units and the first combination word; Generate a word search tree according to the first relationship mapping table; Acquire a text to be detected, perform word unit matching on the text to be detected according to the word search tree to obtain a matching record list, and then aggregate the matching record list to obtain a combined word index dictionary; Determining the minimum interval of word units in the combined word index dictionary, and calculating the matching score of each combined word in the combined word index dictionary according to the minimum interval of word units to obtain a matching score list; The matching score list is normalized according to the text length and the combination word length to obtain the combination word matching result of the text to be detected.

2. A combination word matching detection method according to claim 1, wherein: The obtaining of a preset first combination word, decomposing the first combination word to obtain a plurality of first word-units, and constructing a first relationship mapping table according to the first word-units and the first combination word, includes: Obtaining a plurality of first combination words from a preset combination word table, and determining a combination word ID of each of the first combination words; Decomposing the first combined word according to the combination separator to obtain a plurality of first word-units, and determining the combined word ID corresponding to each of the first word-units; A first key-value pair is generated according to the first word element and the combination word ID, and the first relationship mapping table is constructed according to the first key-value pair.

3. A combination word matching detection method according to claim 2, wherein: Generating a word search tree according to the first relationship mapping table includes: Obtain the first key-value pair according to the first relationship mapping table, perform character differentiation on the first word of the first key-value pair and convert the result into a Unicode encoding sequence; Determine a tree root node and child nodes according to the Unicode encoding sequence and the combined word ID of the first key-value pair; The word search tree is generated according to the tree root node and the child nodes.

4. A combination word matching detection method according to claim 3, wherein: The step of performing word element matching on the text to be detected according to the word search tree to obtain a matching record list includes: Performing character encoding on the text to be detected to obtain a Unicode encoding array; Traversing the Unicode encoding array to obtain a plurality of to-be-matched word element encodings, and determining the combination word ID corresponding to the to-be-matched word element encodings according to the word search tree; The corresponding second word and matching index value are determined according to the word-unit code to be matched, and the matching record list is generated according to the second word, the matching index value and the corresponding combination word ID.

5. A combination word matching detection method according to claim 4, wherein: The aggregating process is performed on the matching record list to obtain a combination word index dictionary, including: Obtaining target word-units from the matching record list, and determining a plurality of matching index values ​​corresponding to each target word-unit; Generate a second key-value pair according to the target word and the corresponding plurality of matching index values; Determine a plurality of target word units corresponding to each of the combination word IDs, and construct a sub-dictionary based on the second key-value pairs generated from the plurality of target word units corresponding to the same combination word ID; A third key-value pair is generated according to the combination word ID and the corresponding sub-dictionary, and the combination word index dictionary is constructed according to the third key-value pair.

6. A combination word matching detection method according to claim 5, wherein: The determining of the minimum interval of word units of the combined word index dictionary and calculating the matching score of each combined word in the combined word index dictionary according to the minimum interval of word units to obtain a matching score list includes: Determine the number of word units of the combination word corresponding to each combination word ID, and determine the number of elements of the sub-dictionary corresponding to each combination word ID; If the number of elements is less than the number of word units, the corresponding third key-value pair is removed to obtain the filtered combined word index dictionary; Determine the minimum word interval of the filtered combination word index dictionary, and determine the combination word length of the combination word corresponding to each combination word ID; The matching score of the combination word corresponding to each combination word ID is calculated according to the combination word length, the word unit number and the word unit minimum interval to obtain the matching score list.

7. A combination word matching detection method according to claim 6, wherein: The normalizing process of the matching score list according to the text length and the combination word length to obtain the combination word matching result of the text to be detected includes: Determining the text length of the text to be detected, and determining a first difference between the text length and the length of the combination word corresponding to each of the combination word IDs; Determining each of the combination word IDs and the corresponding matching scores according to the matching score list, and determining a corresponding normalized matching score according to a quotient of the matching score and the first difference; Compare each normalized matching score with a preset first threshold value. If the normalized matching score is less than the first threshold value, remove the normalized matching score and the corresponding combination word ID to obtain the filtered normalized matching score. Match score; The combination word matching result of the text to be detected is determined according to the filtered normalized matching score and the corresponding combination word ID.

8. A combination word matching detection system, comprising: a relationship mapping table construction module, configured to obtain a preset first combination word, decompose the first combination word to obtain a plurality of first word-units, and construct a first relationship mapping table based on the first word-units and the first combination word; A word search tree generating module, configured to generate a word search tree according to the first relationship mapping table; A word unit matching module is used to obtain a text to be detected, perform word unit matching on the text to be detected according to the word search tree, obtain a matching record list, and then perform aggregation processing on the matching record list to obtain a combined word index dictionary; a matching score calculation module, configured to determine a minimum word-unit interval of the combination word index dictionary, and calculate a matching score for each combination word in the combination word index dictionary based on the minimum word-unit interval to obtain a matching score list; The normalization processing module is used to perform normalization processing on the matching score list according to the text length and the combination word length to obtain the combination word matching result of the text to be detected.

9. An electronic device comprising a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, wherein when the program is executed by the processor, the steps of the combination word matching detection method as described in any one of claims 1 to 7 are realized.

10. A storage medium, which is a computer-readable storage medium used for computer-readable storage, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the combination word matching detection method as described in any one of claims 1 to 7.

Images