Method and device for extracting concerned phrases in text, storage medium and equipment

Abstract

The invention discloses a method and device for extracting concerned phrases in a text, a storage medium and equipment, and belongs to the technical field of computers. The method comprises the following steps: acquiring a reverse index and a multimode matching model which are constructed for concerned phrases; detecting an input text by utilizing the multi-mode matching model to obtain a plurality of fragments; collecting the plurality of fragments by using an inverted index to obtain a first fragment set; segmenting the first fragment set according to a preset editing distance to obtain multiple groups of second fragment sets; for each group of second fragment sets, screening candidate fragments according to editing distances between fragments in the second fragment sets and the concerned phrases; and combining the candidate fragments in each group of the second fragment sets into similar variants of the concerned phrases, and identifying the similar variants of which the editing distances to the concerned phrases are greater than a predetermined threshold value as the concerned phrases. According to the method, the similar variants of the concerned phrases in the text can be identified as the concerned phrases by utilizing the improved multi-mode detection model and the inverted index, so that the extraction accuracy is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and particularly to a method, apparatus, storage medium, and device for extracting phrases of interest in text. Background Art

[0002] In the broad field of Natural Language Processing (NLP), extracting and accurately identifying similar variants of phrases of interest in text has become a crucial and challenging task. Especially in the field of government text monitoring, every slogan, every sensitive phrase, and every speech by leaders carry a high degree of rigor that allows no margin of error. Even expressions with similar meanings must maintain the original words, which makes quickly and precisely capturing various similar variants of phrases of interest in text a key aspect in enhancing system efficiency. For example, the phrase "Zhonghua people's Republic of China" in the text needs to be recognized as "People's Republic of China".

[0003] In related technologies, we can construct a thesaurus or synonym dictionary for phrases of interest and match the similar variants in the text with the thesaurus or synonym dictionary, so as to recognize the similar variants as the correct phrases of interest.

[0004] However, the thesaurus or synonym dictionary is compiled manually, with a limited coverage and its update speed being difficult to keep up with the dynamic development of the language, resulting in the inability to comprehensively identify the phrases of interest in the text. Summary of the Invention

[0005] This application provides a method, apparatus, storage medium, and device for extracting phrases of interest in text, which is used to solve the problem that when using a thesaurus or synonym dictionary, the similar variants in the text cannot be comprehensively recognized as the correct phrases of interest. The technical solutions are as follows:

[0006] According to the first aspect of this application, a method for extracting phrases of interest in text is provided. The method includes:

[0007] Obtain an inverted index and a multi-pattern matching model constructed for the phrases of interest, where the multi-pattern matching model is generated according to the phrases of interest, the similar character set of the phrases of interest, and the ignored character set;

[0008] Use the multi-pattern matching model to detect the input text to obtain multiple fragments;

[0009] Use the inverted index to collect the multiple fragments to obtain a first fragment set;

[0010] According to a preset edit distance, split the first fragment set to obtain multiple groups of second fragment sets;

[0011] For each set of second fragments, screening candidate fragments according to the edit distance between the fragments in the second fragment set and the focus phrase;

[0012] The candidate fragments in each group of the second fragment set are combined into similar variants of the focus phrase, and the similar variants whose edit distance with the focus phrase is greater than a predetermined threshold are identified as the focus phrase.

[0013] In a possible implementation, the multi-modal matching model is used to detect the input text to obtain multiple fragments, including:

[0014] Get the i-th character in the input text, where i is a positive integer;

[0015] If the i-th character is an ignored character in the ignored character set, then i is updated to i+1, and the step of obtaining the i-th character in the input text is continued;

[0016] If the i-th character is not the ignored character, searching for all similar characters of the i-th character in the similar character set, matching the i-th character and all similar characters with the dictionary tree in the multi-mode matching model, and generating fragments according to the matching results;

[0017] The fragments include phrases extracted from the text, the starting position and the ending position of the phrases in the text, and phrases matched in the dictionary tree according to the extracted phrases.

[0018] In a possible implementation, screening candidate fragments according to the edit distance between the fragments in the second fragment set and the focus phrase includes:

[0019] Selecting m consecutive fragments from the n fragments of the second fragment set, 1≤m≤n;

[0020] Determine candidate phrases according to the starting position of the first fragment and the ending position of the mth fragment among the m fragments, and retain candidate phrases whose edit distance with the focus phrase is less than a predetermined threshold;

[0021] Among all the obtained candidate phrases, the candidate phrases with the longest length and non-overlapping start and end positions are selected, and the fragments corresponding to the candidate phrases are determined as candidate fragments.

[0022] In a possible implementation, combining the candidate fragments in each group of the second fragment set into similar variants of the focus phrase includes:

[0023] For each group of second fragment sets, each candidate fragment in the second fragment set is combined according to the starting position and the ending position to obtain a similar variant of the focus phrase.

[0024] In a possible implementation, the step of grouping the plurality of fragments by using the inverted index to obtain a first fragment set includes:

[0025] Determine the focus phrase to which each fragment belongs using the inverted index;

[0026] Extracting the first character and the last character from the focus phrase, and supplementing the fragments of the first character and the fragments of the last character;

[0027] Among all the obtained fragments, the fragments with the longest length and non-overlapping starting and ending positions are selected to obtain the first fragment set.

[0028] In a possible implementation, identifying a similar variant having an edit distance from the focus phrase greater than a predetermined threshold as the focus phrase includes:

[0029] Calculate the edit distance between the similar variants and the focus phrase;

[0030] If the edit distance is less than a predetermined threshold, the similar variant is identified as the focus phrase.

[0031] In a possible implementation, the step of obtaining an inverted index and a multi-mode matching model constructed for a focus phrase includes:

[0032] Splitting the focus phrases, and constructing an inverted index according to the splitting results;

[0033] Obtaining a similar character set for each character in the focus phrase, wherein the similar character set includes at least one of a homophone character set, a phonetically similar character set, and a similar-looking character set;

[0034] Obtain an ignored character set, wherein the ignored character set includes at least one of a number set and a label set, and the label set is a label that needs to be ignored and is set according to an application field;

[0035] A multi-modal matching model is constructed according to the similar character set and the ignored character set.

[0036] According to a second aspect of the present application, a device for extracting a focus phrase in a text is provided, the device comprising:

[0037] An acquisition module, used for acquiring an inverted index and a multi-mode matching model constructed for a focus phrase, wherein the multi-mode matching model is generated according to the focus phrase, a similar character set of the focus phrase, and an ignored character set;

[0038] A detection module, used to detect the input text using the multi-mode matching model to obtain multiple fragments;

[0039] A collection module, configured to collect the plurality of fragments by using the inverted index to obtain a first fragment set;

[0040] A segmentation module, configured to segment the first fragment set according to a preset edit distance to obtain multiple groups of second fragment sets;

[0041] A screening module, configured to screen candidate fragments for each set of second fragments according to the edit distance between the fragments in the second fragment set and the focus phrase;

[0042] The identification module is used to combine the candidate fragments in each group of the second fragment set into similar variants of the focus phrase, and identify the similar variants whose edit distance with the focus phrase is greater than a predetermined threshold as the focus phrase.

[0043] According to a third aspect of the present application, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement the method for extracting focus phrases in a text as described above.

[0044] According to a fourth aspect of the present application, a computer device is provided, the computer device comprising a device for extracting focus phrases from the above text.

[0045] The beneficial effects of the technical solution provided by this application include at least:

[0046] An improved multimodal detection model is generated according to a focus phrase, a similar character set of the focus phrase and an ignored character set, and the input text is detected using the multimodal matching model to obtain multiple fragments; the multiple fragments are then grouped using the inverted index of the focus phrase to obtain a first fragment set; then, the first fragment set is segmented according to a preset edit distance to obtain multiple groups of second fragment sets; for each group of second fragment sets, candidate fragments are screened according to the edit distance between the fragments in the second fragment set and the focus phrase; the candidate fragments in each group of second fragment sets are combined into similar variants of the focus phrase, and similar variants whose edit distance with the focus phrase is greater than a predetermined threshold are identified as the focus phrase, thereby improving the accuracy of extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0048] Figure 1 is a flowchart of a method for extracting focus phrases in a text provided by an embodiment of the present application;

[0049] Figure 2 is a flowchart of a method for extracting focus phrases in a text provided by an embodiment of the present application;

[0050] Figure 3 It is a schematic diagram of extracting a focus phrase instance provided by an embodiment of the present application;

[0051] Figure 4 It is a structural block diagram of a device for extracting focus phrases in a text provided by an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0053] like Figure 1 As shown, it shows a method flow chart of a method for extracting a focus phrase in a text provided by an embodiment of the present application, and the method for extracting a focus phrase in a text can be applied to a computer device. The method for extracting a focus phrase in a text may include:

[0054] Step 101, obtaining an inverted index and a multi-mode matching model constructed for a focus phrase, wherein the multi-mode matching model is generated based on the focus phrase, a similar character set of the focus phrase, and an ignored character set.

[0055] The focus phrase is a proper noun that needs to be accurately identified without errors. For example, the focus phrase may be "People's Republic of China".

[0056] When constructing an inverted index, the focus phrases may be segmented, and an inverted index may be constructed based on the attribution information of the segmented phrases.

[0057] When constructing a multi-modal matching model, it is necessary to obtain a similar character set and an ignored character set of a focus phrase, and use a multi-modal matching algorithm to generate a tree-structured multi-modal matching model for the focus phrase, similar character set and ignored character set.

[0058] Step 102: Detect the input text using a multi-mode matching model to obtain multiple fragments.

[0059] If the input text includes similar variants of the concerned phrases, the similar variants need to be recognized as the concerned phrases. Among them, the similar variants are obtained by replacing some characters in the concerned phrases with similar characters, inserting ignored characters in the concerned phrases, deleting some characters in the concerned phrases, etc. For example, if the input text is "The Constitution of the Zhong2hua Republic is the fundamental law of the Zhonghua People's Republic", and the concerned phrase is "People's Republic of China", the similar variants include "Zhong2hua Republic" and "Zhonghua People's Republic".

[0060] The multi-mode matching model includes a trie composed of multiple concerned phrases. When using the multi-mode matching model, the input text will be preprocessed according to the similar character set and the ignored character set to find the corresponding phrases in the trie. Then, the phrases in the text can be fragmented based on the phrases found in the trie and the start and end positions of the phrases in the text.

[0061] For example, if the input text is "The Constitution of the Zhong2hua Republic is the fundamental law of the Zhonghua People's Republic", and the phrase in the text is "Zhong2hua", and the phrase found in the multi-mode matching model is "Zhonghua", the generated fragment can be recorded as ("Zhong2hua", 0, 3, "Zhonghua").

[0062] Step 103: Aggregate multiple fragments using an inverted index to obtain a first fragment set.

[0063] Specifically, the phrases found in the multi-mode matching model can be used to find the corresponding concerned phrases in the inverted index, and then all the fragments corresponding to one concerned phrase can be combined into the first fragment set.

[0064] For example, the concerned phrase corresponding to "Zhonghua" in the fragment ("Zhong2hua", 0, 3, "Zhonghua") is "People's Republic of China", and the concerned phrase corresponding to "Republic" in the fragment ("Republic", 3, 6, "Republic") is "People's Republic of China". Then, the first fragment set corresponding to "People's Republic of China" is {("Zhong2hua", 0, 3, "Zhonghua"), ("Republic", 3, 6, "Republic")}

[0065] Step 104: Segment the first fragment set according to a preset edit distance to obtain multiple groups of second fragment sets.

[0066] The preset edit distance is set for the concerned phrases. The preset edit distances for different concerned phrases can be the same or different.

[0067] Specifically, the fragments in the first fragment set can be divided according to the number of preset edit distances to obtain multiple groups of second fragment sets.

[0068] For example, the preset edit distance is 4, and the first fragment set includes fragments 1-6. Then, one method is to divide fragments 1-4 into a group of second fragment sets, and divide fragments 5-6 into another group of second fragment sets; another method is to divide fragments 1-2 into a group of second fragment sets, and divide fragments 3-6 into another group of second fragment sets.

[0069] Step 105 : for each set of second fragments, screening candidate fragments according to the edit distance between the fragments in the second fragment set and the focus phrase.

[0070] Specifically, at least one fragment in the second fragment set can be combined into a candidate phrase according to the starting position and the ending position, and the edit distance between the candidate phrase and the focus phrase is calculated. If the edit distance is less than the preset edit distance, the fragment corresponding to the candidate phrase is retained to obtain the candidate fragment; if the edit distance is greater than the preset edit distance, the combination of the fragments corresponding to the candidate phrase is deleted.

[0071] Step 106 , combining the candidate fragments in each group of the second fragment set into similar variants of the focus phrase, and identifying the similar variants whose edit distance with the focus phrase is greater than a predetermined threshold as the focus phrase.

[0072] Specifically, candidate fragments are grouped into similar variants according to the starting position and the ending position, and the edit distance between the similar variant and the focus phrase is calculated. If the edit distance is less than a predetermined threshold, the similar variant is identified as the focus phrase; if the edit distance is greater than the predetermined threshold, the similar variant is not identified as the focus phrase.

[0073] In summary, the method for extracting a focus phrase in a text provided by an embodiment of the present application generates an improved multimodal detection model based on the focus phrase, a similar character set of the focus phrase, and an ignored character set, and uses the multimodal matching model to detect the input text to obtain multiple fragments; then, the multiple fragments are grouped using the inverted index of the focus phrase to obtain a first fragment set; then, the first fragment set is segmented according to a preset edit distance to obtain multiple groups of second fragment sets; for each group of second fragment sets, candidate fragments are screened according to the edit distance between the fragments in the second fragment set and the focus phrase; the candidate fragments in each group of second fragment sets are combined into similar variants of the focus phrase, and similar variants whose edit distance with the focus phrase is greater than a predetermined threshold are identified as the focus phrase, thereby improving the accuracy of extraction.

[0074] like Figure 2 As shown, it shows a flowchart of a method for extracting a focus phrase in a text provided by an embodiment of the present application, and the method for extracting a focus phrase in a text can be applied to a computer device. The method for extracting a focus phrase in a text may include:

[0075] Step 201 : split the focus phrases and construct an inverted index based on the split results.

[0076] When constructing an inverted index, the focus phrases may be segmented, and an inverted index may be constructed based on the attribution information of the segmented phrases.

[0077] For example, if the focus phrase is "People's Republic of China", the segmented words include: "China", "Chinese", "Republic", "People", "Mingong", "Heguo", "Chinese", "People's Gong", "Mingonghe", "Republic", "Chinese People", "Chinese People", "Mingong", "Chinese People's Republic", "Chinese People's Gong", "People's Republic ...People's Republic", "Chinese People's Republic", "People's Republic", "People's Republic of China", "People's Republic of China", "People's Republic of China".

[0078] Step 202, obtaining a similar character set of each character in the focus phrase, wherein the similar character set includes at least one of a homophone character set, a phonetically similar character set, and a similar-looking character set.

[0079] The phonetic similar character set refers to a character set with the same pronunciation as the characters in the concerned phrase. The phonetic similar character set refers to a character set with a pronunciation similar to the characters in the concerned phrase. The shape similar character set refers to a character set with a shape similar to the characters in the concerned phrase.

[0080] Step 203, obtaining an ignored character set, where the ignored character set includes at least one of a number set and a tag set, and the tag set is a tag that needs to be ignored and is set according to an application field.

[0081] The number set includes 0, 1, 2, 3, 4, 5, 6, 7, 8, 9.

[0082] The tag set is a collection of tags set according to the application field. For example, when applied in the webpage field, the tag set may include HyperText Markup Language (HTML) tags.

[0083] Step 204, constructing a multi-mode matching model based on the similar character set and the ignored character set.

[0084] Among them, the multi-modal matching model is generated based on the segmentation of the focus phrase, the similar character set of the focus phrase and the ignored character set.

[0085] Step 205: Use a multi-mode matching model to detect the input text and obtain multiple fragments.

[0086] If the input text includes similar variants of the concerned phrases, the similar variants need to be recognized as the concerned phrases. Among them, the similar variants are obtained by replacing some characters in the concerned phrases with similar characters, inserting ignored characters in the concerned phrases, deleting some characters in the concerned phrases, etc. For example, if the input text is "The Constitution of the 2 Zhonghua Republic is the fundamental law of the Zhonghua People's Republic", and the concerned phrase is "People's Republic of China", the similar variants include "2 Zhonghua Republic" and "Zhonghua People's Republic".

[0087] The multi-mode matching model includes a trie composed of multiple concerned phrases. When using the multi-mode matching model, the input text will be preprocessed according to the similar character set and the ignored character set to find the corresponding phrases in the trie.

[0088] Specifically, using the multi-mode matching model to detect the input text, multiple fragments can be obtained, including: obtaining the i-th character in the input text, where i is a positive integer; if the i-th character is an ignored character in the ignored character set, then update i to i + 1 and continue to execute the step of obtaining the i-th character in the input text; if the i-th character is not an ignored character, find all similar characters of the i-th character in the similar character set, match the i-th character and all similar characters with the trie in the multi-mode matching model, and generate fragments according to the matching results; where the fragments include the phrases extracted from the text, the start position and end position of the phrases in the text, and the phrases matched by the extracted phrases in the trie.

[0089] For example, if the input text is "The Constitution of the 2 Zhonghua Republic is the fundamental law of the Zhonghua People's Republic", for the first character "Zhong", start from the root node of the trie to find the "Zhong" node; for the second character "2", since "2" is a number in the ignored character set, so delete "2"; for the third character "Hua", start from the "Zhong" node and find the "Hua" node; for the fourth character "Gong", start from the "Hua" node and look down, no corresponding node is found, start from the root node of the trie to find the "Gong" node, and so on.

[0090] Then, the phrases in the text can generate fragments based on the phrases found in the trie for the phrases, as well as the start position and end position of the phrases in the text. For example, if the phrase in the text is "2 Zhonghua", and the phrase found in the multi-mode matching model is "Zhonghua", then the generated fragment can be recorded as ("2 Zhonghua", 0, 3, "Zhonghua").

[0091] If the input text is "The Constitution of the Zhong2hua Republic is the fundamental law of the Zhonghua People's Republic", the fragments output include: ("Zhong2hua", 0, 3, "中华"), ("Republic", 3, 6, "共和国"), ("Republic", 3, 5, "共和"), ("Zhonghua People's Republic", 9, 16, "中华人民共"), ("Zhonghua People", 9, 15, "中华人民"), ("Zhonghua People", 9, 12, "中华人"), ("Zhonghua", 9, 11, "中华").

[0092] Step 206, use the inverted index to collect multiple fragments to obtain the first fragment set.

[0093] In this embodiment, the phrases found in the multi-mode matching model can be used to find the corresponding concerned phrases in the inverted index, and then all the fragments corresponding to one concerned phrase are combined into the first fragment set.

[0094] Specifically, using the inverted index to collect multiple fragments to obtain the first fragment set may include:

[0095] (1) Use the inverted index to determine the concerned phrase to which each fragment belongs.

[0096] For example, if the concerned phrase is "People's Republic of China", the first fragment set is [("Zhong2hua", 0, 3, "中华"), ("Republic", 3, 6, "共和国"), ("Republic", 3, 5, "共和"), ("Zhonghua People's Republic", 9, 16, "中华人民共"), ("Zhonghua People", 9, 15, "中华人民"), ("Zhonghua People", 9, 12, "中华人"), ("Zhonghua", 9, 11, "中华")].

[0097] (2) Extract the first character and the last character from the concerned phrase, and supplement the fragments of the first character and the fragments of the last character.

[0098] Optionally, when there is no fragment of the first character and / or no fragment of the last character of the concerned phrase in the first fragment set, the fragments of the first character and the fragments of the last character also need to be supplemented.

[0099] Taking the concerned phrase "People's Republic of China" and the above first fragment set as an example, the fragment of the first character to be supplemented is ("Zhong", 0, 1, "中"), and the fragment of the last character to be supplemented is ("Country", 16, 17, "国").

[0100] (3) Among all the obtained fragments, filter out the fragments with the longest length and non-overlapping start positions and end positions with each other to obtain the first fragment set.

[0101] Since there is an overlap between ("中2华", 0, 3, "中华") and ("中", 0, 1, "中"), retain the one with the longest length, i.e., ("中2华", 0, 3, "中华"), and delete ("中", 0, 1, "中").

[0102] Since there is an overlap between ("共和国", 3, 6 "共和国") and ("共和", 3, 5 "共和"), retain the one with the longest length, i.e., ("共和国", 3, 6 "共和国"), and delete ("共和", 3, 5 "共和").

[0103] Since there is an overlap between ("中化人43民共", 9, 16, "中华人民共"), ("中化人43民", 9, 15, "中华人民"), ("中化人", 9, 12, "中华人"), and ("中化", 9, 11, "中华"), retain the one with the longest length, i.e., ("中化人43民共", 9, 16, "中华人民共"), and delete ("中化人43民", 9, 15, "中华人民"), ("中化人", 9, 12, "中华人"), and ("中化", 9, 11, "中华").

[0104] After the above screening, the final first fragment set obtained is [("中2华", 0, 3, "中华"), ("共和国", 3, 6 "共和国"), ("中化人43民共", 9, 16, "中华人民共"), ("国", 16, 17, "国")].

[0105] Step 207: Split the first fragment set according to the preset edit distance to obtain multiple groups of second fragment sets.

[0106] The preset edit distance is set for the concerned phrases, and the preset edit distances for different concerned phrases can be the same or different.

[0107] Specifically, the fragments in the first fragment set can be divided according to the number of preset edit distances to obtain multiple groups of second fragment sets.

[0108] For example, if the preset edit distance is 4 and the first fragment set includes fragments 1 - 6, one way is to split fragments 1 - 4 into one group of second fragment sets and fragments 5 - 6 into another group of second fragment sets; another way is to split fragments 1 - 2 into one group of second fragment sets and fragments 3 - 6 into another group of second fragment sets.

[0109] Step 208: For each group of second fragment sets, screen the candidate fragments according to the edit distance between the fragments in the second fragment set and the concerned phrases.

[0110] Specifically, screening the candidate fragments according to the edit distance between the fragments in the second fragment set and the concerned phrases may include:

[0111] (1) Select m consecutive fragments from the n fragments of the second fragment set, where 1 ≤ m ≤ n.

[0112] Among them, the value of m ranges from 1 to n. Suppose n = 4. When m = 4, fragments 1 - 4 form a candidate phrase; when m = 3, fragments 1 - 3 form a candidate phrase, and fragments 2 - 4 form a candidate phrase; when m = 2, fragments 1 - 2 form a candidate phrase, fragments 2 - 3 form a candidate phrase, and fragments 3 - 4 form a candidate phrase; when m = 1, fragment 1 forms a candidate phrase, fragment 2 forms a candidate phrase, fragment 3 forms a candidate phrase, and fragment 4 forms a candidate phrase.

[0113] (2) Determine the candidate phrases according to the start position of the first fragment and the end position of the mth fragment among the m fragments, and retain the candidate phrases whose edit distance from the phrase of interest is less than a predetermined threshold.

[0114] Specifically, at least one fragment in the second fragment set can be combined into candidate phrases according to the start position and the end position. Calculate the edit distance between the candidate phrase and the phrase of interest. If the edit distance is less than the preset edit distance, retain the fragments corresponding to the candidate phrase to obtain candidate fragments; if the edit distance is greater than the preset edit distance, delete the combination method of the fragments corresponding to the candidate phrase.

[0115] (3) Among all the obtained candidate phrases, screen out the candidate phrases with the longest length and non - overlapping start and end positions with each other, and determine the fragments corresponding to the candidate phrases as candidate fragments.

[0116] For example, the first phrase of interest is "People's Republic of China", and the corresponding candidate fragments are: ("Zhong2Hua", 0, 3, "ZhongHua"), ("Republic of China", 3, 6, "Republic of China"); the second phrase of interest is "People's Republic of China", and the corresponding candidate fragments are: ("ZhongHuaRen43MinGong", 9, 16, "People's Republic of"), ("Guo", 16, 17, "Guo").

[0117] Step 209: Combine the candidate fragments in each group of the second fragment set into similar variants of the phrase of interest, and identify the similar variants whose edit distance from the phrase of interest is greater than the predetermined threshold as the phrase of interest.

[0118] Specifically, combining the candidate fragments in each group of the second fragment set into similar variants of the phrase of interest includes: for each group of the second fragment set, combine the respective candidate fragments in the second fragment set according to the start position and the end position to obtain similar variants of the phrase of interest.

[0119] Such as Figure 3As shown, the first similar variant of the focus phrase "People's Republic of China" is "Zhong2hua Republic", and the second similar variant of the focus phrase "People's Republic of China" is "Zhonghua People's Republic".

[0120] Specifically, identifying a similar variant with an edit distance greater than a predetermined threshold from the focus phrase as a focus phrase may include: calculating the edit distance between the similar variant and the focus phrase; if the edit distance is less than the predetermined threshold, then identifying the similar variant as a focus phrase.

[0121] In this embodiment, the edit distance between the similar variant and the focus phrase can be calculated. If the edit distance is less than the predetermined threshold, then the similar variant is identified as a focus phrase; if the edit distance is greater than the predetermined threshold, then the similar variant is not identified as a focus phrase.

[0122] In summary, the method for extracting focus phrases in the text provided by the embodiments of the present application generates an improved multi-mode detection model based on the focus phrase, the similar character set of the focus phrase, and the ignored character set, uses the multi-mode matching model to detect the input text to obtain multiple fragments; then uses the inverted index of the focus phrase to归集 multiple fragments to obtain the first fragment set; then, according to the preset edit distance, the first fragment set is segmented to obtain multiple groups of second fragment sets; for each group of second fragment sets, candidate fragments are screened according to the edit distance between the fragments in the second fragment set and the focus phrase; the candidate fragments in each group of second fragment sets are combined into similar variants of the focus phrase, and the similar variants with an edit distance greater than the predetermined threshold from the focus phrase are identified as focus phrases, improving the extraction accuracy.

[0123] As Figure 4 shown, it shows a structural block diagram of an apparatus for extracting focus phrases in the text provided by an embodiment of the present application. The apparatus for extracting focus phrases in the text can be applied to a computer device. The apparatus for extracting focus phrases in the text may include:

[0124] An acquisition module 410, configured to acquire an inverted index and a multi-mode matching model constructed for the focus phrase, where the multi-mode matching model is generated according to the focus phrase, the similar character set of the focus phrase, and the ignored character set;

[0125] A detection module 420, configured to use the multi-mode matching model to detect the input text to obtain multiple fragments;

[0126] An aggregation module 430, configured to aggregate multiple fragments using the inverted index to obtain the first fragment set;

[0127] A segmentation module 440, configured to segment the first fragment set according to a preset edit distance to obtain multiple groups of second fragment sets;

[0128] A screening module 450 is used to screen candidate fragments for each set of second fragments according to the edit distance between the fragments in the second fragment set and the focus phrase;

[0129] The identification module 460 is configured to combine the candidate fragments in each group of the second fragment set into similar variants of the focus phrase, and identify the similar variants whose edit distance with the focus phrase is greater than a predetermined threshold as the focus phrase.

[0130] In an optional embodiment, the detection module 420 is further configured to:

[0131] Get the i-th character in the input text, where i is a positive integer;

[0132] If the i-th character is an ignored character in the ignored character set, then i is updated to i+1, and the step of obtaining the i-th character in the input text is continued;

[0133] If the i-th character is not an ignored character, then search for all similar characters of the i-th character in the similar character set, match the i-th character and all similar characters with the dictionary tree in the multi-mode matching model, and generate fragments according to the matching results;

[0134] The fragments include phrases extracted from the text, the starting position and the ending position of the phrases in the text, and the phrases matched in the dictionary tree according to the extracted phrases.

[0135] In an optional embodiment, the screening module 450 is further configured to:

[0136] Select m consecutive fragments from the n fragments of the second fragment set, 1≤m≤n;

[0137] Determine candidate phrases according to the starting position of the first fragment and the ending position of the mth fragment among the m fragments, and retain candidate phrases whose edit distance with the focus phrase is less than a predetermined threshold;

[0138] Among all the obtained candidate phrases, the candidate phrases with the longest length and whose starting positions and ending positions do not overlap with each other are selected, and the fragments corresponding to the candidate phrases are determined as candidate fragments.

[0139] In an optional embodiment, the identification module 460 is further configured to:

[0140] For each group of second fragment sets, each candidate fragment in the second fragment set is combined according to the starting position and the ending position to obtain a similar variant of the focus phrase.

[0141] In an optional embodiment, the collection module 430 is further configured to:

[0142] Use the inverted index to determine the focus phrases to which each fragment belongs;

[0143] Extracting the first word and the last word from the focus phrase, and supplementing the first word fragment and the last word fragment;

[0144] Among all the obtained fragments, the fragments with the longest length and non-overlapping starting and ending positions are selected to obtain the first fragment set.

[0145] In an optional embodiment, the identification module 460 is further configured to:

[0146] Calculate the edit distance between similar variant domain focus phrases;

[0147] If the edit distance is less than a predetermined threshold, the similar variant is identified as a phrase of interest.

[0148] In an optional embodiment, the acquisition module 410 is further configured to:

[0149] Split the focus phrases and build an inverted index based on the split results;

[0150] Obtaining a similar character set for each character in the focus phrase, the similar character set comprising at least one of a homophone character set, a phonetically similar character set, and a similar-looking character set;

[0151] Obtain an ignored character set, where the ignored character set includes at least one of a number set and a label set, and the label set is a label that needs to be ignored and is set according to an application field;

[0152] A multi-modal matching model is constructed based on similar character sets and ignored character sets.

[0153] In summary, the device for extracting a focus phrase in a text provided by an embodiment of the present application generates an improved multimodal detection model according to the focus phrase, a similar character set of the focus phrase, and an ignored character set, and uses the multimodal matching model to detect the input text to obtain multiple fragments; then uses the inverted index of the focus phrase to group the multiple fragments to obtain a first fragment set; then, the first fragment set is segmented according to a preset edit distance to obtain multiple groups of second fragment sets; for each group of second fragment sets, candidate fragments are screened according to the edit distance between the fragments in the second fragment set and the focus phrase; the candidate fragments in each group of second fragment sets are combined into similar variants of the focus phrase, and similar variants whose edit distance with the focus phrase is greater than a predetermined threshold are identified as the focus phrase, thereby improving the accuracy of extraction.

[0154] An embodiment of the present application provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement the above-mentioned method for extracting focus phrases in a text.

[0155] An embodiment of the present application provides a computer device, which includes a device for extracting a focus phrase from any of the above-mentioned texts.

[0156] It should be noted that: the device for extracting focus phrases in a text provided in the above embodiment only uses the division of the above functional modules as an example when extracting focus phrases in a text. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device for extracting focus phrases in a text is divided into different functional modules to complete all or part of the functions described above. In addition, the device for extracting focus phrases in a text provided in the above embodiment and the method for extracting focus phrases in a text belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0157] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0158] The above description is not intended to limit the embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the protection scope of the embodiments of the present application.

Claims

1. A method for extracting focus phrases in a text, characterized in that: The method comprises: Obtaining an inverted index and a multi-mode matching model constructed for a focus phrase, wherein the multi-mode matching model is generated according to the focus phrase, a similar character set of the focus phrase, and an ignored character set; Using the multi-mode matching model to detect the input text, and obtain multiple fragments; The plurality of fragments are grouped by using the inverted index to obtain a first fragment set; Splitting the first fragment set according to a preset edit distance to obtain multiple groups of second fragment sets; For each set of second fragments, screening candidate fragments according to the edit distance between the fragments in the second fragment set and the focus phrase; The candidate fragments in each group of the second fragment set are combined into similar variants of the focus phrase, and the similar variants whose edit distance with the focus phrase is greater than a predetermined threshold are identified as the focus phrase.

2. The method for extracting focus phrases in a text according to claim 1, characterized in that: The multi-mode matching model is used to detect the input text to obtain multiple fragments, including: Get the i-th character in the input text, where i is a positive integer; If the i-th character is an ignored character in the ignored character set, then i is updated to i+1, and the step of obtaining the i-th character in the input text is continued; If the i-th character is not the ignored character, searching for all similar characters of the i-th character in the similar character set, matching the i-th character and all similar characters with the dictionary tree in the multi-mode matching model, and generating fragments according to the matching results; The fragments include phrases extracted from the text, the starting position and the ending position of the phrases in the text, and phrases matched in the dictionary tree according to the extracted phrases.

3. The method for extracting focus phrases in a text according to claim 2, characterized in that: The screening of candidate fragments according to the edit distance between the fragments in the second fragment set and the focus phrase comprises: Selecting m consecutive fragments from the n fragments of the second fragment set, 1≤m≤n; Determine candidate phrases according to the starting position of the first fragment and the ending position of the mth fragment among the m fragments, and retain candidate phrases whose edit distance with the focus phrase is less than a predetermined threshold; Among all the obtained candidate phrases, the candidate phrases with the longest length and whose starting positions and ending positions do not overlap with each other are selected, and the fragments corresponding to the candidate phrases are determined as candidate fragments.

4. The method for extracting focus phrases in a text according to claim 2, characterized in that: The step of combining the candidate fragments in each group of the second fragment set into similar variants of the concerned phrase comprises: For each group of second fragment sets, each candidate fragment in the second fragment set is combined according to the starting position and the ending position to obtain a similar variant of the focus phrase.

5. The method for extracting focus phrases in a text according to claim 1, characterized in that: The step of using the inverted index to group the plurality of fragments to obtain a first fragment set includes: Determine the focus phrase to which each fragment belongs using the inverted index; Extracting the first character and the last character from the focus phrase, and supplementing the fragments of the first character and the fragments of the last character; Among all the obtained fragments, the fragments with the longest length and non-overlapping starting and ending positions are selected to obtain the first fragment set.

6. The method for extracting focus phrases in a text according to claim 1, characterized in that: The step of identifying a similar variant having an edit distance greater than a predetermined threshold from the focus phrase as the focus phrase comprises: Calculate the edit distance between the similar variants and the focus phrase; If the edit distance is less than a predetermined threshold, the similar variant is identified as the focus phrase.

7. The method for extracting focus phrases in a text according to any one of claims 1 to 6, characterized in that: The step of obtaining an inverted index and a multi-mode matching model constructed for a focus phrase includes: Splitting the focus phrases, and constructing an inverted index according to the splitting results; Obtaining a similar character set for each character in the focus phrase, wherein the similar character set includes at least one of a homophone character set, a phonetically similar character set, and a similar-looking character set; Obtain an ignored character set, wherein the ignored character set includes at least one of a number set and a label set, and the label set is a label that needs to be ignored and is set according to an application field; A multi-modal matching model is constructed according to the similar character set and the ignored character set.

8. A device for extracting a focus phrase in a text, characterized in that: The device comprises: An acquisition module, used for acquiring an inverted index and a multi-mode matching model constructed for a focus phrase, wherein the multi-mode matching model is generated according to the focus phrase, a similar character set of the focus phrase, and an ignored character set; A detection module, used to detect the input text using the multi-mode matching model to obtain multiple fragments; A collection module, configured to collect the plurality of fragments by using the inverted index to obtain a first fragment set; A segmentation module, configured to segment the first fragment set according to a preset edit distance to obtain multiple groups of second fragment sets; A screening module, configured to screen candidate fragments for each set of second fragments according to the edit distance between the fragments in the second fragment set and the concerned phrase; The identification module is used to combine the candidate fragments in each group of the second fragment set into similar variants of the focus phrase, and identify the similar variants whose edit distance with the focus phrase is greater than a predetermined threshold as the focus phrase.

9. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the method for extracting focus phrases in a text as described in any one of claims 1 to 7.

10. A computer device, characterized in that: The computer device comprises: the device for extracting focus phrases in the text as described in claim 8.

Images