Grammar-based unsupervised word segmentation method, device and storage medium

Abstract

The application relates to the technical field of natural language processing, in particular to a grammar-based unsupervised word segmentation method, device and storage medium. The method comprises the following steps: setting a word segmenter weight and constructing a word vector model, generating an initial hot word table and a search hot word table based on a domain data set and the word segmenter; obtaining a search field input by a user and preprocessing the search field; re-integrating the preprocessed search field and generating an integrated word segmentation group; performing intent recognition based on the integrated word segmentation group, performing word rewriting and replacement and adding a word weight to obtain a word segmentation group with a word weight; filtering the word segmentation group with the word weight one by one and retaining only keywords; encapsulating and returning the keywords to perform matching search and fuzzy search through a search engine, and reordering a result set. The application can obtain more accurate results by identifying the search intent of the user, thereby effectively improving the accuracy of the search results fed back to the user.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to a syntax-based unsupervised word segmentation method, device, and storage medium. Background Technology

[0002] As national policies continue to evolve, businesses need to leverage government service systems to access the latest government information in a timely manner. These systems provide users with search functionality; users can enter search terms into the system, which will then match and display the results containing those terms.

[0003] Current government service systems mainly use word segmenters to split the user's input search fields based on the national corpus or certain specific corpora, and then select the more key search terms to match the search results based on the splitting results of the search fields.

[0004] However, in the process of matching search results based on user-input search fields, word segmentation based on national corpora or certain specific corpora may not be ideal when applied to a specific business domain, which may result in lower accuracy of search results returned to the user. Summary of the Invention

[0005] This application provides a syntax-based unsupervised word segmentation method, device, and storage medium, which can obtain more and more accurate results by recognizing user search intent, thereby effectively improving the accuracy of search results fed back to the user. This application provides the following technical solutions:

[0006] Firstly, this application provides a syntax-based unsupervised word segmentation method, the method comprising:

[0007] Set the weights of the word segmenter and build a word vector model. Generate an initial hot word list and a search hot word list based on the domain dataset and the word segmenter.

[0008] Obtain the search field input by the user and preprocess the search field;

[0009] The preprocessed search fields are re-integrated to generate integrated word groups;

[0010] Based on the integrated word segments, intent recognition is performed, words are rewritten and replaced, and word weights are added to obtain word segments with word weights.

[0011] Based on the word groups with word weights, each word is filtered individually, and only keywords are retained;

[0012] The keywords are encapsulated and returned, and then subjected to multi-dimensional matching and fuzzy search by a search engine. The result set is then reordered based on the weight of the words.

[0013] In one specific implementation, setting the word segmenter weights and constructing a word vector model, generating an initial hot word list and a search hot word list based on the domain dataset and the word segmenter, includes:

[0014] For a limited number of word segmenters, different weights are set based on expert evaluation and test set, and they are combined into a word segmenter. Then, an unsupervised word vector model is built based on the dataset of this domain.

[0015] Subsequently, an initial hot word list and a manually configured search hot word list were generated based on the dataset in this field and multiple word segmenters.

[0016] In one specific implementation, the step of re-integrating the preprocessed search fields and generating integrated word segments includes:

[0017] The preprocessed search field is segmented using multiple word segmenters to obtain word groups and their corresponding semantics.

[0018] Based on the aforementioned word segmentation groups and the aforementioned word vector model, a word segmenter is used to perform secondary word splitting and integration;

[0019] The weighted probabilities of each segmented word group are obtained based on the weights, and then integrated into a new segmented word group.

[0020] In a specific implementation scheme, the secondary word segmentation and integration based on the word segmentation group and the word vector model using a word segmenter includes:

[0021] Try concatenating multiple or more split words and determine whether the concatenated words exist in the word vector model and the search hot word table;

[0022] When the new concatenation word itself does not exist in the initial hot word list or the word vector model, the words before and after are split into individual words, and then the concatenation is attempted one by one, and it is checked again whether it exists in the word vector model.

[0023] If a new word is obtained, the semantics of the new word are derived based on the semantics of the original word;

[0024] Certain parts of speech are prohibited from being combined based on their parts of speech.

[0025] Based on the results of splitting and integrating, optimizations are made to address the shortcomings of different word segmenters.

[0026] In one specific implementation, the step of performing intent recognition based on the integrated word segments, rewriting and replacing words, and attaching word weights to obtain word segments with word weights includes:

[0027] First, the integrated word groups are analyzed one by one. Based on the initial hot word list and word vector model, the top-n related words of the preceding and following words are calculated. If words with similar pinyin and higher correlation appear, they are replaced.

[0028] Then, the part of speech of each integrated word group is determined, and different weights are set for each word based on the grammatical rules of different fields. Finally, word groups with word weights are generated.

[0029] In one specific implementation, the syntax rules are dynamically configured, including:

[0030] In the area of ​​relevant policies, the weight of modifiers in attributive clauses is set to low;

[0031] When a verb is followed by a noun or verb + noun, the importance of the following noun is set to high.

[0032] When rankings appear consecutively, the last noun is considered more important than the preceding nouns.

[0033] In one specific implementation, the filtering based on the weighted word groups, retaining only keywords, includes:

[0034] When there are many words in a word segment with word weight, the number of words to be retained should be limited as needed;

[0035] Based on part-of-speech, nouns that require special processing and searching from word groups with word weights are extracted using separate logic.

[0036] Based on the search hot word list, pre-defined words are regarded as attractive words, and higher weights are assigned to attractive words.

[0037] In one specific implementation scheme, the preset words based on the search hot word list are regarded as attractive words, and the higher weight is assigned to attractive words, including:

[0038] The distance between word vectors is calculated based on the word vector model. The word with the smallest distance between its word vector and the search term is the attractive word of the search term.

[0039] In a second aspect, this application provides an electronic device, the device including a processor and a memory; the memory stores a program, the program being loaded and executed by the processor to implement a syntax-based unsupervised word segmentation method as described in any one of claims 1 to 8.

[0040] Thirdly, this application provides a computer-readable storage medium storing a program that, when executed by a processor, is used to implement a syntax-based unsupervised word segmentation method as described in any one of claims 1 to 8.

[0041] In summary, the beneficial effects of this application include at least the following:

[0042] 1) Only unsupervised pre-trained word vectors based on different corpora are needed, requiring few training resources. In addition, the algorithm does not involve complex matrix operations, ensuring the real-time nature of the data.

[0043] 2) The search is broken down into multiple sub-processes with low coupling between them. Different processes handle different business needs, which facilitates expansion and customization of the search process and makes it easy to extract specific types of words.

[0044] 3) Identifying user search intent can yield more and more accurate results, thereby effectively improving the accuracy of search results fed back to users.

[0045] First, the preprocessed search field is cleaned and segmented, then reorganized. Next, intent recognition is performed based on part-of-speech tags, words are rewritten and assigned different weights, and multiple word segmenters and word vectors are integrated to optimize word segmentation. Finally, keywords are retained and Elasticsearch is used for keyword search. Identifying user search intent yields more accurate results, effectively improving the accuracy of search results presented to users.

[0046] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, the preferred embodiments of this application are described in detail below with reference to the accompanying drawings. Attached Figure Description

[0047] Figure 1 This is a flowchart illustrating the syntax-based unsupervised word segmentation method in this application embodiment.

[0048] Figure 2 This is a flowchart illustrating the process of recombinizing preprocessed search fields in an embodiment of this application.

[0049] Figure 3 This is a schematic diagram of the process of filtering word groups one by one based on word weights in the embodiments of this application.

[0050] Figure 4 This is a block diagram of an electronic device based on syntax-based unsupervised word segmentation in an embodiment of this application. Detailed Implementation

[0051] The specific embodiments of this application will be described in further detail below with reference to the accompanying drawings and examples. The following examples are used to illustrate this application, but are not intended to limit the scope of this application.

[0052] First, let me introduce some of the terms used in this application.

[0053] Word segmentation: Word segmentation is an important task in natural language processing (NLP), which refers to the process of dividing a continuous text into words or phrases.

[0054] Word vectors are an important concept in Natural Language Processing (NLP). They are techniques for representing words as numerical vectors, enabling computers to better understand and process text data. The main idea behind word vectors is to map words into a high-dimensional vector space, making words with similar meanings closer together in this space.

[0055] Elasticsearch: Elasticsearch is an open-source distributed search and analytics engine specifically designed for handling large-scale text and structured data.

[0056] String matching algorithms are a class of algorithms in computer science used to find the location or occurrence of one or more target strings (usually called "patterns") within a given string (usually called "text"). The main goal of string matching algorithms is to determine whether a pattern exists in the text, find the pattern's location, or count the number of times the pattern appears.

[0057] Unsupervised: "Unsupervised" is a commonly used term in the fields of machine learning and data analysis. It refers to a learning or analytical method in which the model or algorithm is trained or analyzed without relying on pre-labeled data or supervisory signals. The main goal of unsupervised learning is to discover hidden patterns, structures, or information in unlabeled data, or to perform operations such as dimensionality reduction and clustering, without requiring explicit target variables or labels.

[0058] Regular expressions (or regular expressions) are powerful text processing tools used to search, match, and manipulate string patterns in text. They offer a flexible syntax for describing specific patterns in strings, enabling operations such as searching, replacing, and validating.

[0059] Reference Figure 1 This is a flowchart illustrating a syntax-based unsupervised word segmentation method provided in an embodiment of this application. The method includes at least the following steps:

[0060] S101. Set the weights of the word segmenter and build a word vector model. Generate an initial hot word list and a search hot word list based on the domain dataset and the word segmenter.

[0061] Optionally, the tokenizer is the Elasticsearch tokenizer, but other tokenizers can also be used. This application does not restrict the type of tokenizer.

[0062] In implementation, for a limited number of word segmenters E1…En, different weights K1…Kn are set based on expert evaluation and a test set, and these weights are combined into a single word segmenter. An unsupervised word vector model W1…Wn is then constructed based on the domain dataset. Subsequently, an initial hot word list H and a manually configured search hot word list L are generated based on the domain dataset and multiple word segmenters.

[0063] Alternatively, the field in this application typically refers to the field of government services.

[0064] S102. Obtain the search field input by the user and preprocess the search field.

[0065] Specifically, after obtaining the search field input by the user, special characters and unnecessary fixed expressions in the search field are removed based on prior NLP syntax knowledge of the domain.

[0066] S103. Reintegrate the preprocessed search fields and generate integrated word groups.

[0067] Reference Figure 2 This is a flowchart illustrating the recombination of preprocessed search fields according to an embodiment of this application. The method includes at least the following steps:

[0068] S1031. The preprocessed search field is segmented using multiple word segmenters to obtain word groups and their corresponding semantics.

[0069] S1032. Based on word segmentation groups and word vector models, a word segmenter is used to perform secondary word splitting and integration.

[0070] Specifically, the steps for secondary splitting and integration are as follows:

[0071] (1) Try to combine two or more split words and determine whether the combined word exists in W and L.

[0072] (2) When the new concatenation word itself does not exist in H or W, it may be that the word split by the word segmenter does not exist in this domain. In this case, split the words before and after into individual words, try to concatenate them one by one, and judge again whether they exist in W.

[0073] (3) If a new word is obtained, the semantics of the new word is obtained based on the semantics of the original word. For example, verb + noun is considered as gerund, and noun + noun is still considered as noun, etc.

[0074] (4) Certain parts of speech are prohibited from being combined based on their parts of speech. The part of speech of a word segment is used to indicate the characteristics of the words in that word segment.

[0075] Optionally, the parts of speech of the word group include, but are not limited to: nouns, verbs, adjectives, classifiers, pronouns, and adverbs.

[0076] (5) Optimize the shortcomings of different word segmenters based on the results of splitting and integration, such as not splitting numbers or letters with subsequent rank.

[0077] S1033. Based on the weights of K1...Kn, obtain the weight probability of each segmented word group after processing by each word segmenter, and integrate them into a new segmented word group.

[0078] In one example, the importance of word segments is positively correlated with the numerical value of the word segment weight.

[0079] S104. Based on the integrated word segmentation group, perform intent recognition, rewrite and replace words, and add word weights to obtain word segmentation groups with word weights.

[0080] Specifically, the process begins by parsing and integrating the word segments one by one. Based on Heuristics (H) and Word Meaning (W), the top-n related words are calculated. If words with similar pinyin and higher relevance are found, they are attempted to be replaced. Then, the part-of-speech of each integrated word segment is determined. Based on the grammatical rules of different domains, different weights are assigned to the words, and finally, word segments with word weights are generated. The grammatical rules are dynamically configured and vary across different domains. These rules include, but are not limited to:

[0081] (1) In the field of relevant policies, the weight of modifiers in attributive clauses is generally low.

[0082] (2) When a verb is followed by a noun or verb + noun, the following noun is of higher importance.

[0083] (3) When rankings appear consecutively, the last noun is more important than the previous nouns.

[0084] S105. Filter each word group based on word weights, retaining only keywords.

[0085] Reference Figure 3 This is a flowchart illustrating the process of filtering word groups based on word weights according to an embodiment of this application. The method includes at least the following steps:

[0086] S1051. When there are many words in a word segmentation group with word weight, the number of words to be retained should be limited as needed.

[0087] Alternatively, when searching for policies, you may only need to retain 2-3 keywords.

[0088] S1052. Based on part-of-speech, extract nouns that require special processing and searching, such as regional terms, departmental terms, and time terms, from the word groups with word weights using separate logic.

[0089] In one example, to extract regions, a list of region-related words is obtained based on a region table and word groups with weighted word segments whose parts of speech correspond to regions. Various string matching algorithms are then used to obtain this list. Notably, distance calculations using word vectors are generally unnecessary for regions.

[0090] In another example, to extract departments, based on the department table and the regions extracted above, as well as word groups with word weights whose parts of speech are proper nouns, nouns, departments, etc., a set of departments under certain regions is obtained. Similarly, based on multiple string matching algorithms, a list of matching departments is obtained. If no matching department is found, it may be because of the abbreviation of the department. Therefore, the word vector distance between H and W is used for judgment. If a certain threshold is met, the department can be extracted.

[0091] In another example, to extract the time, a regular expression is used to iterate through possible times and extract them one by one. Based on the combination of extracted time words, such as "this year" and "Chinese New Year", a specific timestamp or time period is obtained.

[0092] S1053. Based on the L preset words, treat them as attraction words and set higher weights for attraction words.

[0093] Specifically, based on the distance between word vectors calculated using W, the word with the smallest distance to the search term among a subset of words is identified as the "gravitational word" for that search term. This operation can, to a certain extent, ensure a high level of recall in the search and prevent deviation from the intended outcome. These words include, but are not limited to, tag words, business terms, and trending search terms.

[0094] S106. Encapsulate and return the keywords, perform multi-dimensional matching and fuzzy search through the search engine, and reorder the result set based on the weight of the words.

[0095] Specifically, the parsed keywords are packaged into JSON and returned. Based on this JSON, Elasticsearch performs multi-dimensional matching and fuzzy search, and the result set is reordered based on the weight of the words.

[0096] Optionally, this application uses the Elasticsearch search engine, but other search engines may also be used. This application does not restrict the type of search engine.

[0097] S107. Based on the above process and the corpus of this field, generate a new hot word list through unsupervised iteration.

[0098] In summary, this application first cleanses and segments the preprocessed search field, then reorganizes it. Next, it performs intent recognition based on part-of-speech tags, rewrites words and assigns different weights, and integrates multiple word segmenters and word vectors to optimize word segmentation. Finally, it retains keywords and uses Elasticsearch (ES) for keyword search. It only requires unsupervised pre-training of word vectors based on different corpora, requiring minimal training resources. Furthermore, the algorithm does not involve complex matrix operations, ensuring real-time data processing. By breaking down the search into multiple sub-processes with low coupling between them, different processes handle different business needs, facilitating expansion and customization of the search process, and enabling the extraction of specific word types. Identifying user search intent yields more and more accurate results, effectively improving the accuracy of search results presented to users.

[0099] Figure 4 This is a block diagram of an electronic device provided in one embodiment of this application. The device includes at least a processor 401 and a memory 402.

[0100] Processor 401 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 401 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 401 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 401 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 401 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0101] Memory 402 may include one or more computer-readable storage media, which may be non-transitory. Memory 402 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in memory 402 is used to store at least one instruction, which is executed by processor 401 to implement the syntax-based unsupervised word segmentation method provided in the method embodiments of this application.

[0102] In some embodiments, the electronic device may also optionally include: a peripheral device interface and at least one peripheral device. The processor 401, memory 402, and peripheral device interface can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface via a bus, signal line, or circuit board. Indicatively, peripheral devices include, but are not limited to: radio frequency circuitry, a touch display screen, audio circuitry, and a power supply.

[0103] Of course, electronic devices may also include fewer or more components, and this embodiment does not limit this.

[0104] Optionally, this application also provides a computer-readable storage medium storing a program that is loaded and executed by a processor to implement the syntax-based unsupervised word segmentation method of the above method embodiments.

[0105] Optionally, this application also provides a computer product including a computer-readable storage medium storing a program that is loaded and executed by a processor to implement the syntax-based unsupervised word segmentation method of the above method embodiments.

[0106] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0107] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A syntax-based unsupervised word segmentation method, characterized in that, The method includes: setting segmenter weights and constructing a word vector model; generating an initial hot word list and a search hot word list based on the domain dataset and the segmenter; including: for a limited number of segmenters, setting different weights based on expert evaluation and a test set, combining them into a segmenter, and constructing a word vector model unsupervised based on the domain dataset; subsequently generating an initial hot word list and a manually configured search hot word list based on the domain dataset and multiple segmenters. Obtain the search field input by the user and preprocess the search field; The preprocessed search fields are then re-integrated to generate integrated word groups; Based on the integrated word segments, intent recognition is performed, words are rewritten and replaced, and word weights are added to obtain word segments with word weights. Based on the word groups with word weights, each word is filtered individually, and only keywords are retained; The keywords are encapsulated and returned, and then subjected to multi-dimensional matching and fuzzy search by a search engine. The result set is then reordered based on the weight of the words.

2. The syntax-based unsupervised word segmentation method according to claim 1, characterized in that, The step of re-integrating the preprocessed search fields and generating integrated word groups includes: The preprocessed search field is segmented using multiple word segmenters to obtain word groups and their corresponding semantics. Based on the aforementioned word segmentation groups and the aforementioned word vector model, a word segmenter is used to perform secondary word splitting and integration; The weighted probabilities of each segmented word group are obtained based on the weights, and then integrated into a new segmented word group.

3. The syntax-based unsupervised word segmentation method according to claim 2, characterized in that, The secondary word segmentation and integration based on the word segmentation group and the word vector model using a word segmenter includes: Try concatenating multiple or more split words and determine whether the concatenated words exist in the word vector model and the search hot word table; When the new concatenation word itself does not exist in the initial hot word list or the word vector model, the words before and after are split into individual words, and then the concatenation is attempted one by one, and it is checked again whether it exists in the word vector model. If a new word is obtained, its meaning is derived based on the meaning of the original word. Certain parts of speech are prohibited from being combined based on their parts of speech. Based on the results of splitting and integrating, optimizations are made to address the shortcomings of different word segmenters.

4. The syntax-based unsupervised word segmentation method according to claim 1, characterized in that, The process of performing intent recognition based on the integrated word segments, rewriting and replacing words, and assigning word weights to obtain word segments with word weights includes: First, the integrated word groups are analyzed one by one. Based on the initial hot word list and word vector model, the top-n related words of the preceding and following words are calculated. If words with similar pinyin and higher correlation appear, they are replaced. Then, the part of speech of each integrated word group is determined one by one. Based on the grammatical rules of different fields, different weights are set for each word, and finally, word groups with word weights are generated.

5. The syntax-based unsupervised word segmentation method according to claim 4, characterized in that, The syntax rules are dynamically configured and include: In the area of ​​relevant policies, the weight of modifiers in attributive clauses is set to low; When a verb is followed by a noun or verb + noun, the importance of the following noun is set to high. When rankings appear consecutively, the last noun is considered more important than the preceding nouns.

6. The syntax-based unsupervised word segmentation method according to claim 1, characterized in that, The process of filtering each word group based on its weighted word segmentation, retaining only keywords, includes: When there are many words in a word segment with word weight, the number of words to be retained should be limited as needed; Based on part-of-speech, nouns that require special processing and searching in word groups with word weights are extracted using separate logic. Based on the search hot word list, pre-defined words are regarded as attractive words, and higher weights are assigned to attractive words.

7. The syntax-based unsupervised word segmentation method according to claim 6, characterized in that, The terms preset based on the search hot keyword list are considered as "attraction words," and higher weights are assigned to these attraction words, including: The distance between word vectors is calculated based on the word vector model. The word with the smallest distance between its word vector and the search term is the attractive word of the search term.

8. An electronic device, characterized in that, The device includes a processor and a memory; the memory stores a program that is loaded and executed by the processor to implement a syntax-based unsupervised word segmentation method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The storage medium stores a program that, when executed by a processor, is used to implement a syntax-based unsupervised word segmentation method as described in any one of claims 1 to 7.

Images