Text matching method and device based on word segmentation device and dictionary tree, equipment and medium
The text matching method using a word segmenter and dictionary tree addresses the issue of poor word segmentation in financial search systems by improving accuracy and efficiency through domain-specific knowledge integration and typo correction.
Patent Information
- Application Number
- CN202411179742.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-12-13
AI Technical Summary
Existing search systems in the financial field suffer from poor word segmentation accuracy due to lack of domain knowledge, leading to incorrect or no search results when users input keywords with typos or mismatches.
A text matching method using a word segmenter and dictionary tree to segment user input and match it with a preset dictionary tree, improving accuracy by utilizing domain-specific knowledge and correcting typos.
Enhances the accuracy and efficiency of text matching in the financial field by accurately identifying and matching words and phrases, reducing misidentification and mismatching.
Smart Images
Figure CN119129593A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the fields of data analysis and digital finance, and in particular to a text matching method, device, equipment and medium based on a word segmenter and a dictionary tree. Background Art
[0002] In the financial field, the business analysis search system is an important system that uses search algorithms to help managers conduct business analysis and assist in decision-making. The search system is a very important computer application technology in the era of big data, which can help users quickly find the required content from massive amounts of data.
[0003] At present, there are numerous search systems based on open source engines such as Lucene and ElasticSearch in the market. Such search engines mainly use words as the basic search unit, rely on word segmentation to segment documents, build inverted indexes based on word segmentation results, and then perform word segmentation matching on user input. Such search engines have the following two problems: first, the word segmentation effect of the word segmentation is poor in specific fields due to lack of domain knowledge; second, the forced word segmentation of user input may result in no result return or wrong result return due to mismatch between word segmentation fragments and indexes. For example, for the business analysis indicator "comprehensive cost rate", the word segmentation will divide it into "comprehensive", "cost" and "rate", so these three words are used as indexes; when the user enters "zonghe cost rate" for search, because "zong" is a typo here, it affects the word segmentation's understanding of it, and the word segmentation will divide it into "zong", "synthesis" and "ben rate". These three word segmentation fragments will be matched with the index separately, and no results will be matched, so the search return result is empty, and the accuracy of the search results is greatly affected. Therefore, how to improve the accuracy of text matching based on keywords input by users in the financial field has become a technical problem that needs to be solved urgently. Summary of the invention
[0004] The present application provides a text matching method, device, equipment and medium based on a word segmenter and a dictionary tree to improve the accuracy of text matching based on keywords input by users in the financial field.
[0005] In a first aspect, the present application provides a text matching method based on a word segmenter and a dictionary tree, the method comprising:
[0006] Determine at least one set of input word segmentation segments based on the user's initial input text and a preset word segmenter;
[0007] Using each of the input word segmentation fragments as a word segmentation index, and matching the word segmentation index with each preset text in a preset dictionary tree;
[0008] The preset texts matching each of the word segmentation indexes are determined as target texts matching the initial input text.
[0009] In a second aspect, the present application further provides a text matching device based on a word segmenter and a dictionary tree, the device comprising:
[0010] An input word segmentation segment determination module, used to determine at least one set of input word segmentation segments based on the user's initial input text and a preset word segmenter;
[0011] A text matching module, used to use each of the input word segmentation fragments as a word segmentation index, and match the word segmentation index with each preset text in a preset dictionary tree;
[0012] The target text determination module is used to determine the preset texts matching each of the word segmentation indexes as target texts matching the initial input text.
[0013] In a third aspect, the present application also provides a computer device, comprising a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and implement the text matching method based on a word segmenter and a dictionary tree as described above when executing the computer program.
[0014] In a fourth aspect, the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor implements the text matching method based on a word segmenter and a dictionary tree as described above.
[0015] The present application discloses a text matching method, device, equipment and medium based on a word segmenter and a dictionary tree. The text matching method based on a word segmenter and a dictionary tree includes determining at least one group of input word segmentation fragments based on the user's initial input text and a preset word segmenter; using each of the input word segmentation fragments as a word segmentation index, and matching the word segmentation index with each preset text in a preset dictionary tree; determining the preset text that matches each of the word segmentation indexes as a target text that matches the initial input text. In the above manner, the present application can more accurately identify and match words and phrases in the text through the combined use of a preset word segmenter and a dictionary tree, thereby reducing the possibility of misidentification and mismatching. By using a preset dictionary tree, a large amount of preset texts can be quickly searched and matched, thereby improving the efficiency of the entire word segmentation and matching process, and improving the accuracy of text matching based on keywords input by users in the financial field. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0017] Figure 1 is a schematic flow chart of a text matching method based on a word segmenter and a dictionary tree provided in the first embodiment of the present application;
[0018] Figure 2 is a schematic flow chart of a text matching method based on a word segmenter and a dictionary tree provided in the second embodiment of the present application;
[0019] Figure 3 A diagram of the association between a dictionary tree and a word segmenter in a text matching method based on a word segmenter and a dictionary tree provided in the second embodiment of the present application;
[0020] Figure 4 A schematic block diagram of a text matching device based on a word segmenter and a dictionary tree provided in an embodiment of the present application;
[0021] Figure 5 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0022] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0023] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.
[0024] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0025] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0026] The embodiments of the present application provide a text matching method, device, equipment and medium based on a word segmenter and a dictionary tree. Among them, the text matching method based on a word segmenter and a dictionary tree can be applied to a server. By using a preset word segmenter and a dictionary tree in combination, words and phrases in the text can be more accurately identified and matched, thereby reducing the possibility of misidentification and mismatching. By using a preset dictionary tree, a large amount of preset text can be searched and matched quickly, thereby improving the efficiency of the entire word segmentation and matching process, and improving the accuracy of text matching based on keywords entered by users in the financial field. Among them, the server can be an independent server or a server cluster.
[0027] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.
[0028] See also Figure 1 , Figure 1 This is a schematic flow chart of a text matching method based on a word segmenter and a dictionary tree provided by the first embodiment of the present application. The text matching method based on a word segmenter and a dictionary tree can be applied to a server, and is used to more accurately identify and match words and phrases in a text by combining a preset word segmenter and a dictionary tree, thereby reducing the possibility of misidentification and mismatching. By using a preset dictionary tree, a large amount of preset text can be searched and matched quickly, thereby improving the efficiency of the entire word segmentation and matching process, and improving the accuracy of text matching based on keywords entered by users in the financial field.
[0029] like Figure 1 As shown, the text matching method based on the word segmenter and the dictionary tree specifically includes steps S10 to S30.
[0030] Step S10: determining at least one set of input word segmentation segments based on the user's initial input text and a preset word segmenter;
[0031] Step S20: taking each of the input word segmentation fragments as a word segmentation index, and matching the word segmentation index with each preset text in a preset dictionary tree;
[0032] Step S30: Determine the preset texts that match the respective word segmentation indexes as target texts that match the initial input texts.
[0033] In one embodiment, after accessing the data source, this embodiment pre-processes the data, cleans invalid data and classifies and organizes them into a structured database according to the type of data source. Then, for each type of indicator name, a word segmenter is used in combination with a domain-specific knowledge base to segment the short text in the database, and a mapping table from word segmentation to indicator name address is maintained.
[0034] For example, for the indicator name "comprehensive cost rate", the word segmentation model will divide it into three words: "comprehensive", "cost" and "rate". At the same time, the mapping table records that the three words "comprehensive", "cost" and "rate" correspond to the address of the indicator name "comprehensive cost rate".
[0035] This embodiment uses a word segmenter to automatically segment words according to the characteristics of the indicator name in the word library, and constructs the segmentation results into a dictionary tree. The dictionary tree can improve the search speed, and the dictionary tree can be used as a word segmenter for user input to improve the accuracy of word segmentation; when the user enters a string for search, the system will retrieve word fragments through the dictionary tree, and automatically segment the user input into word fragments with a high degree of matching with the word segmentation results in the dictionary tree, which greatly reduces the impact of missing words and typos on the search, greatly improves the speed and accuracy of searches in special scenarios in the financial field, and improves the user's experience
[0036] The present embodiment discloses a text matching method, device, equipment and medium based on a word segmenter and a dictionary tree. The text matching method based on a word segmenter and a dictionary tree includes determining at least one group of input word segmentation fragments based on the user's initial input text and a preset word segmenter; using each of the input word segmentation fragments as a word segmentation index, and matching the word segmentation index with each preset text in a preset dictionary tree; determining the preset text that matches each of the word segmentation indexes as a target text that matches the initial input text. In the above manner, the present application can more accurately identify and match words and phrases in the text through the combined use of a preset word segmenter and a dictionary tree, thereby reducing the possibility of misidentification and mismatching. By using a preset dictionary tree, a large amount of preset texts can be quickly searched and matched, thereby improving the efficiency of the entire word segmentation and matching process, and improving the accuracy of text matching based on keywords input by users in the financial field.
[0037] See also Figure 2 , Figure 2This is a schematic flow chart of a text matching method based on a word segmenter and a dictionary tree provided in the second embodiment of the present application. The text matching method based on a word segmenter and a dictionary tree can be applied to a server. This embodiment performs accurate word segmentation processing on the training set text and generates high-quality training set word segmentations, laying a solid foundation for subsequent text analysis and processing. Through deep integration with the knowledge domain library, the system can establish an accurate mapping relationship between the training set word segmentations and the names of knowledge domain indicators, enhance the depth and breadth of text analysis, quickly determine the matching relationship between the user's initial input text and the preset word segmenter, generate at least one set of input word segmentation fragments, achieve efficient text matching, and improve the accuracy of text matching based on user input keywords in the financial field.
[0038] based on Figure 1 In the illustrated embodiment, in this embodiment, the steps before step S10 include steps S01 to S03.
[0039] Step S01, obtaining a training set text, and performing word segmentation processing on the training set text to generate training set word segmentations;
[0040] Step S02: Establishing a mapping relationship between the training set segmentation words and the indicator names in the knowledge domain library through the knowledge domain library and the training set text;
[0041] Step S03: Generate the preset word segmenter based on each of the mapping relationships.
[0042] In one embodiment, Figure 3 As shown, Figure 3 A diagram of the association between a dictionary tree and a word segmenter is provided for a text matching method based on a word segmenter and a dictionary tree in the second embodiment of the present application. Figure 3 As shown, the word segmenter and the dictionary tree are pre-trained and applied to this embodiment. The text content input by the user is received, and the text content input by the user is segmented and matched with each segmentation result in the pre-trained dictionary tree. The word fragment is retrieved from the dictionary tree, and then the index is determined by the mapping relationship established by the word segmenter and the word fragment. The index here is the noun confirmed in the word segmenter after training in advance.
[0043] The present embodiment discloses a text matching method, device, equipment and medium based on a word segmenter and a dictionary tree. The text matching method based on a word segmenter and a dictionary tree includes obtaining a training set text, performing word segmentation processing on the training set text, and generating training set word segmentations; establishing a mapping relationship between the training set word segmentations and each indicator name in the knowledge domain library through each of the knowledge domain libraries and the training set text; generating the preset word segmenter based on each of the mapping relationships; determining at least one group of input word segmentation fragments based on the user's initial input text and the preset word segmenter; using each of the input word segmentation fragments as a word segmentation index, and matching the word segmentation index with each preset text in a preset dictionary tree; determining the preset text matching each of the word segmentation indexes as the target text matching the initial input text. Through the above method, this embodiment performs accurate word segmentation processing on the training set text, generates high-quality training set word segmentations, and lays a solid foundation for subsequent text analysis and processing. Through deep integration with the knowledge domain library, the system can establish an accurate mapping relationship between the training set word segmentations and the knowledge domain indicator names, enhance the depth and breadth of text analysis, quickly determine the matching relationship between the user's initial input text and the preset word segmenter, generate at least one set of input word segmentation fragments, achieve efficient text matching, and improve the accuracy of text matching based on user input keywords in the financial field.
[0044] based on Figure 2 In the embodiment shown, in this embodiment, before step S20, the following steps are included:
[0045] Based on the preset saving rules, each of the training set word segments is constructed into a dictionary tree format, and the preset dictionary tree is generated according to each of the training set word segments in the dictionary tree format.
[0046] In one embodiment, after accessing the data source, various types of data are classified and organized into a structured database, and then for each type of indicator name, a word segmenter is used in combination with a domain-specific knowledge base to segment the short text in the database, and a mapping table from segmented words to indicator name addresses is maintained.
[0047] For example, for the indicator name "comprehensive cost rate", the word segmentation model will divide it into three words: "comprehensive", "cost" and "rate". At the same time, the mapping table records that the three words "comprehensive", "cost" and "rate" correspond to the address of the indicator name "comprehensive cost rate".
[0048] The words with a length of not less than 2 in all the above segmentation results are constructed into a dictionary tree form and stored in memory for real-time fast search. After the word is queried from the dictionary tree, the original indicator name is found from the word through the mapping table. Only the words with a length of not less than 2 are retained here to avoid too many words and reduce the search efficiency. It can be understood that the specific value of the retained length can be set by the user and is not limited here.
[0049] The present embodiment discloses a text matching method, device, equipment and medium based on a word segmenter and a dictionary tree. The text matching method based on a word segmenter and a dictionary tree includes obtaining a training set text, performing word segmentation processing on the training set text, and generating training set word segments; establishing a mapping relationship between the training set word segments and each indicator name in the knowledge domain library through each of the knowledge domain libraries and the training set text; generating the preset word segmenter based on each of the mapping relationships; determining at least one group of input word segmentation fragments based on the user's initial input text and the preset word segmenter; constructing each of the training set word segments into a dictionary tree format based on a preset saving rule, and generating the preset dictionary tree according to each of the training set word segments in the dictionary tree format; using each of the input word segmentation fragments as a word segmentation index, and matching the word segmentation index with each preset text in the preset dictionary tree; determining the preset text matching each of the word segmentation indexes as the target text matching the initial input text. Through the above method, this application can generate accurate training set word segmentation through word segmentation processing of training set text, provide high-quality basic data for subsequent processing, and based on the mapping relationship, the system can generate preset word segmenters that meet different needs, improving the flexibility and adaptability of word segmentation. Through preset saving rules, the system can construct the training set word segmentation into a dictionary tree format, optimize the storage structure, speed up the retrieval speed, and improve the accuracy of text matching based on the keywords entered by the user in the financial field.
[0050] In a specific embodiment, determining the preset text matching each of the word segmentation indexes as the target text matching the initial input text includes:
[0051] Based on the word segmentation index, determining the indicator name in the preset dictionary tree;
[0052] Based on the indicator name and the mapping relationship, the target text is determined.
[0053] In one embodiment, the user needs to provide the text data to be analyzed, and the system automatically performs word segmentation on the input text to generate training set word segments. This embodiment establishes a mapping relationship between the training set word segments and the indicator names in the knowledge domain library, and generates a preset word segmenter based on the mapping relationship for subsequent text matching. This embodiment constructs the training set word segments in a dictionary tree format to improve matching efficiency.
[0054] The user provides a word segmentation index, and the system determines the corresponding index name in the preset dictionary tree. This embodiment determines the target text that matches the user input text based on the determined index name and mapping relationship to ensure the quality of the input text and obtain a more accurate word segmentation result. The word segmentation index should accurately reflect the user's query intent.
[0055] The present embodiment discloses a text matching method, device, equipment and medium based on a word segmenter and a dictionary tree. The text matching method based on the word segmenter and the dictionary tree includes obtaining a training set text, performing word segmentation processing on the training set text, and generating training set word segments; establishing a mapping relationship between the training set word segments and each indicator name in the knowledge domain library through each of the knowledge domain libraries and the training set text; generating the preset word segmenter based on each of the mapping relationships; determining at least one group of input word segmentation fragments based on the user's initial input text and the preset word segmenter; constructing each of the training set word segments into a dictionary tree format based on a preset saving rule, and generating the preset dictionary tree according to each of the training set word segments in the dictionary tree format; using each of the input word segmentation fragments as a word segmentation index, and matching the word segmentation index with each preset text in the preset dictionary tree; determining the indicator name in the preset dictionary tree based on the word segmentation index; and determining the target text based on the indicator name and the mapping relationship. Through the above method, this application uses word segmentation index and preset dictionary tree, and the system can quickly locate and retrieve the indicator name in the text, which greatly improves the efficiency of text retrieval, and uses word segmentation index to accurately determine the indicator name in the dictionary tree, reducing the indicator identification error caused by word segmentation errors. Through the construction of mapping relationships and dictionary trees, the system helps to manage and organize indicator names in the knowledge field, facilitates user access and use, and improves the accuracy of text matching based on user-entered keywords in the financial field.
[0056] based on Figure 1 In the embodiment shown, in this embodiment, step S10 includes:
[0057] Dividing the initial input text into at least one group of initial phrases by the preset word segmenter;
[0058] Determining the index association between each of the initial phrases through the preset dictionary tree;
[0059] At least one group of the input word segments is determined based on the index association and each of the initial word groups.
[0060] The present embodiment discloses a text matching method, device, equipment and medium based on a word segmenter and a dictionary tree. The text matching method based on a word segmenter and a dictionary tree includes dividing the initial input text into at least one group of initial phrases by the preset word segmenter; determining the index association between each of the initial phrases by the preset dictionary tree; determining at least one group of input word segmentation fragments based on the index association and each of the initial phrases; using each of the input word segmentation fragments as a word segmentation index, and matching the word segmentation index with each preset text in the preset dictionary tree; determining the preset text matched with each of the word segmentation indexes as the target text matched with the initial input text. In the above manner, the present application can more accurately identify and match words and phrases in the text through the combined use of the preset word segmenter and the dictionary tree, thereby reducing the possibility of misidentification and mismatching. By using the preset dictionary tree, a large amount of preset texts can be quickly searched and matched, thereby improving the efficiency of the entire word segmentation and matching process, and improving the accuracy of text matching based on keywords input by users in the financial field.
[0061] In a specific embodiment, based on the index relevance and each of the initial phrases, determining at least one group of the input word segments includes:
[0062] Determining the initial phrases whose index relevance is lower than a preset relevance threshold as low-relevance phrases;
[0063] The low-association phrase is discarded, and the initial phrase after discarding the low-association phrase is determined as the input word segmentation segment.
[0064] In one embodiment, when the user enters "comprehensive cost rate", the system will retrieve the word fragments "comprehensive" and "cost" through the dictionary tree; when the user enters "total cost rate", the traditional search engine's word segmentation results are "total", "synthesis" and "rate". At this time, any fragment will not match any results in the index library; but in this system, the system will retrieve the word fragment "cost" through the dictionary tree, which is equivalent to dividing the word into "total", "cost" and "rate", but "total" and "rate" are discarded because they have no corresponding word segmentation index.
[0065] The present embodiment discloses a text matching method, device, equipment and medium based on a word segmenter and a dictionary tree. The text matching method based on a word segmenter and a dictionary tree includes dividing the initial input text into at least one group of initial phrases through the preset word segmenter; determining the index association between each of the initial phrases through the preset dictionary tree; determining the initial phrases whose index association is lower than the preset association threshold as low association phrases; discarding the low association phrases, and determining the initial phrases after discarding the low association phrases as the input word segmentation fragments; using each of the input word segmentation fragments as a word segmentation index, and matching the word segmentation index with each preset text in the preset dictionary tree; determining the preset texts matched with each of the word segmentation indexes as target texts matched with the initial input text. In the above manner, the present application can more accurately identify and match words and phrases in the text through the combined use of the preset word segmenter and the dictionary tree, thereby reducing the possibility of misidentification and mismatching. By presetting the dictionary tree, a large amount of preset text can be quickly searched and matched, thereby improving the efficiency of the entire word segmentation and matching process, and improving the accuracy of text matching based on user-entered keywords in the financial field.
[0066] In a specific embodiment, determining at least one group of the input word segments based on the index relevance and each of the initial word groups further includes:
[0067] Determining the initial phrases whose index relevance is greater than or equal to the preset relevance threshold as high-relevance phrases;
[0068] Using the highly associated phrase as the word segmentation index, determining related phrases from the preset dictionary tree;
[0069] The related word group is combined with the high-association word group to generate the input word segment segment.
[0070] In one embodiment, as described in the above embodiment, when a user inputs "combined cost rate", the word segmentation results of the traditional search engine are "zong", "synthesis" and "ben rate". At this time, any fragment cannot match any results in the index library; but in this system, the system will retrieve the word fragment "cost" through the dictionary tree, which is equivalent to dividing the word into "zong", "cost" and "rate", but "zong" and "rate" are discarded because there is no corresponding word segmentation index, and only the word fragment "cost" is used as the word segmentation index, and then the mapping table finds the indicator name "comprehensive cost rate" corresponding to "cost" as the return.
[0071] In addition, since "zonghe" can be recognized as a misspelling in this embodiment, according to the index correlation degree, it is corrected to "comprehensive", and then the user input is recognized as "zonghe", "cost", and "rate", realizing the correction of the misspelling in the user input and improving the accuracy and efficiency of text matching.
[0072] This embodiment discloses a text matching method, device, equipment and medium based on a word segmenter and a trie tree. The text matching method based on the word segmenter and the trie tree includes dividing the initial input text into at least one group of initial phrases through the preset word segmenter; determining the index correlation degree between the initial phrases through the preset trie tree; determining the initial phrases with the index correlation degree greater than or equal to the preset correlation degree threshold as high-correlation phrases; using the high-correlation phrases as the word segment indexes to determine relevant phrases from the preset trie tree; combining the relevant phrases with the high-correlation phrases to generate the input word segment fragments; using each of the input word segment fragments as a word segment index and matching the word segment index with each preset text in the preset trie tree; and determining the preset text that matches each word segment index as the target text that matches the initial input text. By the above method, through the combined use of the preset word segmenter and the trie tree, the present application can more accurately identify and match the words and phrases in the text, thereby reducing the possibility of misrecognition and incorrect matching. Through the preset trie tree, it can quickly search and match in a large number of preset texts, thereby improving the efficiency of the entire word segmenting and matching process and improving the accuracy of text matching according to the keywords input by the user in the financial field.
[0073] Based on the above embodiments, the innovation points of this embodiment are mainly as follows:
[0074] (1) Construct the word segmentation results into a trie tree. The traditional search engine only uses the word segmentation results as indexes for matching with the user input segments. This solution constructs these words into a trie tree. On the one hand, it can improve the search speed, and on the other hand, it can utilize the characteristics of the trie tree and use it as the word segmenter for user input.
[0075] (2) Two different word segmenters are used for the word library index and user input. Traditional search engines use the same word segmenter, which results in forced and incorrect word segmentation of user input, affecting the accuracy of the search. To address this problem, this solution uses two different word segmenters to process the original data index and user input respectively. For the original index data, a word segmenter combined with a domain-specific knowledge base is used to segment the words, and the word segmentation results are constructed into a dictionary tree; the dictionary tree is used as the word segmenter for the user's input, and the dictionary tree constructed with the word segmentation results is used to partially match the user's input, quickly retrieve the word fragments of the user input that exist in the dictionary tree, so that the word segmentation fragments input by the user and the word segmentation in the dictionary tree have a very high match, reducing the impact of missing words and typos on the search and improving the accuracy of the search.
[0076] See also Figure 4 , Figure 4 The embodiment of the present application provides a schematic block diagram of a text matching device based on a word segmenter and a dictionary tree, and the text matching device based on a word segmenter and a dictionary tree is used to execute the aforementioned text matching method based on a word segmenter and a dictionary tree. The text matching device based on a word segmenter and a dictionary tree can be configured on a server.
[0077] like Figure 4 As shown, the text matching device 400 based on a word segmenter and a dictionary tree includes:
[0078] An input word segmentation segment determination module 410 is used to determine at least one set of input word segmentation segments based on the user's initial input text and a preset word segmenter;
[0079] A text matching module 420, configured to use each of the input word segmentation fragments as a word segmentation index, and match the word segmentation index with each preset text in a preset dictionary tree;
[0080] The target text determination module 430 is configured to determine the preset texts matching each of the word segmentation indexes as target texts matching the initial input text.
[0081] Furthermore, the text matching device 400 based on a word segmenter and a dictionary tree further includes:
[0082] A training set word segmentation generation module is used to obtain training set text, perform word segmentation processing on the training set text, and generate training set word segmentations;
[0083] A mapping relationship establishment module, used to establish a mapping relationship between the training set segmentation words and the indicator names in the knowledge domain library through the knowledge domain library and the training set text;
[0084] The preset word segmenter generation module is used to generate the preset word segmenter based on each of the mapping relationships.
[0085] Furthermore, the text matching device 400 based on a word segmenter and a dictionary tree further includes:
[0086] The preset dictionary tree generation module is used to construct each of the training set word segments into a dictionary tree format based on a preset saving rule, and generate the preset dictionary tree according to each of the training set word segments in the dictionary tree format.
[0087] Furthermore, the target text determination module 430 includes:
[0088] An indicator name determination unit, configured to determine the indicator name in the preset dictionary tree based on the word segmentation index;
[0089] A target text determination unit is used to determine the target text based on the indicator name and the mapping relationship.
[0090] Furthermore, the input word segmentation segment determination module 410 includes:
[0091] An initial phrase generating unit, configured to divide the initial input text into at least one group of initial phrases by means of the preset word segmenter;
[0092] An index relevance determination unit, used to determine the index relevance between the initial phrases through the preset dictionary tree;
[0093] The input word segmentation fragment determination unit is used to determine at least one group of the input word segmentation fragments based on the index association degree and each of the initial word groups.
[0094] Furthermore, the input word segmentation fragment determination unit includes:
[0095] A low-association phrase determination subunit, configured to determine the initial phrase whose index association is lower than a preset association threshold as a low-association phrase;
[0096] The first input word segmentation fragment determination subunit is used to discard the low-association phrase and determine the initial phrase after discarding the low-association phrase as the input word segmentation fragment.
[0097] Furthermore, the input word segmentation segment determination unit further includes:
[0098] A high-association phrase determination subunit, configured to determine an initial phrase whose index association is greater than or equal to the preset association threshold as a high-association phrase;
[0099] A related phrase determination subunit, configured to use the highly associated phrase as the word segmentation index and determine the related phrase from the preset dictionary tree;
[0100] The second input word segmentation segment determination subunit is used to combine the related word group with the high-association word group to generate the input word segmentation segment.
[0101] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and each module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0102] The above-mentioned device can be implemented in the form of a computer program. Figure 5 Runs on the computer device shown.
[0103] See also Figure 5 , Figure 5 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device may be a server.
[0104] See also Figure 5 The computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.
[0105] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any text matching method based on a word segmenter and a dictionary tree.
[0106] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.
[0107] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any text matching method based on the word segmenter and the dictionary tree.
[0108] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will appreciate that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0109] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0110] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:
[0111] Determine at least one set of input word segmentation segments based on the user's initial input text and a preset word segmenter;
[0112] Using each of the input word segmentation fragments as a word segmentation index, and matching the word segmentation index with each preset text in a preset dictionary tree;
[0113] The preset texts matching each of the word segmentation indexes are determined as target texts matching the initial input text.
[0114] In one embodiment, before determining at least one set of input word segmentation segments based on the user's initial input text and a preset word segmenter, it is used to implement:
[0115] Acquire training set text, and perform word segmentation processing on the training set text to generate training set word segmentations;
[0116] By using each of the knowledge domain libraries and the training set text, a mapping relationship between the training set word segmentation and each indicator name in the knowledge domain library is established;
[0117] Based on each of the mapping relationships, the preset word segmenter is generated.
[0118] In one embodiment, each of the input word segmentation fragments is used as a word segmentation index, and before the word segmentation index is matched with each preset text in a preset dictionary tree, it is used to achieve:
[0119] Based on the preset saving rules, each of the training set word segments is constructed into a dictionary tree format, and the preset dictionary tree is generated according to each of the training set word segments in the dictionary tree format.
[0120] In one embodiment, the preset texts matching each of the word segmentation indexes are determined as target texts matching the initial input texts, so as to achieve:
[0121] Based on the word segmentation index, determining the indicator name in the preset dictionary tree;
[0122] Based on the indicator name and the mapping relationship, the target text is determined.
[0123] In one embodiment, based on the user's initial input text and a preset word segmenter, at least one set of input word segmentation segments is determined to achieve:
[0124] Dividing the initial input text into at least one group of initial phrases by the preset word segmenter;
[0125] Determining the index association between each of the initial phrases through the preset dictionary tree;
[0126] At least one group of the input word segments is determined based on the index association and each of the initial word groups.
[0127] In one embodiment, based on the index relevance and each of the initial word groups, at least one group of the input word segments is determined to achieve:
[0128] Determining the initial phrases whose index relevance is lower than a preset relevance threshold as low-relevance phrases;
[0129] The low-association phrase is discarded, and the initial phrase after discarding the low-association phrase is determined as the input word segmentation segment.
[0130] In one embodiment, based on the index association and each of the initial phrases, at least one group of the input word segmentation segments is determined, and is also used to implement:
[0131] Determining the initial phrases whose index relevance is greater than or equal to the preset relevance threshold as high-relevance phrases;
[0132] Using the highly associated phrase as the word segmentation index, determining related phrases from the preset dictionary tree;
[0133] The related word group is combined with the high-association word group to generate the input word segment segment.
[0134] A computer-readable storage medium is also provided in an embodiment of the present application, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the processor executes the program instructions to implement any one of the text matching methods based on a word segmenter and a dictionary tree provided in the embodiments of the present application.
[0135] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (Smart Medi a Card, SMC), a secure digital (Secure Digital, SD) card, a flash memory card (Flash Card), etc., equipped on the computer device.
[0136] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A text matching method based on word segmenter and dictionary tree, It is characterized in that include: Determine at least one set of input word segmentation segments based on the user's initial input text and a preset word segmenter; Using each of the input word segmentation fragments as a word segmentation index, and matching the word segmentation index with each preset text in a preset dictionary tree; The preset texts matching each of the word segmentation indexes are determined as target texts matching the initial input text.
2. The text matching method based on a word segmenter and a dictionary tree according to claim 1, It is characterized in that The preset word segmenter includes at least one knowledge domain library. Before determining at least one group of input word segmentation segments based on the user's initial input text and the preset word segmenter, the method includes: Acquire training set text, and perform word segmentation processing on the training set text to generate training set word segmentations; By using each of the knowledge domain libraries and the training set text, a mapping relationship between the training set word segmentation and each indicator name in the knowledge domain library is established; Based on each of the mapping relationships, the preset word segmenter is generated.
3. The text matching method based on a word segmenter and a dictionary tree according to claim 2, It is characterized in that Before taking each input word segmentation fragment as a word segmentation index and matching the word segmentation index with each preset text in a preset dictionary tree, the method includes: Based on the preset saving rules, each of the training set word segments is constructed into a dictionary tree format, and the preset dictionary tree is generated according to each of the training set word segments in the dictionary tree format.
4. The text matching method based on a word segmenter and a dictionary tree according to claim 3, It is characterized in that The step of determining the preset texts matching each of the word segmentation indexes as target texts matching the initial input text includes: Based on the word segmentation index, determining the indicator name in the preset dictionary tree; Based on the indicator name and the mapping relationship, the target text is determined.
5. The text matching method based on a word segmenter and a dictionary tree according to claim 1, It is characterized in that The step of determining at least one set of input word segmentation segments based on the user's initial input text and a preset word segmenter includes: Dividing the initial input text into at least one group of initial phrases by the preset word segmenter; Determining the index association between each of the initial phrases through the preset dictionary tree; At least one group of the input word segments is determined based on the index association and each of the initial word groups.
6. The text matching method based on a word segmenter and a dictionary tree according to claim 5, It is characterized in that The step of determining at least one group of input word segments based on the index association and each of the initial word groups includes: Determining the initial phrases whose index relevance is lower than a preset relevance threshold as low-relevance phrases; The low-association phrase is discarded, and the initial phrase after discarding the low-association phrase is determined as the input word segmentation segment.
7. The text matching method based on a word segmenter and a dictionary tree according to claim 6, It is characterized in that The step of determining at least one group of the input word segments based on the index association and each of the initial word groups further includes: Determining the initial phrases whose index relevance is greater than or equal to the preset relevance threshold as high-relevance phrases; Using the highly associated phrase as the word segmentation index, determining related phrases from the preset dictionary tree; The related word group is combined with the high-association word group to generate the input word segment segment.
8. A text matching device based on word segmenter and dictionary tree, It is characterized in that include: An input word segmentation segment determination module, used to determine at least one set of input word segmentation segments based on the user's initial input text and a preset word segmenter; A text matching module, used to use each of the input word segmentation fragments as a word segmentation index, and match the word segmentation index with each preset text in a preset dictionary tree; The target text determination module is used to determine the preset texts matching each of the word segmentation indexes as target texts matching the initial input text.
9. A computer device, It is characterized in that The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is used to execute the computer program and implement the text matching method based on a word segmenter and a dictionary tree as described in any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the text matching method based on a word segmenter and a dictionary tree as claimed in any one of claims 1 to 7.