Word segmentation search method, device, equipment, medium and product based on configuration table

Through the word segmentation search method based on the configuration table, the problem of lack of customized functions and sensitive word processing in the prior art is solved, and more accurate search results and data security are achieved, and user experience is improved.

CN118627496BActive Publication Date: 2024-12-03CHENGDU SHANGSHANG YUNHUI INFORMATION TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410808788.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-21
Publication Date
2024-12-03
Estimated Expiration
2044-06-21

AI Technical Summary

Technical Problem

The existing word segmentation search solution lacks customizable functions and cannot effectively handle the filtering of proper nouns, sensitive word and Chinese character pinyin processing, affecting the accuracy, data security and user experience of search results.

Method used

The word segmentation search method based on the configuration table is adopted, and word segmentation processing, industry-specific word addition, sensitive word processing and pinyin processing are carried out by obtaining pre-configured word segmentation tools, industry-specific word addition, sensitive word processing and pinyin processing. Finally, the text to be retrieved is imported into the distributed search engine Elasticsearch for query.

Benefits of technology

It improves the accuracy and customization of search results, supports proprietary noun word segmentation, sensitive word filtering and Chinese character pinyin processing, ensuring data security and user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118627496B_ABST
    Figure CN118627496B_ABST
Patent Text Reader

Abstract

The present invention discloses a word segmentation search method, device, equipment, medium and product based on a configuration table, and relates to the technical field of information retrieval and natural language processing. The method is to, after receiving a word segmentation search request initiated by a client and carrying an original text for a word segmentation search document, select a word segmenter in turn to perform word segmentation, add industry-specific words, process sensitive words, add pinyin words, and assemble the text to be retrieved based on a configuration table configured with multiple word segmenters corresponding to multiple text types, multiple industry-specific word sets corresponding to multiple industries, sensitive word sets, sensitive word processing rules, and text assembly rules, and finally import the text to be retrieved into a distributed search and analysis engine Elasticsearch, obtain a document recall result returned by Elasticsearch, and give feedback to the client, so that the accuracy of search results, data security and user search experience can be guaranteed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of information retrieval and natural language processing, and specifically relates to a word segmentation search method, device, equipment, medium and product based on a configuration table. Background Art

[0002] At present, the basic principle of the existing word segmentation search function is to store data in a distributed, real-time and scalable data storage layer, and provide full-text search, structured search and analysis functions; its current status is very widespread and is used to build various search engines, log analysis systems, monitoring systems and recommendation systems. Therefore, the word segmentation search function has a wide range of applications in the fields of Internet, e-commerce, finance, medical care and media.

[0003] However, the existing word segmentation search solution also has some defects. For example, although it provides rich functions and flexible configuration options, its learning curve is steep, and configuration and tuning require certain experience and skills, which will lead to the lack of customizable functions. It also does not support proper noun segmentation, sensitive word filtering, and Chinese character pinyin processing, which affects the accuracy of search results, data security and user search experience. Summary of the invention

[0004] The purpose of the present invention is to provide a word segmentation search method, device, computer equipment, computer-readable storage medium and computer program product based on a configuration table, so as to solve the problems existing in existing word segmentation search schemes, such as the lack of customizable functions and the lack of support for proper noun segmentation, sensitive word filtering and Chinese character pinyin processing, which affect the accuracy of search results, data security and user search experience.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] In a first aspect, a word segmentation search method based on a configuration table is provided, comprising:

[0007] Obtaining a configuration table pre-configured with a plurality of word segmenters corresponding one-to-one to a plurality of text types, a plurality of industry-specific word sets corresponding one-to-one to a plurality of industries, a sensitive word set, sensitive word processing rules, and text assembly rules;

[0008] Receiving a word segmentation search request initiated by a client, wherein the word segmentation search request carries an original text for a word segmentation search document;

[0009] If the text type of the original text is specified as the first text type among the multiple text types in the word segmentation search request, a first word segmenter corresponding to the first text type is found in the configuration table, and the first word segmenter is applied to perform word segmentation processing on the original text to obtain a word segmentation result;

[0010] If the search target industry field is specified as a certain industry among the multiple industries in the word segmentation search request, a certain industry-specific word set corresponding to the certain industry is found in the configuration table, and for each industry-specific word in the certain industry-specific word set, if a corresponding word is found to appear in the original text, the corresponding word is added to the word segmentation result;

[0011] Extracting the sensitive word set from the configuration table, and for each sensitive word in the sensitive word set, if a corresponding word is found to appear in the word segmentation result, extracting the sensitive word processing rule from the configuration table to process the corresponding word in the word segmentation result;

[0012] For each Chinese word in the word segmentation result, generate a corresponding pinyin word, and add the pinyin word to the word segmentation result;

[0013] Extracting the text assembly rule from the configuration table, and applying the text assembly rule to perform text assembly on the words in the word segmentation result to obtain the text to be retrieved;

[0014] Importing the to-be-searched text into the distributed search and analysis engine Elasticsearch to obtain a document recall result returned by the distributed search and analysis engine Elasticsearch;

[0015] The document recall result is fed back to the client.

[0016] Based on the above invention content, a new solution for word segmentation search based on a configuration table and a distributed search and analysis engine Elasticsearch is provided, that is, after receiving a word segmentation search request initiated by a client and carrying an original text for a word segmentation search document, based on a configuration table configured with multiple word segmenters corresponding to multiple text types, multiple industry-specific word sets corresponding to multiple industries, sensitive word sets, sensitive word processing rules and text assembly rules, a word segmenter is selected in sequence to perform word segmentation, add industry-specific words, process sensitive words, add pinyin words and assemble the text to be retrieved, and finally the text to be retrieved is imported into the distributed search and analysis engine Elasticsearch, and a document recall result returned by Elasticsearch is obtained, and feedback is given to the client, so that not only can the search function of Elasticsearch be utilized and matching can be performed according to the previously segmented words, which greatly improves the accuracy and industry customization and completes the customization function, but also can support proper noun segmentation, sensitive word filtering and Chinese character pinyin processing, thereby ensuring the accuracy of search results, data security and user search experience, and facilitating practical application and promotion.

[0017] In one possible design, if the text type of the original text is not specified in the word segmentation search request or the text type of the original text is specified in the word segmentation search request and is not in the multiple text types, the method further includes:

[0018] Using a natural language processing library to perform text type recognition processing on the original text, to obtain a recognition result indicating that the text type of the original text is a second text type among the multiple text types;

[0019] Finding a second word segmenter corresponding to the second text type in the configuration table;

[0020] The second word segmenter is used to perform word segmentation processing on the original text to obtain a word segmentation result.

[0021] In a possible design, if the search target industry field is not specified in the word segmentation search request or the search target industry field specified in the word segmentation search request is not in the multiple industries, the method further includes:

[0022] Extracting the plurality of industry-specific word sets from the configuration table;

[0023] For each industry-specific word in the plurality of industry-specific word sets, if a corresponding word is found to appear in the original text, the corresponding word is added to the word segmentation result.

[0024] In one possible design, after receiving the word segmentation search request and before performing word segmentation processing on the original text, the method further includes:

[0025] The original text is put into the distributed publish-subscribe messaging system Kafka, so as to perform word segmentation processing on the original text with high throughput in combination with the distributed publish-subscribe messaging system Kafka.

[0026] In one possible design, the method is integrated into a Spring project that introduces the distributed publish-subscribe messaging system Kafka and the distributed search and analysis engine Elasticsearch into an open source Java / Java EE full-stack application framework.

[0027] In a possible design, when there are multiple sensitive word processing rules and they have a one-to-many relationship or a one-to-one relationship with the sensitive words in the sensitive word set, for each sensitive word in the sensitive word set, if a corresponding word is found to appear in the word segmentation result, the sensitive word processing rule is extracted from the configuration table to process the corresponding word in the word segmentation result, including:

[0028] If it is found that a certain sensitive word in the sensitive word set appears in the word segmentation result, the sensitive word processing rule corresponding to the certain sensitive word is extracted from the configuration table to process the certain sensitive word in the word segmentation result.

[0029] In a second aspect, a word segmentation search device based on a configuration table is provided, comprising a configuration table acquisition unit, a search request receiving unit, an initial word segmentation processing unit, an industry-specific word adding unit, a sensitive word processing unit, a pinyin word adding unit, a search text assembly unit, a search engine application unit, and a search result feedback unit;

[0030] The configuration table acquisition unit is used to acquire a configuration table pre-configured with a plurality of word segmenters corresponding one-to-one to a plurality of text types, a plurality of industry-specific word sets corresponding one-to-one to a plurality of industries, a sensitive word set, sensitive word processing rules, and text assembly rules;

[0031] The search request receiving unit is used to receive a word segmentation search request initiated by a client, wherein the word segmentation search request carries an original text for a word segmentation search document;

[0032] The initial word segmentation processing unit is respectively connected to the configuration table acquisition unit and the search request receiving unit for communication, and is used to find a first word segmenter corresponding to the first text type in the configuration table if the text type of the original text is specified as a first text type among the multiple text types in the word segmentation search request, and use the first word segmenter to perform word segmentation processing on the original text to obtain a word segmentation result;

[0033] The industry-specific word adding unit is respectively connected to the configuration table acquisition unit and the initial word segmentation processing unit for communication, and is used to find a certain industry-specific word set corresponding to the certain industry in the configuration table if the search target industry field is specified as a certain industry among the multiple industries in the word segmentation search request, and for each industry-specific word in the certain industry-specific word set, if the corresponding word is found to appear in the original text, then add the corresponding word to the word segmentation result;

[0034] The sensitive word processing unit is respectively connected to the configuration table acquisition unit and the industry-specific word adding unit for extracting the sensitive word set from the configuration table, and for each sensitive word in the sensitive word set, if a corresponding word is found to appear in the word segmentation result, extracting the sensitive word processing rule from the configuration table to process the corresponding word in the word segmentation result;

[0035] The pinyin word adding unit is communicatively connected to the sensitive word processing unit, and is used to generate a corresponding pinyin word for each Chinese word in the word segmentation result, and add the pinyin word to the word segmentation result;

[0036] The search text assembly unit is respectively connected to the configuration table acquisition unit and the pinyin word adding unit for extracting the text assembly rule from the configuration table and applying the text assembly rule to perform text assembly on the words in the word segmentation result to obtain the text to be searched;

[0037] The search engine application unit is communicatively connected to the search text assembly unit, and is used to import the to-be-searched text into the distributed search and analysis engine Elasticsearch, and obtain the document recall result returned by the distributed search and analysis engine Elasticsearch;

[0038] The search result feedback unit is communicatively connected to the search engine application unit and is used to feed back the document recall result to the client.

[0039] In a third aspect, the present invention provides a computer device comprising a memory, a processor and a transceiver which are communicatively connected in sequence, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the word segmentation search method as described in the first aspect or any possible design of the first aspect.

[0040] In a fourth aspect, the present invention provides a computer-readable storage medium having instructions stored thereon. When the instructions are executed on a computer, the word segmentation search method as described in the first aspect or any possible design of the first aspect is executed.

[0041] In a fifth aspect, the present invention provides a computer program product, comprising a computer program or instructions, which, when executed by a computer, implements the word segmentation search method as described in the first aspect or any possible design of the first aspect.

[0042] Beneficial effects of the above scheme:

[0043] (1) The present invention creatively provides a new solution for word segmentation search based on a configuration table and a distributed search and analysis engine Elasticsearch, that is, after receiving a word segmentation search request initiated by a client and carrying an original text for a word segmentation search document, based on a configuration table configured with multiple word segmenters corresponding to multiple text types, multiple industry-specific word sets corresponding to multiple industries, sensitive word sets, sensitive word processing rules, and text assembly rules, a word segmenter is selected in sequence to perform word segmentation, add industry-specific words, process sensitive words, add pinyin words, and assemble the text to be retrieved. Finally, the text to be retrieved is imported into the distributed search and analysis engine Elasticsearch, and the document recall result returned by Elasticsearch is obtained, and feedback is given to the client. In this way, not only can the search function of Elasticsearch be utilized and matching can be performed according to the previously segmented words, thereby greatly improving the accuracy and industry customization and completing the customization function, but also proper noun segmentation, sensitive word filtering, and Chinese character pinyin processing can be supported, thereby ensuring the accuracy of search results, data security, and user search experience.

[0044] (2) Compared with the prior art, the present invention supports the optimal selection of a word segmenter, can segment words according to a specified word segmenter or automatically determine a word segmenter, thereby improving the accuracy of word segmentation, and supports the segmentation of industry-specific nouns. Only the industry-specific nouns need to be configured, and then word segmentation can be performed according to the configured vocabulary to achieve customized word segmentation. It also supports customized processing of sensitive words, that is, the matched sensitive words can be blurred, modified, replaced, deleted, etc., to ensure data security, and also supports automatic Chinese character pinyin processing and word segmentation assembly according to configuration, which is convenient for practical application and promotion. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 A flowchart of a word segmentation search method based on a configuration table provided in an embodiment of the present application.

[0047] Figure 2 A schematic diagram of the structure of a word segmentation search device based on a configuration table provided in an embodiment of the present application.

[0048] Figure 3 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in combination with the drawings and the description of the embodiments or the prior art. Obviously, the following description of the structures of the drawings is only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can be obtained based on these embodiments without creative work. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention.

[0050] It should be understood that although the terms first and second, etc. may be used herein to describe various objects, these objects should not be limited by these terms. These terms are only used to distinguish one object from another object. For example, a first object can be referred to as a second object, and similarly, a second object can be referred to as a first object without departing from the scope of the exemplary embodiments of the present invention.

[0051] It should be understood that the term "and / or" that may appear in this document is merely a description of the association relationship between associated objects, indicating that there may be three relationships. For example, A and / or B can indicate three situations: A exists alone, B exists alone, or A and B exist at the same time. For another example, A, B and / or C can indicate the existence of any one of A, B and C or any combination of them. The term " / and" that may appear in this document describes another type of association object relationship, indicating that there may be two relationships. For example, A / and B can indicate two situations: A exists alone or A and B exist at the same time. In addition, the character " / " that may appear in this document generally indicates that the previous and next associated objects are in an "or" relationship.

[0052] Example

[0053] like Figure 1 As shown, the word segmentation search method provided in the first aspect of this embodiment and based on the configuration table can be, but is not limited to, executed by a computer device having certain computing resources and communicating with the client, such as an electronic device such as a search server. Figure 1 As shown, the word segmentation search method may include but is not limited to the following steps S1 to S9.

[0054] S1. Obtain a configuration table that is pre-configured but not limited to multiple word segmenters corresponding to multiple text types, multiple industry-specific word sets corresponding to multiple industries, sensitive word sets, sensitive word processing rules, and text assembly rules.

[0055] In step S1, the configuration table can be configured by an administrator through a conventional human-computer interaction method, or can be configured by a client user through a conventional human-computer interaction method before initiating a word segmentation search request. In the specific content of the configuration table, the multiple text types include but are not limited to Chinese text types, English text types and / or Japanese text types, etc.; the multiple word segmenters include but are not limited to Standard word segmenters corresponding to Chinese text types or Japanese text types. Tokenizer (which performs word segmentation on Chinese text based on dictionaries and rules to divide the input sentences into independent words) and / or LetterTokenizer (which is an existing word segmenter) corresponding to the English text type; the multiple industries include but are not limited to the e-commerce industry, the intellectual property industry or the printing industry, and each industry and each industry-specific word can be configured and plugged in the configuration table through a conventional disable / enable method to achieve the purpose of flexible configuration; each sensitive word in the sensitive word set can also be configured and plugged in the configuration table through a conventional disable / enable method to achieve the purpose of flexible configuration; the sensitive word processing rules include but are not limited to blurring, modifying, replacing or deleting sensitive words, and there can be multiple sensitive words and have a one-to-many relationship with the sensitive words in the sensitive word set (that is, one sensitive word processing rule corresponds to multiple different sensitive words) or a one-to-one relationship (that is, one sensitive word processing rule corresponds to one sensitive word); the text assembly rules are specifically but not limited to the following examples: Chinese uses Chinese commas, English uses English commas, and / or letters use spaces, etc. In addition, the configuration table may specifically be, but not limited to, a MySQL (ie, relational database management system) configuration table.

[0056] S2. Receive a word segmentation search request initiated by a client, wherein the word segmentation search request carries an original text for the word segmentation search document.

[0057] In step S2, the client is an electronic device held by the search user, such as a personal computer or a smart phone, so as to initiate the word segmentation search request through a conventional human-computer interaction method. In addition, the word segmentation search request may also carry the text type of the original text and / or the search target industry field specified by the search user.

[0058] S3. If the text type of the original text has been specified as the first text type among the multiple text types in the word segmentation search request, the first word segmenter corresponding to the first text type is found in the configuration table, and the first word segmenter is applied to perform word segmentation processing on the original text to obtain a word segmentation result.

[0059] In step S3, for example, if the text type of the original text is specified as a Chinese text type in the word segmentation search request, a standard tokenizer corresponding to the Chinese text type can be found in the configuration table to perform word segmentation on the original text. In addition, in order to find the most suitable tokenizer for word segmentation when the text type is not specified or incorrectly specified, preferably, if the text type of the original text is not specified in the word segmentation search request or the text type of the original text is specified in the word segmentation search request and is not in the multiple text types, the method also includes but is not limited to: first use a natural language processing library to perform text type recognition processing on the original text, and obtain a recognition result indicating that the text type of the original text is a second text type in the multiple text types; then find a second tokenizer corresponding to the second text type in the configuration table; finally, use the second tokenizer to perform word segmentation processing on the original text to obtain a word segmentation result. The aforementioned natural language processing libraries are important tools in the field of artificial intelligence. They can help parse and understand complex language phenomena, and through these libraries, functions such as automatic text classification, sentiment analysis, and intelligent question and answer can be realized. Therefore, the natural language processing library can be used to perform text type recognition processing on the original text to obtain the recognition result.

[0060] In step S3, it is also considered that multiple word segmentation search requests from different clients may be received at the same time. In order to be able to respond to these requests in an orderly and timely manner, preferably, after receiving the word segmentation search request and before the original text is segmented, the method also includes but is not limited to: putting the original text into the distributed publish-subscribe messaging system Kafka, so as to perform word segmentation processing on the original text with high throughput in combination with the distributed publish-subscribe messaging system Kafka. The distributed publish-subscribe messaging system Kafka is an open source stream processing platform developed by the Apache Software Foundation and written in Scala and Java. It can process all action stream data of consumers on the website, so it can be applied to this embodiment to respond to all the word segmentation search requests in an orderly manner.

[0061] S4. If the search target industry field has been specified as a certain industry among the multiple industries in the word segmentation search request, a certain industry-specific word set corresponding to the certain industry is found in the configuration table, and for each industry-specific word in the certain industry-specific word set, if the corresponding word is found to appear in the original text, the corresponding word is added to the word segmentation result.

[0062] In step S4, for example, if the search target industry field has been specified as the e-commerce industry in the word segmentation search request, and there is an abbreviation "B2B" in the e-commerce industry proprietary word set and the abbreviation "B2B" also appears in the original text, then the abbreviation "B2B" is also added to the word segmentation result. In addition, in order to extract industry-specific words from the original text to add to the word segmentation result when the search target industry field is not specified or incorrectly specified, preferably, if the search target industry field is not specified in the word segmentation search request or the search target industry field specified in the word segmentation search request is not in the multiple industries, then the method also includes but is not limited to: extracting the multiple industry-specific word sets from the configuration table; for each industry-specific word in the multiple industry-specific word sets, if the corresponding word is found to appear in the original text, then the corresponding word is added to the word segmentation result. In addition, the natural language processing library can also be used to identify the industry to which the original text belongs, and obtain an identification result indicating that the industry to which the original text belongs is any one of the multiple industries; then, a set of industry-specific words corresponding to any one of the industries is found in the configuration table, and for each industry-specific word in the set of industry-specific words, if the corresponding word is found to appear in the original text, the corresponding word is added to the word segmentation result.

[0063] S5. Extract the sensitive word set from the configuration table, and for each sensitive word in the sensitive word set, if a corresponding word is found to appear in the word segmentation result, extract the sensitive word processing rule from the configuration table to process the corresponding word in the word segmentation result.

[0064] In step S5, for example, if there is a sensitive word "China's No. 1" or "Nationwide No. 1" in the sensitive word set and the sensitive word "China's No. 1" or "Nationwide No. 1" appears in the word segmentation result, the sensitive word "China's No. 1" or "Nationwide No. 1" in the word segmentation result needs to be blurred, modified, replaced or deleted. In addition, considering that different sensitive words may correspond to different sensitive word processing rules, preferably, when there are multiple sensitive word processing rules and they have a one-to-many relationship or a one-to-one relationship with the sensitive words in the sensitive word set, for each sensitive word in the sensitive word set, if the corresponding word is found to appear in the word segmentation result, the sensitive word processing rule is extracted from the configuration table to process the corresponding word in the word segmentation result, including but not limited to: if a certain sensitive word in the sensitive word set is found to appear in the word segmentation result, the sensitive word processing rule corresponding to the certain sensitive word is extracted from the configuration table to process the certain sensitive word in the word segmentation result.

[0065] S6. For each Chinese word in the word segmentation result, generate a corresponding pinyin word, and add the pinyin word to the word segmentation result.

[0066] In step S6, for example, if there are Chinese words "clothing" or "jacket" in the word segmentation result, a pinyin word "yifu" or "waitao" can be generated and added to the word segmentation result.

[0067] S7. Extract the text assembly rule from the configuration table, and apply the text assembly rule to perform text assembly on the words in the word segmentation result to obtain the text to be retrieved.

[0068] S8. Import the text to be retrieved into the distributed search and analysis engine Elasticsearch, and obtain the document recall result returned by the distributed search and analysis engine Elasticsearch.

[0069] In the step S8, Elasticsearch is a distributed, highly scalable and highly real-time search and data analysis engine, which can easily enable a large amount of data to have the ability to search, analyze and explore, and make full use of the horizontal scalability of Elasticsearch to make data more valuable in the production environment; the implementation principle of Elasticsearch is mainly divided into the following steps: first, the user submits the data to the Elasticsearch database, and then the corresponding sentence is segmented through the word segmentation controller, and its weight and word segmentation results are stored in the data together. When the user searches for data, the results are ranked and scored according to the weight, and finally the returned results are presented to the user. In this way, not only can the document recall results corresponding to the original text and the text to be retrieved be obtained through the distributed search and analysis engine Elasticsearch, but also the search function of Elasticsearch can be used to match according to the previously segmented words, which greatly improves accuracy and industry customization. In addition, the method is preferably integrated into the Spring project, an application framework that introduces the distributed publish-subscribe messaging system Kafka and the distributed search and analysis engine Elasticsearch into an open source Java / Java EE full-function stack, wherein the Spring framework is an open source Java platform that aims to simplify the development of enterprise-level applications; it provides a comprehensive programming and configuration model to solve many common problems encountered in J2EE development; Spring's core features include Inversion of Control (IoC) and Aspect Oriented Programming (AOP), as well as a Web MVC framework, which make Spring a powerful application framework, and therefore can be applied to the present embodiment to provide a basic project environment that integrates the distributed publish-subscribe messaging system Kafka and the distributed search and analysis engine Elasticsearch.

[0070] S9. Feedback the document recall result to the client.

[0071] Therefore, based on the word segmentation search method described in the aforementioned steps S1 to S9, a new solution for word segmentation search based on a configuration table and a distributed search and analysis engine Elasticsearch is provided, that is, after receiving a word segmentation search request initiated by a client and carrying an original text for a word segmentation search document, based on a configuration table configured with multiple word segmenters corresponding to multiple text types, multiple industry-specific word sets corresponding to multiple industries, sensitive word sets, sensitive word processing rules and text assembly rules, a word segmenter is selected in turn to perform word segmentation, add industry-specific words, process sensitive words, add pinyin words and assemble the text to be retrieved, and finally the text to be retrieved is imported into the distributed search and analysis engine Elasticsearch, and a document recall result returned by Elasticsearch is obtained, and feedback is given to the client, so that not only can the search function of Elasticsearch be used and matching can be performed according to the previously segmented words, thereby greatly improving the accuracy and industry customization and completing the customization function, but also proper noun segmentation, sensitive word filtering and Chinese character pinyin processing can be supported, thereby ensuring the accuracy of search results, data security and user search experience, and facilitating practical application and promotion.

[0072] like Figure 2 As shown, the second aspect of this embodiment provides a virtual device for implementing the word segmentation search method described in the first aspect, including a configuration table acquisition unit, a search request receiving unit, an initial word segmentation processing unit, an industry-specific word adding unit, a sensitive word processing unit, a pinyin word adding unit, a search text assembly unit, a search engine application unit and a search result feedback unit;

[0073] The configuration table acquisition unit is used to acquire a configuration table pre-configured with a plurality of word segmenters corresponding one-to-one to a plurality of text types, a plurality of industry-specific word sets corresponding one-to-one to a plurality of industries, a sensitive word set, sensitive word processing rules, and text assembly rules;

[0074] The search request receiving unit is used to receive a word segmentation search request initiated by a client, wherein the word segmentation search request carries an original text for a word segmentation search document;

[0075] The initial word segmentation processing unit is respectively connected to the configuration table acquisition unit and the search request receiving unit for communication, and is used to find a first word segmenter corresponding to the first text type in the configuration table if the text type of the original text is specified as a first text type among the multiple text types in the word segmentation search request, and use the first word segmenter to perform word segmentation processing on the original text to obtain a word segmentation result;

[0076] The industry-specific word adding unit is respectively connected to the configuration table acquisition unit and the initial word segmentation processing unit for communication, and is used to find a certain industry-specific word set corresponding to the certain industry in the configuration table if the search target industry field is specified as a certain industry among the multiple industries in the word segmentation search request, and for each industry-specific word in the certain industry-specific word set, if the corresponding word is found to appear in the original text, then add the corresponding word to the word segmentation result;

[0077] The sensitive word processing unit is respectively connected to the configuration table acquisition unit and the industry-specific word adding unit for extracting the sensitive word set from the configuration table, and for each sensitive word in the sensitive word set, if a corresponding word is found to appear in the word segmentation result, extracting the sensitive word processing rule from the configuration table to process the corresponding word in the word segmentation result;

[0078] The pinyin word adding unit is communicatively connected to the sensitive word processing unit, and is used to generate a corresponding pinyin word for each Chinese word in the word segmentation result, and add the pinyin word to the word segmentation result;

[0079] The search text assembly unit is respectively connected to the configuration table acquisition unit and the pinyin word adding unit for extracting the text assembly rule from the configuration table and applying the text assembly rule to perform text assembly on the words in the word segmentation result to obtain the text to be searched;

[0080] The search engine application unit is communicatively connected to the search text assembly unit, and is used to import the to-be-searched text into the distributed search and analysis engine Elasticsearch, and obtain the document recall result returned by the distributed search and analysis engine Elasticsearch;

[0081] The search result feedback unit is communicatively connected to the search engine application unit and is used to feed back the document recall result to the client.

[0082] The working process, working details and technical effects of the aforementioned device provided in the second aspect of this embodiment can be referred to the word segmentation search method described in the first aspect, and will not be repeated here.

[0083] like Figure 3As shown, the third aspect of this embodiment provides a computer device for executing the word segmentation search method as described in the first aspect, including a memory, a processor and a transceiver connected in sequence, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the word segmentation search method as described in the first aspect. Specifically, the memory may include, but is not limited to, a random access memory (Random-Access Memory, RAM), a read-only memory (Read-Only Memory, ROM), a flash memory (FlashMemory), a first-in-first-out memory (First Input First Output, FIFO) and / or a first-in-last-out memory (First Input Last Output, FILO), etc.; the processor may include, but is not limited to, a microprocessor of the STM32F105 series. In addition, the computer device may also include, but is not limited to, a power module, a display screen and other necessary components.

[0084] The working process, working details and technical effects of the aforementioned computer device provided in the third aspect of this embodiment can be referred to the word segmentation search method described in the first aspect, and will not be repeated here.

[0085] In a fourth aspect of this embodiment, there is provided a computer-readable storage medium storing instructions including the word segmentation search method as described in the first aspect, that is, the computer-readable storage medium stores instructions, and when the instructions are executed on a computer, the word segmentation search method as described in the first aspect is executed. The computer-readable storage medium refers to a carrier for storing data, which may include but is not limited to computer-readable storage media such as floppy disks, optical disks, hard disks, flash memory, USB flash drives, and / or memory sticks, and the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0086] The working process, working details and technical effects of the aforementioned computer-readable storage medium provided in the fourth aspect of this embodiment can be referred to the word segmentation search method described in the first aspect, and will not be repeated here.

[0087] In a fifth aspect, the present embodiment provides a computer program product, including a computer program or an instruction, wherein the computer program or the instruction, when executed by a computer, implements the word segmentation search method as described in the first aspect. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0088] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A word segmentation search method based on a configuration table, characterized in that: include: Obtaining a configuration table pre-configured with a plurality of word segmenters corresponding one-to-one to a plurality of text types, a plurality of industry-specific word sets corresponding one-to-one to a plurality of industries, a sensitive word set, sensitive word processing rules, and text assembly rules; Receiving a word segmentation search request initiated by a client, wherein the word segmentation search request carries an original text for a word segmentation search document; If the text type of the original text is specified as the first text type among the multiple text types in the word segmentation search request, a first word segmenter corresponding to the first text type is found in the configuration table, and the first word segmenter is applied to perform word segmentation processing on the original text to obtain a word segmentation result; If the search target industry field is specified as a certain industry among the multiple industries in the word segmentation search request, a certain industry-specific word set corresponding to the certain industry is found in the configuration table, and for each industry-specific word in the certain industry-specific word set, if a corresponding word is found to appear in the original text, the corresponding word is added to the word segmentation result; Extracting the sensitive word set from the configuration table, and for each sensitive word in the sensitive word set, if a corresponding word is found to appear in the word segmentation result, extracting the sensitive word processing rule from the configuration table to process the corresponding word in the word segmentation result; For each Chinese word in the word segmentation result, generate a corresponding pinyin word, and add the pinyin word to the word segmentation result; Extracting the text assembly rule from the configuration table, and applying the text assembly rule to perform text assembly on the words in the word segmentation result to obtain the text to be retrieved; Importing the to-be-searched text into the distributed search and analysis engine Elasticsearch, and obtaining a document recall result returned by the distributed search and analysis engine Elasticsearch; The document recall result is fed back to the client.

2. The word segmentation search method according to claim 1, characterized in that: If the text type of the original text is not specified in the word segmentation search request or the text type of the original text is specified in the word segmentation search request and is not in the multiple text types, the method further includes: Using a natural language processing library to perform text type recognition processing on the original text, to obtain a recognition result indicating that the text type of the original text is a second text type among the multiple text types; Finding a second word segmenter corresponding to the second text type in the configuration table; The second word segmenter is used to perform word segmentation processing on the original text to obtain a word segmentation result.

3. The word segmentation search method according to claim 1, characterized in that: If the search target industry field is not specified in the word segmentation search request or the search target industry field specified in the word segmentation search request is not in the multiple industries, the method further includes: Extracting the plurality of industry-specific word sets from the configuration table; For each industry-specific word in the plurality of industry-specific word sets, if a corresponding word is found to appear in the original text, the corresponding word is added to the word segmentation result.

4. The word segmentation search method according to claim 1, characterized in that: After receiving the word segmentation search request and before performing word segmentation processing on the original text, the method further includes: The original text is put into the distributed publish-subscribe messaging system Kafka, so as to perform word segmentation processing on the original text with high throughput in combination with the distributed publish-subscribe messaging system Kafka.

5. The word segmentation search method according to claim 4, characterized in that: The method is integrated into the Spring project, an application framework that introduces the distributed publish-subscribe messaging system Kafka and the distributed search and analysis engine Elasticsearch into an open source Java / JavaEE full-function stack.

6. The word segmentation search method according to claim 1, characterized in that: When there are multiple sensitive word processing rules and they have a one-to-many relationship or a one-to-one relationship with the sensitive words in the sensitive word set, for each sensitive word in the sensitive word set, if a corresponding word is found to appear in the word segmentation result, the sensitive word processing rule is extracted from the configuration table to process the corresponding word in the word segmentation result, including: If it is found that a certain sensitive word in the sensitive word set appears in the word segmentation result, the sensitive word processing rule corresponding to the certain sensitive word is extracted from the configuration table to process the certain sensitive word in the word segmentation result.

7. A word segmentation search device based on a configuration table, characterized in that: It includes a configuration table acquisition unit, a search request receiving unit, an initial word segmentation processing unit, an industry-specific word adding unit, a sensitive word processing unit, a pinyin word adding unit, a search text assembly unit, a search engine application unit and a search result feedback unit; The configuration table acquisition unit is used to acquire a configuration table pre-configured with a plurality of word segmenters corresponding one-to-one to a plurality of text types, a plurality of industry-specific word sets corresponding one-to-one to a plurality of industries, a sensitive word set, sensitive word processing rules, and text assembly rules; The search request receiving unit is used to receive a word segmentation search request initiated by a client, wherein the word segmentation search request carries an original text for a word segmentation search document; The initial word segmentation processing unit is respectively connected to the configuration table acquisition unit and the search request receiving unit for communication, and is used to find a first word segmenter corresponding to the first text type in the configuration table if the text type of the original text is specified as a first text type among the multiple text types in the word segmentation search request, and use the first word segmenter to perform word segmentation processing on the original text to obtain a word segmentation result; The industry-specific word adding unit is respectively connected to the configuration table acquisition unit and the initial word segmentation processing unit for communication, and is used to find a certain industry-specific word set corresponding to the certain industry in the configuration table if the search target industry field is specified as a certain industry among the multiple industries in the word segmentation search request, and for each industry-specific word in the certain industry-specific word set, if the corresponding word is found to appear in the original text, then add the corresponding word to the word segmentation result; The sensitive word processing unit is respectively connected to the configuration table acquisition unit and the industry-specific word adding unit for extracting the sensitive word set from the configuration table, and for each sensitive word in the sensitive word set, if a corresponding word is found to appear in the word segmentation result, extracting the sensitive word processing rule from the configuration table to process the corresponding word in the word segmentation result; The pinyin word adding unit is communicatively connected to the sensitive word processing unit, and is used to generate a corresponding pinyin word for each Chinese word in the word segmentation result, and add the pinyin word to the word segmentation result; The search text assembly unit is respectively connected to the configuration table acquisition unit and the pinyin word adding unit for extracting the text assembly rule from the configuration table and applying the text assembly rule to perform text assembly on the words in the word segmentation result to obtain the text to be searched; The search engine application unit is communicatively connected to the search text assembly unit, and is used to import the to-be-searched text into the distributed search and analysis engine Elasticsearch, and obtain the document recall result returned by the distributed search and analysis engine Elasticsearch; The search result feedback unit is communicatively connected to the search engine application unit and is used to feed back the document recall result to the client.

8. A computer device, characterized in that: It includes a memory, a processor and a transceiver which are communicatively connected in sequence, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the word segmentation search method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, and when the instructions are executed on the computer, the word segmentation search method as described in any one of claims 1 to 6 is executed.

10. A computer program product comprising a computer program or instructions, characterized in that When the computer program or the instruction is executed by a computer, the word segmentation search method according to any one of claims 1 to 6 is implemented.

Citation Information

Patent Citations

  • Method and apparatus for automatically adding pinyin for Chinese character

    CN101482867A

  • New-generation industry knowledge full-text search method

    CN103823799A