Data processing method, data processing equipment and storage medium
By using preprocessed downgraded word segmenters and multiple word segmentation modes to process input information in search engines, the problem that search engines cannot meet the variable search scenarios is solved and the search accuracy is improved.
Patent Information
- Application Number
- CN202411395696.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-08
- Publication Date
- 2025-02-14
AI Technical Summary
Current search engines cannot meet the changing search scenarios, resulting in low search accuracy.
When a data input instruction is detected, the preprocessed downgraded word segmenter processes the received input information to obtain a word segmentation result set, and synonyms, pinyin, expression and/or Filter processing to obtain the target word segmentation result set.
Ensure that data search can be carried out in different scenarios, improve the data search accuracy when data matching based on the target word segmentation result set, and meet complex and changeable search scenarios.
Smart Images
Figure CN119441260A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, a data processing device and a storage medium. Background Art
[0002] In related search functions, a distributed search and analysis engine is usually built based on Elasticsearch (distributed search and engine), and the IK (IK Analyzer, an open source plug-in) word segmenter is used for indexing and searching. However, the word segmentation mode provided by the IK word segmenter can only be applied to Chinese search scenarios with low precision requirements. It supports fewer search word segmentation modes and cannot meet the changing search scenarios, resulting in low search precision.
[0003] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention
[0004] The main purpose of this application is to provide a data processing method, a data processing device and a storage medium, aiming to solve the technical problem that the current search engine cannot meet the changing search scenarios, resulting in low search accuracy.
[0005] To achieve the above objectives, the present application proposes a data processing method, which includes:
[0006] When a data input instruction is detected, the degraded word segmenter based on the preprocessing processes the received input information to obtain a word segmentation result set;
[0007] When the word segmentation result set is not empty, performing word segmentation correction processing on the word segmentation result set to obtain a target word segmentation result set;
[0008] Data matching processing is performed according to the target word segmentation result set to obtain target data.
[0009] In one embodiment, when the data input instruction is detected, before the step of processing the received input information by the pre-processed degraded word segmenter to obtain the word segmentation result set, the step further includes:
[0010] Build an index configuration based on the ES index, the index configuration information includes at least a segmentation mode, and the segmentation mode includes at least NLP segmentation, index segmentation, perceptron segmentation, CRF segmentation, N-shortest path segmentation, shortest path segmentation, and fast dictionary segmentation;
[0011] Initialize a word segmenter corresponding to the word segmentation pattern, and perform downgraded word segmentation processing on the initialized word segmenter to obtain the pre-processed downgraded word segmenter.
[0012] In one embodiment, the step of performing synonym, pinyin, expression and / or filtering processing on the word segmentation result set to obtain a target word segmentation result set includes:
[0013] Based on the word segmentation result set, obtaining synonym information and pinyin set information of the original word after word segmentation;
[0014] Generate mixed word segmentation information of the original word according to the synonym information associated with the original word and the pinyin set information;
[0015] The mixed word segmentation information is updated to the word segmentation result set to obtain the target word segmentation result set.
[0016] In one embodiment, when the word segmentation result set is not empty, before the step of performing synonym, pinyin, expression and / or filtering processing on the word segmentation result set to obtain the target word segmentation result set, the step further includes:
[0017] Obtain a pre-stored expression file, and load the expression file into the prefix tree;
[0018] Traversing the prefix tree, and calculating the start and end offset information of each expression in the prefix tree;
[0019] The step of performing synonym, pinyin, expression and / or filtering processing on the word segmentation result set to obtain a target word segmentation result set comprises:
[0020] Identify the expressions in the word segmentation result set based on the prefix tree, and obtain part of speech, position and offset information to be corrected of the expressions;
[0021] The word segmentation result set is updated based on the part of speech to be corrected, the position and the offset information to obtain the target word segmentation result set.
[0022] In one embodiment, after the step of performing data matching processing according to the target word segmentation result set to obtain target data, the step further includes:
[0023] After the dictionary singleton class is initialized, obtain the monitoring information of the Nacos dictionary file;
[0024] When the monitoring information includes configuration change information, obtaining the change data based on HTTP long polling, and loading the change data into the core dictionary; or
[0025] Create a file system monitor and create monitoring objects based on the current processing system;
[0026] Listen to the custom dictionary according to the detection object and block the acquisition of the monitoring pool;
[0027] When there is a change file in the custom dictionary, the core dictionary is reloaded based on the reload structure, and the core dictionary is the dictionary used by the current search engine.
[0028] In one embodiment, after the step of performing data matching processing according to the target word segmentation result set to obtain target data, the step further includes:
[0029] Obtain a pre-stored custom ES source, and broadcast the custom ES source to each search instance;
[0030] After listening to the channel event, HighLevelClient is constructed and registered in the Spring IOC container so that the current search engine calls the custom ES source for data search.
[0031] In one embodiment, when the data input instruction is detected, after the step of processing the received input information based on the preprocessed degraded word segmenter to obtain a word segmentation result set, the step further includes:
[0032] When the word segmentation result set is empty, performing downgraded word segmentation processing and operation prompt processing on the input information;
[0033] Obtaining the first result after the downgraded word segmentation processing and the second result corresponding to the operation prompt processing, and updating the first result and the second result into the word segmentation result set;
[0034] If the word segmentation result set is empty, the current query action is stopped, otherwise the step of performing word segmentation correction processing on the word segmentation result set to obtain the target word segmentation result set is executed.
[0035] In one embodiment, when the data input instruction is detected, before the step of processing the received input information by the pre-processed degraded word segmenter to obtain the word segmentation result set, the step further includes:
[0036] When receiving a call instruction from a website's search engine, the call instruction is parsed based on a pre-stored software development kit, and the parsed engine call permission is sent back to the website, so that the website calls the current search engine.
[0037] In addition, to achieve the above-mentioned purpose, the present application also proposes a data processing device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the data processing method described above.
[0038] In addition, to achieve the above objectives, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data processing method described above are implemented.
[0039] One or more technical solutions proposed in this application have at least the following technical effects:
[0040] After the initial word segmentation of the input, when the word segmentation result set is not empty, the word segmentation result set is processed by synonyms, pinyin, expressions and filtering to ensure that data search can be performed in different scenarios, so that the current search engine can meet complex and changeable search scenarios, thereby improving the data search accuracy when matching data based on the target word segmentation result set. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0043] Figure 1 A system architecture diagram of a search engine for a data processing device of the present application;
[0044] Figure 2 A flowchart of the first embodiment of the data processing method of the present application is provided;
[0045] Figure 3 An optional flowchart of a data processing method provided in the first embodiment of the present application;
[0046] Figure 4 A flowchart of the second embodiment of the data processing method of the present application is provided;
[0047] Figure 5 This is a schematic diagram of a data storage structure in the second embodiment of the data processing method of the present application;
[0048] Figure 6 A flowchart of the third embodiment of the data processing method of the present application is provided;
[0049] Figure 7 This is a schematic diagram of a flow chart of performing expression recognition processing in the third embodiment of the data processing method of the present application;
[0050] Figure 8 A flowchart of implementing quasi-real-time hot update of a remote extended vocabulary based on Nacos file monitoring provided in the fourth embodiment of the data processing method of the present application;
[0051] Fig. 9 A flowchart of using the Linux kernel inotify file monitoring mechanism to achieve efficient and real-time word library hot update provided in the fourth embodiment of the data processing method of the present application;
[0052] Fig.10 An exemplary flowchart of a custom ES data source extension provided in the fifth embodiment of the data processing method of the present application;
[0053] Fig.11 Schematic diagram of the device structure of the hardware operating environment involved in the data processing method in the embodiment of the present application.
[0054] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0055] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.
[0056] As the functions and related operation manuals of the systems such as Wealth W+, Market W+, and CMS (Content Management System) in the bank increase, product managers and users often lack in-depth understanding of the massive functions of the system, and often need to explain and preach repeatedly. In addition, the massive functions of the system also increase the difficulty of user use, resulting in extremely high entry barriers and usage costs of the products, which seriously affects the user experience of the system. In order to improve this situation, the above systems all provide in-site search functions, but there are problems in the use process, including but not limited to low search accuracy, poor search experience, single function, poor scalability, business function coupling, poor portability, and no support for custom data sources.
[0057] Based on this, in related search functions, a distributed search and analysis engine is usually built based on Elasticsearch (distributed search and engine), and the IK (IK Analyzer, an open source plug-in) word segmenter is used for indexing and searching. However, the word segmentation mode provided by the IK word segmenter can only be applied to Chinese search scenarios with low precision requirements. It supports fewer search word segmentation modes and cannot meet the changing search scenarios, resulting in low search precision.
[0058] The main solution of the embodiment of the present application is: when a data input instruction is detected, the degraded word segmenter based on the preprocessing processes the received input information to obtain a word segmentation result set;
[0059] When the word segmentation result set is not empty, performing synonym, pinyin, expression and / or filtering processing on the word segmentation result set to obtain a target word segmentation result set;
[0060] Data matching processing is performed according to the target word segmentation result set to obtain target data.
[0061] Specifically, after the initial word segmentation processing of the input information, if the obtained word segmentation result set is not empty, the word segmentation result set is processed by synonyms, pinyin, emoticons and filtering to obtain the target word segmentation result set, so that the current search engine can effectively adapt to various scenarios when performing search processing based on the target word segmentation result set, thereby improving the search accuracy when facing different search scenarios.
[0062] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.
[0063] Please refer to Figure 1 , the embodiment of the present application provides a system architecture diagram of a search engine of a data processing device, the search system includes an application layer, a gateway, a service layer, a parsing layer and a data layer. Among them, the application layer includes a web application Vue APP built using the Vue framework, a tool set containing a set of JavaScript libraries and APIs, a JSSDK used to help developers more easily implement specific functions or integrate third-party services, and a web application ReactAPP built using the React library. The gateway is used for data request forwarding, gateway grayscale processing, and permission management. The service layer provides search services, data synchronization and data source management; the parsing layer provides word segmentation processing, synonym processing, pinyin processing, emoji processing, hot update vocabulary processing and word segmentation degradation processing, thereby improving the scene adaptability of the search engine; the data layer includes the Nacos configuration center, which provides centralized configuration management functions, allowing developers to dynamically manage and update application configuration information at runtime, RabbitMQ, which is used to achieve asynchronous communication and decoupling between applications, an open source search engine and analysis engine for real-time search and analysis of large amounts of data Elasticarch, and MySQL (relational database management system) that helps developers store and manage structured data to meet the data persistence needs of applications.
[0064] Since the IK analyzer has poor accuracy and single function, it cannot meet higher search requirements. In order to improve the user experience and search accuracy of the search component, this application provides a self-developed high-precision, multi-mode, feature-rich ES analyzer that supports real-time word library hot update and word segmentation degradation. It can be understood that the ES analyzer is a component in Elasticsearch that is used to process text data. Its main function is to decompose text data into vocabulary units so that Elasticsearch can correctly index and search these data. As a component in the search engine, the ES analyzer is used to process text data and support full-text search functions.
[0065] Optionally, the ES analyzer of the search engine of the embodiment of the present application introduces seven efficient word segmentation algorithms into the ES word segmentation in terms of word segmentation mode, and then supports seven word segmentation modes including CRF (Conditional Random Field) word segmentation, NLP (Natural Language Processing) word segmentation, N-shortest path word segmentation, index word segmentation, perceptron word segmentation, shortest path word segmentation, and ultra-fast dictionary word segmentation. In terms of search functions, advanced functions such as name recognition, place name recognition, synonym search, Chinese pinyin mixed search, emoji search, and traditional and simplified conversion are provided. In terms of usability, in order to prevent word segmentation failure from affecting user experience, word segmentation supports downgrade to improve search accuracy.
[0066] It should be noted that different word segmentation modes are usually set when the search engine is used, and the operator can adjust the word segmentation mode used by the current search engine through background modification.
[0067] The present application embodiment provides a data processing method, which is applied to the search engine of the above-mentioned data processing device, referring to Figure 2 , Figure 2 This is a flowchart of the first embodiment of the data processing method of the present application.
[0068] In this embodiment, the data processing method includes steps S10 to S30:
[0069] Step S10, when a data input instruction is detected, the degraded word segmenter based on the preprocessing processes the received input information to obtain a word segmentation result set;
[0070] It should be noted that the executor of this application is a data processing device, which is an intelligent device capable of performing computing and processing, including tablets, mobile phones, and computers, etc., and is usually used in scenarios such as web pages, programs, or systems that can provide query functions.
[0071] In this embodiment, the data input instruction refers to the instruction triggered by the operator inputting the search term and clicking the search button in the search function interface provided by the search engine on the web page, program or system. After the search engine of the data processing device detects the instruction, the input information is subjected to initial word segmentation processing by the pre-processed downgraded word segmenter, thereby obtaining a word segmentation result set. Among them, the reasonable word segmentation processing of the input information by the pre-processed downgraded word segmenter can improve the searchability of the document and the relevance of the search results, thereby improving the flexibility and accuracy of the search.
[0072] Specifically, the process of processing the received input information by the degraded word segmenter based on preprocessing mainly includes processing steps such as word segmentation, word embedding and special tag addition. Based on these processing operations, a preliminary word segmentation result can be obtained, so as to perform secondary word segmentation processing based on the preliminary word segmentation result, and finally obtain a word segmentation result set suitable for data matching. Among them, when the degraded word segmenter is used to perform word segmentation processing on the input, a simple word segmentation annotation can be performed first. The annotation process does not include degraded word segmentation recognition processing such as name recognition and place name recognition. Then, when there is no word segmentation result or word segmentation cannot be performed in the word segmentation result set, the degraded word segmentation processing function of the degraded word segmenter is started.
[0073] Optionally, when processing is performed based on a degraded word segmenter, it is also necessary to obtain the degraded word segmenter after preprocessing, so before step S10, steps S01 to S02 are also included:
[0074] Step S01, building index configuration based on ES index;
[0075] In this embodiment, the ES index creation Configuration can be read. In Elasticsearch, the index creation process not only involves defining the name of the index, but also includes setting its internal structure, number of shards, replica settings, mapping rules and other key parameters, which can be achieved by sending HTTP requests. In Elasticsearch, the constructed index configuration usually includes the word segmentation mode used by the current search engine, wherein the word segmentation mode includes at least NLP word segmentation, index word segmentation, perceptron word segmentation, CRF word segmentation, N-shortest path word segmentation, shortest path word segmentation and fast dictionary word segmentation. Based on this, the current word segmentation mode can be selected through the constructed index, and then data retrieval processing can be performed based on the word segmentation mode. By providing a variety of different word segmentation modes, the requirements for speed and accuracy in various search scenarios can be fully met, thereby improving search accuracy.
[0076] Step S02, initializing a word segmenter corresponding to the word segmentation pattern, and performing downgraded word segmentation processing on the initialized word segmenter to obtain the pre-processed downgraded word segmenter.
[0077] In this embodiment, after building an index and determining the word segmentation pattern corresponding to the index, it is necessary to initialize the current word segmenter in order to perform downgrading processing on the word segmenter.
[0078] Exemplarily, in the downgraded word segmentation process, downgraded word segmentation judgment can be performed, including whether to start name recognition, place name recognition, traditional and simplified conversion, and recognition of professional vocabulary in specific fields, etc., so that the downgraded word segmenter can accurately recognize information such as names and place names in the input information and improve the effectiveness of word segmentation.
[0079] In another alternative embodiment, when a call instruction from the search engine of the website is received, the call instruction is parsed based on a pre-stored software development kit (SDK), and the parsed engine call permission is passed back to the website so that the website can call the current search engine. Based on this, through the JSSDK, only a few lines of extremely simple code are required to access the complete in-site search function for the system, which is completely decoupled from the business logic, making the search engine highly portable and easy to use.
[0080] Step S20, when the word segmentation result set is not empty, perform synonym, pinyin, emoji, and / or filtering processing on the word segmentation result set to obtain a target word segmentation result set;
[0081] In this embodiment, when the word segmentation result set is not empty, it indicates that the input information can be accurately recognized and segmented. However, after word segmentation processing based on the preprocessed downgraded word segmenter, its word segmentation results usually cannot meet the changing search requirements. For example, when the search content contains pinyin or emoji symbols, it is difficult to output results that match the current content. Therefore, further processing of a subset of the word segmentation result set is also required.
[0082] Specifically, when further processing the word segmentation result set, synonym processing, pinyin processing, emoji processing, and stop word filtering processing or stem extraction processing can be performed in sequence. The specific processing order can be to complete the processing first and then perform data filtering or stem extraction. Among them, the stop word filtering process includes filtering stop words such as "de", "shi", "le", etc. and punctuation marks, and stem extraction includes removing the prefix or suffix of a certain word to obtain its root. For example, when the input has the English word "unfriendly", it needs to be extracted as "friendly". Among them, emoji processing can be emoji symbol processing or other recognizable emoji symbol processing.
[0083] Optionally, in addition to processing the above actions in sequence, one of the processing actions can also be selected based on the actual situation. For example, when only pinyin processing is required, only the action of pinyin processing can be performed.
[0084] Optionally, synonym processing, pinyin processing and expression processing may be performed first to obtain a result set, and then the result set may be filtered for stop words or stem extraction to obtain a target word segmentation result set.
[0085] When the word segmentation result set is not empty, by performing synonym processing, pinyin processing, emoji processing and filtering processing on the word segmentation result set, the search engine can meet complex and changeable search needs and improve the accuracy of data search when searching based on the processed target word segmentation results.
[0086] In another optional implementation, when the word segmentation result set is empty, it means that effective word segmentation processing cannot be performed based on the current input information. For example, when downgrading the input information, the downgraded word segmenter does not enable traditional-simplified conversion, or when the downgraded word segmenter processes the input information, it only performs simple word segmentation annotation processing and does not perform downgraded word segmentation processing, resulting in the inability to segment the input traditional information.
[0087] Therefore, when the word segmentation result set is empty, the input information is subjected to downgraded word segmentation processing and operation prompt processing. Operation prompt processing refers to sending the current word segmentation result to the back-end administrator, who responds to the operation and performs manual word segmentation or word segmentation confirmation processing. Subsequently, the first result after the downgraded word segmentation processing and the second result corresponding to the operation prompt processing are obtained, and the first result and the second result are updated to the word segmentation result set; if the word segmentation result set is empty, it means that the current input information is incorrect and the current query action needs to be stopped. Otherwise, step S20 can be executed to perform word segmentation correction processing on the word segmentation result set to obtain the processing action of the target word segmentation result set.
[0088] Step S30, performing data matching processing according to the target word segmentation result set to obtain target data.
[0089] In this embodiment, after obtaining the target word segmentation result set, it is necessary to record the word segmentation results, part of speech, offset, and storage location information, and finally perform data matching processing based on this information to obtain the target data.
[0090] Optionally, after obtaining the target data, the searched content may be displayed in a feedback interface.
[0091] For example, to help understand the implementation process of the data processing method combined with this embodiment, please refer to Figure 3 , Figure 3The invention provides an optional implementation process of creating a word segmenter and a word segmentation processing process of a data processing method. In the word segmenter creation process, the ES index is first read and the configuration is constructed to determine the corresponding word segmentation mode, and then the word segmenter is initialized. The process includes selecting a suitable word segmentation algorithm or word segmenter to perform necessary training on the model, such as selecting an N-Short word segmenter for processing, and then initializing an N-Short downgraded word segmenter, that is, downgrading the N-Short word segmenter, including judging whether to enable name recognition, place name recognition, and traditional-simplified conversion, etc. In the affirmative answer, the corresponding name recognition parameters, place name recognition parameters, and traditional-simplified conversion parameters are added, otherwise jump to the next judgment or next processing process. Based on this, the word segmenter creation is completed.
[0092] When performing word segmentation based on the created word segmenter, the input information can be obtained in batches and basic word segmentation processing actions can be performed. When the word segmentation is successful and the word segmentation result set is not empty, the obtained word segmentation result set is processed with synonyms, pinyin, and emoji, and then the processed word segmentation result set is returned. Then, the stop words / stem extraction operation is performed, and the final word segmentation result, part of speech, offset, and position information are recorded to obtain the target word segmentation result set. At this time, data retrieval actions can be performed, or input information can be continued to be obtained and word segmentation processing can be performed. In addition, in the initial word segmentation stage, if the word segmentation fails, the downgraded word segmentation processing can be performed, and then it is confirmed whether there is a result after the downgraded word segmentation processing in the word segmentation result set. If not, the current processing flow can be terminated. Or when the word segmentation is successful and the word segmentation result set is empty, the current processing flow is terminated. By performing synonym processing, pinyin processing, emoji processing, and filtering processing on the word segmentation result set, the search needs and scenario needs when performing data retrieval based on the target word segmentation result set are improved, thereby improving the accuracy of data search.
[0093] This embodiment provides a data processing method. When a data input instruction is detected, the input information is first processed by a downgraded word segmenter, and then the word segmentation result set is processed by synonyms, pinyin, expressions and / or filtering to obtain a target word segmentation result set, so that when the search engine searches based on the processed target word segmentation results, it can meet complex and changeable search needs and improve the accuracy of data search.
[0094] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the first embodiment can refer to the above introduction, and will not be repeated in the following. Figure 4 , step S20 also includes steps S21-S22:
[0095] Step S21, based on the word segmentation result set, obtaining synonym information and pinyin set information of the original word after word segmentation;
[0096] Step S22, generating mixed word segmentation information of the original word according to the synonym information associated with the original word and the pinyin set information;
[0097] Step S23, updating the mixed word segmentation information to the word segmentation result set to obtain a target word segmentation result set.
[0098] In the present embodiment, as an optional implementation method, the word of the word segmentation result set can be processed by synonyms and pinyin, thereby obtaining the target word segmentation result set. In the process, for each word segmentation in the original word segmentation result set, it is necessary to parse out the contents such as synonyms and pinyin sets, and synonyms or pinyin need to share the information such as the position, offset, and part of speech of the original word, in order to ensure the correctness of the word segmentation process, when storing the new word segmentation result in the result set, it is necessary to store in a specific order, and then process in sequence. Wherein, the pinyin set information also includes the information of full spelling, initials and Chinese pinyin mixing, for example, the original word is "school", and its corresponding pinyin set information includes but is not limited to "xuexiao", "learn x", "xuex", "x school", "xxiao" and "xx" etc.
[0099] For example, the synonyms, Chinese Pinyin mixed words and original words are stored as follows: Figure 5 As shown, after obtaining the synonym information and the pinyin set information, the synonym information and the pinyin set information can be combined to generate mixed word segmentation information, and then the mixed word segmentation information is updated to the word segmentation result set to obtain the target word segmentation result set. Based on the obtained target word segmentation result set, when performing data search processing, synonyms and Chinese pinyin mixed words can be effectively retrieved. For example, when the search term is "tomato", the relevant content of "tomato" can also be obtained. When the search term is "tomato", the search results related to "tomato" can also be accurately obtained.
[0100] Based on this, unlike the Pinyin word segmenter which can only split Chinese into single words and requires cumbersome indexing and search configuration, this application not only provides seven efficient word segmentation modes, but also natively provides powerful pinyin search, supporting full spelling, initial letters, and Chinese pinyin mixed search. Improve the applicability of scenarios and thus improve the accuracy of data search.
[0101] This embodiment provides a data processing method, which performs synonym processing and pinyin processing on a word segmentation result set, extracts synonym information and pinyin set information corresponding to the original words, and stores this information in an original storage location. Based on this storage method, when a search engine performs a data search, it can effectively identify the data expressed by the synonyms and the data corresponding to the Chinese pinyin mixed information, thereby improving the data search precision and accuracy.
[0102] Based on the first embodiment of the present application, in the third embodiment of the present application, the same or similar content as the above-mentioned first embodiment can be referred to the above introduction, and will not be repeated later. On this basis, when the expression of the search content is recognized and processed, it is usually necessary to load the expression file into the prefix tree, and then identify the expression information through the prefix tree. For example, if the expression to be recognized is an emoji expression, all emoji expressions need to be loaded into the prefix tree, starting from the root node of the prefix tree, and each node path stores the various Unicode (international code) characters of the emoji in turn, and the leaf nodes of the prefix tree store the emoji data including Unicode encoding.
[0103] Therefore, before step S20, it is necessary to obtain a pre-stored expression file and load the expression file into a prefix tree, then traverse the prefix tree and calculate the start and end offset information of each expression in the prefix tree.
[0104] After getting the prefix tree, please refer to Figure 6 , step S20 also includes steps S24 to S25:
[0105] Step S24, identifying the expression in the word segmentation result set based on the prefix tree, and obtaining the part of speech, position and offset information to be corrected of the expression;
[0106] Step S25, updating the word segmentation result set based on the part of speech to be corrected, the position and the offset information to obtain a target word segmentation result set.
[0107] It should be noted that the expressions to be recognized are emoji expressions. When processing the input information, if no special expression marking is done, the input emoji expressions will not be recognized, and emoji expressions will be treated as punctuation marks and will not be segmented, resulting in emoji expressions being filtered out as stop words. Therefore, by loading emoji expressions into the prefix tree, it can be ensured that the search engine can recognize and process the expressions after segmentation.
[0108] In this embodiment, after obtaining the prefix tree, the expressions in the word segmentation result set are identified based on the prefix tree, thereby obtaining the parameters such as the modified part of speech, position and offset information of the expression, and then the original word segmentation is corrected by using this information, that is, based on the modified part of speech, position and offset information, the word segmentation result set is updated to obtain the target word segmentation result set. The expression file is processed by the prefix tree, and the efficient data retrieval structure based on the prefix tree can quickly locate the expression information to be processed. At the same time, when the expression file is expanded, the update can be completed by adding a new leaf node, without the need for large-scale update of the original expression file, thereby improving data processing efficiency.
[0109] For example, please refer to Figure 7 , the expression is an emoji expression, and the expression file is an emoji expression file. When processing, the emoji file is first loaded into the prefix tree, and then the prefix tree is traversed to parse the emoji, and then the start and end offsets of the emoji are calculated, and then the emoji information of the word segmentation result set is identified, and the word segmentation part of speech, position and offset are corrected for the information to obtain the target word segmentation result set.
[0110] By adding the emoticon file to the prefix tree for processing, the search efficiency of the emoticon is improved based on the prefix tree structure, and the search engine can perform search processing based on the emoticon, thereby improving the data search precision and accuracy.
[0111] Based on the first embodiment of the present application, in the fourth embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above description, and will not be described in detail later. On this basis, the search engine's vocabulary can also be updated. Specifically, after the search results are obtained in step S30, the vocabulary can be updated. Optionally, the vocabulary update processing operation can also be triggered before or after any processing step.
[0112] It should be noted that ES word segmenters such as IK and Ansj often start threads in the background, regularly traverse files in a custom directory, and determine whether the file has been modified by comparing information such as the file type, path, and last updated timestamp. If modified, the word library is reloaded. This method is very cumbersome, inefficient, and not real-time, and there is a delay in dictionary updates. Based on this, this embodiment can implement efficient and real-time word library hot update functions based on Nacos and JDK WatchService.
[0113] Specifically, the quasi-real-time hot update process of the remote extended vocabulary library based on Nacos file monitoring is as follows: Figure 8 As shown, in the analyzer plug-in, the dictionary singleton class can be initialized first. After the dictionary singleton class is initialized, a monitoring of the remote specific Nacos dictionary file is established to obtain the monitoring information of the Nacos dictionary file. Then, when the monitoring information contains configuration change information, the change data is obtained based on HTTP long polling, and then the reload method is called to reload the core dictionary, that is, the change data is loaded into the core dictionary, so as to achieve the purpose of efficient and quasi-real-time hot update of the dictionary. In this process, the Nacos configuration center is used to synchronize the new Nacos configuration file, and at the same time, the MD5 value of the dictionary data is checked and processed. If the MD5 value changes, the change data is obtained by long polling. The management end is used to edit the dictionary file and then update the dictionary data.
[0114] Optionally, when performing local vocabulary expansion and update based on JDK WatchService, you can first create a file system monitor and create a monitoring object based on the current processing system. Then, based on the monitoring object, listen to the custom dictionary and block the acquisition of the monitoring pool. Subsequently, when there is a change file in the custom dictionary, reload the core dictionary based on the reloading structure. The core dictionary is the dictionary used by the current search engine.
[0115] For example, the Linux kernel inotify file monitoring mechanism can be used to implement efficient and real-time word library hot update function, and the implementation process is as follows: Fig. 9 As shown, you can first call the FileSystem.newWatchServce() method to create a file system monitor, and then determine whether the current system is a linus system. If so, create the object corresponding to the system, that is, create a LinusWatchService object, otherwise create a PollingWatchService object, and then listen to the custom dictionary directory through the created object, and block the acquisition of WatchKey monitoring pool information at the same time. When there is a file change, call the Reload API to reload the core dictionary and reset WatchKey. After the recharge is successful, continue to block the acquisition of the monitoring pool, or block the acquisition of the monitoring pool when there is no change in the file. If the reset fails, the current processing flow ends.
[0116] In this embodiment, based on the solution of JDK WatchService and Nacos monitoring, the vocabulary is updated in real time without restarting the ES server, which greatly improves the user search experience and service availability.
[0117] Based on the first embodiment of the present application, in the fifth embodiment of the present application, the same or similar content as the above-mentioned first embodiment can be referred to the above introduction, and will not be repeated later. On this basis, a custom ES data source extension function can also be added to improve data security. Specifically, after step S30 obtains the search results, for systems that are more sensitive to data security, in addition to using the professional version of the ES cluster provided by the system, a user-defined ES data source can also be used. Optionally, a custom ES data source extension can also be added before and after any processing step.
[0118] Specifically, in this process, it is necessary to first obtain the pre-stored custom ES source and broadcast the custom ES source to each search instance. Then, after listening to the channel event, build a HighLevelClient and register the HighLevelClient in the Spring IOC container so that the current search engine calls the custom ES source for data search.
[0119] For example, please refer to Fig.10 , when a custom ES source is added and submitted, the system will broadcast related events to RabbitMQ (open source message broker and queue server) and transmit the events to the optical waveguide. In each embodiment, when the search service of each embodiment listens to the event message of a specific Channel, it will create a corresponding ESHighLevelClient (high-level client of the Elasticsearch cluster) and register it in its own Spring IOC (Inversion of Control) container. At the same time, it will associate the specified appidKey (application identifier) with the Bean (object instance in the Java and Spring framework). Finally, it will find the associated client according to the appidKey and perform search processing to make the search effective in real time.
[0120] Based on this, an event-driven architecture is used to ensure loose coupling between services and efficient event processing. Spring IOC container is used to manage object lifecycles and dependencies to improve the modularity and maintainability of the system. By mapping appidKey with ESHighLevelClient Bean, precise control of different indexes and contexts is achieved. By immediately responding to event messages and performing search index updates, the real-time nature of the search service is achieved. By supporting custom ES data source extensions, the access party can use its own ES data source for data search processing, which improves data security.
[0121] The present application provides a data processing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data processing method in the above-mentioned first embodiment.
[0122] Reference below Fig.11 , which shows a schematic diagram of the structure of a data processing device suitable for implementing an embodiment of the present application. The data processing device in the embodiment of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.11The data processing device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0123] like Fig.11 As shown, the data processing device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the data processing device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the data processing device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a data processing device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.
[0124] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.
[0125] The data processing device provided by the present application adopts the data processing method in the above embodiment, which can solve the technical problem that the current search engine cannot meet the changing search scenarios, resulting in low search accuracy. Compared with the prior art, the beneficial effects of the data processing device provided by the present application are the same as the beneficial effects of the data processing method provided by the above embodiment, and other technical features in the data processing device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.
[0126] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.
[0127] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
[0128] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the data processing method in the above-mentioned embodiment.
[0129] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.
[0130] The computer-readable storage medium may be included in a data processing device, or may exist independently without being incorporated into a data processing device.
[0131] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by a data processing device, the data processing device:
[0132] When a data input instruction is detected, the degraded word segmenter based on the preprocessing processes the received input information to obtain a word segmentation result set;
[0133] When the word segmentation result set is not empty, performing word segmentation correction processing on the word segmentation result set to obtain a target word segmentation result set;
[0134] Data matching processing is performed according to the target word segmentation result set to obtain target data.
[0135] Computer program code for performing the operations of the present application may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0136] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0137] The modules involved in the embodiments described in the present application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.
[0138] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned data processing method, and can solve the technical problem that the current search engine cannot meet the changing search scenarios, resulting in low search accuracy. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the data processing method provided by the above-mentioned embodiment, and will not be repeated here.
[0139] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.
Claims
1. A data processing method, characterized in that: The method comprises: When a data input instruction is detected, the degraded word segmenter based on the preprocessing processes the received input information to obtain a word segmentation result set; When the word segmentation result set is not empty, performing synonym, pinyin, expression and / or filtering processing on the word segmentation result set to obtain a target word segmentation result set; Data matching processing is performed according to the target word segmentation result set to obtain target data.
2. The method according to claim 1, characterized in that Before the step of processing the received input information based on the pre-processed degraded word segmenter to obtain a word segmentation result set when the data input instruction is detected, the method further includes: Build an index configuration based on the ES index, the index configuration information includes at least a segmentation mode, and the segmentation mode includes at least NLP segmentation, index segmentation, perceptron segmentation, CRF segmentation, N-shortest path segmentation, shortest path segmentation, and fast dictionary segmentation; Initialize a word segmenter corresponding to the word segmentation pattern, and perform downgraded word segmentation processing on the initialized word segmenter to obtain the pre-processed downgraded word segmenter.
3. The method according to claim 1, characterized in that The step of performing synonym, pinyin, expression and / or filtering processing on the word segmentation result set to obtain a target word segmentation result set comprises: Based on the word segmentation result set, obtaining synonym information and pinyin set information of the original word after word segmentation; Generate mixed word segmentation information of the original word according to the synonym information associated with the original word and the pinyin set information; The mixed word segmentation information is updated to the word segmentation result set to obtain the target word segmentation result set.
4. The method according to claim 1, characterized in that Before the step of performing synonym, pinyin, expression and / or filtering processing on the word segmentation result set to obtain the target word segmentation result set when the word segmentation result set is not empty, the step further includes: Obtain a pre-stored expression file, and load the expression file into the prefix tree; Traversing the prefix tree, and calculating the start and end offset information of each expression in the prefix tree; The step of performing synonym, pinyin, expression and / or filtering processing on the word segmentation result set to obtain a target word segmentation result set comprises: Identify the expressions in the word segmentation result set based on the prefix tree, and obtain part of speech, position and offset information to be corrected of the expressions; The word segmentation result set is updated based on the part of speech to be corrected, the position and the offset information to obtain the target word segmentation result set.
5. The method according to claim 1, characterized in that After the step of performing data matching processing according to the target word segmentation result set to obtain target data, the method further includes: After the dictionary singleton class is initialized, obtain the monitoring information of the Nacos dictionary file; When the monitoring information includes configuration change information, obtaining the change data based on HTTP long polling, and loading the change data into the core dictionary; or Create a file system monitor and create monitoring objects based on the current processing system; Listen to the custom dictionary according to the detection object and block the acquisition of the monitoring pool; When there is a change file in the custom dictionary, the core dictionary is reloaded based on the reload structure, and the core dictionary is the dictionary used by the current search engine.
6. The method according to claim 1, characterized in that After the step of performing data matching processing according to the target word segmentation result set to obtain target data, the method further includes: Obtain a pre-stored custom ES source, and broadcast the custom ES source to each search instance; After listening to the channel event, HighLevelClient is constructed and registered in the Spring IOC container so that the current search engine calls the custom ES source for data search.
7. The method according to claim 1, characterized in that After the step of processing the received input information based on the pre-processed degraded word segmenter to obtain a word segmentation result set when the data input instruction is detected, the method further includes: When the word segmentation result set is empty, performing downgraded word segmentation processing and operation prompt processing on the input information; Obtaining the first result after the downgraded word segmentation processing and the second result corresponding to the operation prompt processing, and updating the first result and the second result into the word segmentation result set; If the word segmentation result set is empty, the current query action is stopped, otherwise the step of performing word segmentation correction processing on the word segmentation result set to obtain the target word segmentation result set is executed.
8. The method according to any one of claims 1 to 7, characterized in that Before the step of processing the received input information based on the pre-processed degraded word segmenter to obtain a word segmentation result set when the data input instruction is detected, the method further includes: When receiving a call instruction from a website's search engine, the call instruction is parsed based on a pre-stored software development kit, and the parsed engine call permission is sent back to the website, so that the website calls the current search engine.
9. A data processing device, characterized in that: The data processing device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data processing method according to any one of claims 1 to 8.
10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 8 are implemented.