Retrieval optimization method and device in retrieval system, equipment, medium and product

Abstract

The invention discloses a retrieval optimization method and device in a retrieval system, equipment, a medium and a product, and relates to the technical field of information retrieval, and the method comprises the following steps: obtaining an original document, and extracting a target keyword from the original document; matching the target keyword with each current partition to determine a first partition matched with the target keyword, storing the original document to the first partition, and calculating a document score corresponding to the original document; wherein the document score is dynamically updated according to a preset rule; when a retrieval request of a user side is obtained, determining a target partition parameter and a target scoring parameter based on the retrieval request; and positioning the second partition based on the target partition parameter, and screening out matched document content from the second partition based on the target scoring parameter so as to return the document content to the user side. According to the method, the retrieval process of the retrieval system can be optimized, so that the retrieval performance and efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of information retrieval, and particularly relates to a retrieval optimization method, device, equipment, medium and product in a retrieval system. Background Art

[0002] The performance bottlenecks of existing distributed retrieval systems are mainly reflected in two aspects: (1) Low index construction efficiency: High time and space complexity, using a fixed word segmentation mode to construct indexes, which is time-consuming when processing a large amount of data, and the index structure is only suitable for conventional retrieval; (2) Low retrieval process efficiency: Low data transmission and calculation efficiency, and the optimization of traditional algorithms based on the scan matching mechanism has great limitations, and it is impossible to significantly improve the retrieval performance without departing from the original scan mechanism. The reason is that existing retrieval and index optimization methods are all based on a single index structure, which is difficult to meet diverse retrieval needs and cannot efficiently handle complex requirements in large-scale distributed retrieval scenarios.

[0003] In summary, how to optimize the retrieval process of a retrieval system to improve retrieval performance and efficiency is a problem to be solved at present. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a retrieval optimization method, device, equipment, medium and product, which can optimize the retrieval process of a retrieval system to improve retrieval performance and efficiency. The specific solutions are as follows: In the first aspect, the present application discloses a retrieval optimization method in a retrieval system, including: Obtain the original document, and extract the target keyword from the original document; Match the target keyword with each current partition to determine the first partition that matches the target keyword, store the original document in the first partition, and calculate the document score corresponding to the original document; wherein, the document score is dynamically updated according to a preset rule; When a retrieval request from the user side is obtained, determine the target partition parameter and the target score parameter based on the retrieval request; Locate the second partition based on the target partition parameter, and screen out the matching document content from the second partition based on the target score parameter, so as to return the document content to the user side.

[0005] Optionally, extracting the target keyword from the original document includes: Use a preset word segmenter to extract the corresponding keyword from the original document as the target keyword; wherein, the preset word segmenter is constructed based on a first word segmenter and a second word segmenter; wherein, the first word segmenter is constructed based on a general word segmentation library, and the second word segmenter is constructed based on a custom word segmentation library set by the user side.

[0006] Optionally, extracting the target keyword from the original document includes: Slice the original document to obtain multiple sliced segments; Calculate the semantic density of each sliced segment, and filter out the target sliced segments with semantic density greater than the preset density threshold from each sliced segment; Use a preset tokenizer to extract the corresponding keywords from the target sliced segments and use them as target keywords.

[0007] Optionally, calculating the semantic density of each sliced segment includes: For any sliced segment, calculate the term frequency-inverse document frequency of each word in the sliced segment, and sum up the term frequency-inverse document frequencies to obtain a summation result; Determine the semantic density of any sliced segment based on the summation result and the length of the sliced segment of any sliced segment.

[0008] Optionally, matching the target keyword with the current partitions to determine the first partition that matches the target keyword includes: Obtain the keyword list corresponding to each current partition, and determine whether the target keyword is included in each keyword list; If there is a target keyword list that includes the target keyword, then use the partition corresponding to the target keyword list as the first partition that matches the target keyword.

[0009] Optionally, matching the target keyword with the current partitions to determine the first partition that matches the target keyword includes: Obtain the regular expression corresponding to each current partition; wherein, the regular expression is constructed based on the text feature and the target character; Calculate the similarity between the target keyword and each regular expression respectively; If the similarity between any regular expression and the target keyword is greater than the preset similarity threshold, then use the partition corresponding to any regular expression as the first partition that matches the target keyword.

[0010] Optionally, in the process of matching the target keyword with the current partitions to determine the first partition that matches the target keyword, it includes: If there is no partition in the current partitions that matches the target keyword, then create a new partition based on the target keyword and use the new partition as the first partition that matches the target keyword.

[0011] Optionally, calculating the document score corresponding to the original document includes: The original document is scored based on a preset multi-dimensional scoring model to obtain the corresponding document score; among them, the multi-dimensional scoring model is constructed based on the original score value, the document click volume, the time decay factor, and the corresponding weight coefficients. The original score value is the score obtained by using the internal scoring mechanism of the distributed retrieval system, the document click volume is the cumulative click volume of the original document, and the time decay factor is calculated based on a preset time decay formula.

[0012] Optionally, the document score is dynamically updated based on preset rules, including: Obtain the latest cumulative click volume and the latest time decay factor; Recalculate the score of the original document based on the original score value, the latest cumulative click volume, the latest time decay factor, and the corresponding weight coefficients to update the document score.

[0013] Optionally, obtaining the latest cumulative click volume and the latest time decay factor includes: Obtain the current latest cumulative click volume and the latest time decay factor at preset time intervals; Or, when the current load of the distributed retrieval system is lower than the preset load threshold and the current time is within the preset time range, obtain the current latest cumulative click volume and the latest time decay factor.

[0014] Optionally, the target partition parameter includes a partition retrieval parameter and a target search parameter indicating whether to search for an associated partition; Correspondingly, locating the second partition based on the target partition parameter includes: If the target search parameter indicates not to search for an associated partition, determine the corresponding original retrieval partition based on the partition retrieval parameter, and use the original retrieval partition as the second partition; If the target search parameter indicates to search for an associated partition, locate the corresponding original retrieval partition based on the partition retrieval parameter, and determine the associated partition associated with the original retrieval partition, so as to use the original retrieval partition and the associated partition as the second partition.

[0015] Optionally, the retrieval optimization method in the retrieval system of the present application further includes: If the number of target keywords is multiple, perform semantic analysis on each target keyword to judge whether there is a semantic similarity relationship between each target keyword based on the semantic analysis result; If there are multiple target keywords that satisfy the semantic similarity relationship, establish an association relationship between the first partitions corresponding to the multiple target keywords.

[0016] Optionally, determining the target partition parameter and the target score parameter based on the retrieval request includes: Judge whether the retrieval request carries a partition parameter and a score parameter; If so, the partition parameter and the scoring parameter carried in the retrieval request are respectively used as the target partition parameter and the target scoring parameter.

[0017] Optionally, after determining whether the retrieval request carries a partition parameter and a scoring parameter, it further includes: If the retrieval request does not carry a partition parameter and a scoring parameter, perform word segmentation processing and semantic parsing on the retrieval content in the retrieval request to obtain partition retrieval parameters; Query whether there are target search parameters and scoring parameters pre-configured by the client in the preset configuration file; If they exist, obtain the target partition parameter based on the target search parameter pre-configured by the client and the partition retrieval parameter determined by using the retrieval content, and use the scoring parameter pre-configured by the client as the target scoring parameter; If they do not exist, obtain the target partition parameter based on the default search parameter stored in the preset configuration file and the partition retrieval parameter determined by using the retrieval content, and use the default scoring parameter stored in the preset configuration file as the target scoring parameter.

[0018] Optionally, the target scoring parameter includes a scoring threshold; Correspondingly, screening out matching document content from the second partition based on the target scoring parameter includes: Screen out target documents in the second partition whose document scores are higher than the scoring threshold, and determine the document content from the target documents.

[0019] Optionally, when storing the original document into the first partition, it further includes: Extract the global information of the original document to construct a main index; Segment and store the original document according to the document size, and construct a sub-index for each segment; wherein, each sub-index is an associated index of the main index.

[0020] Optionally, the target partition parameter further includes a document retrieval mode. Determining the document content from the target document includes: If the partition retrieval parameter represents full-document retrieval, use all the content in the target document as the document content; If the partition retrieval parameter does not represent full-document retrieval, match the partition retrieval parameter with the main index and each sub-index respectively; If the partition retrieval parameter matches the main index successfully, use the global information corresponding to the main index as the document content; If the partition retrieval parameter matches any sub-index successfully, use the segment corresponding to any sub-index as the document content.

[0021] In a second aspect, the present application discloses a retrieval optimization device in a retrieval system, including: A keyword extraction module, configured to obtain an original document and extract target keywords from the original document; A document storage module, configured to match the target keywords with current partitions to determine a first partition that matches the target keywords, store the original document in the first partition, and calculate a document score corresponding to the original document; wherein, the document score is dynamically updated according to a preset rule; A parameter determination module, configured to determine target partition parameters and target score parameters based on a retrieval request when the retrieval request from the client is obtained; A document retrieval module, configured to locate a second partition based on the target partition parameters and screen out matching document content from the second partition based on the target score parameters, so as to return the document content to the client.

[0022] In a third aspect, the present application discloses an electronic device, including: A memory, configured to store a computer program; A processor, configured to execute the computer program to implement the steps of the retrieval optimization method in the aforementioned disclosed retrieval system.

[0023] In a fourth aspect, the present application discloses a computer-readable storage medium, configured to store a computer program; wherein, when the computer program is executed by a processor, the steps of the retrieval optimization method in the aforementioned disclosed retrieval system are implemented.

[0024] In a fifth aspect, the present application discloses a computer program product, including computer program / instructions, and when the computer program / instructions are executed by a processor, the steps of the retrieval optimization method in the aforementioned disclosed retrieval system are implemented.

[0025] It can be seen that the present application obtains an original document, extracts target keywords from the original document; matches the target keywords with current partitions to determine a first partition that matches the target keywords, stores the original document in the first partition, and calculates a document score corresponding to the original document; wherein, the document score is dynamically updated according to a preset rule; when the retrieval request from the client is obtained, determines target partition parameters and target score parameters based on the retrieval request; locates a second partition based on the target partition parameters and screens out matching document content from the second partition based on the target score parameters, so as to return the document content to the client.

[0026] Beneficial effects: After obtaining the original document, the present application first needs to extract target keywords from the original document. Moreover, the present application has pre-constructed multiple partitions. After the target keywords are extracted, the target keywords are matched with the current partitions to determine the first partition that matches the target keywords, and the original document is stored in the first partition. That is, the present application stores the original document by matching keywords with partitions, achieving the centralized storage of documents classified by semantics, that is, it can be realized that the documents stored in the same partition have a strong correlation in content semantics. In addition, when storing the original document, the document score corresponding to the original document will be calculated, and over time, the document score will be dynamically updated according to the preset rules. Further, when a retrieval request from the user side is obtained, it is first necessary to determine the target partition parameter and the target score parameter based on the retrieval request. In this way, it is possible to quickly locate the corresponding second partition by means of this directional retrieval method based on the target partition parameter and the target score parameter, and extract the matching document content from the second partition for return, thereby improving the retrieval performance. Through the above solutions, the retrieval system can ensure the retrieval accuracy while having better response performance and stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on the provided drawings.

[0028] Figure 1 It is a flowchart of a retrieval optimization method in a retrieval system disclosed in the present application; Figure 2 It is a flowchart of a retrieval optimization method in a specific retrieval system disclosed in the present application; Figure 3 It is an architecture diagram of a retrieval optimization in a retrieval system disclosed in the present application; Figure 4 It is a schematic structural diagram of a retrieval optimization device in a retrieval system disclosed in the present application; Figure 5 It is a structural diagram of an electronic device disclosed in the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0029] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0030] The performance bottlenecks of existing distributed retrieval systems are mainly reflected in two aspects: (1) Low index construction efficiency: High time and space complexity, using a fixed word segmentation mode to construct indexes, which is time-consuming when processing a large amount of data, and the index structure is only applicable to conventional retrieval; (2) Low retrieval process efficiency: Low data transmission and calculation efficiency, and the optimization of traditional algorithms based on the scan matching mechanism has great limitations and cannot significantly improve the retrieval performance without departing from the original scan mechanism. The reason is that existing retrieval and index optimization methods are all based on a single index structure, which is difficult to meet diverse retrieval needs and cannot efficiently handle complex requirements in large-scale distributed retrieval scenarios. Therefore, the embodiments of the present application disclose a retrieval optimization method, device, equipment, medium and product in a retrieval system, which can optimize the retrieval process of the retrieval system to improve the retrieval performance and efficiency.

[0031] See Figure 1 As shown, the embodiments of the present application disclose a retrieval optimization method in a retrieval system, and the method includes: Step S11: Obtain the original document and extract target keywords from the original document.

[0032] The retrieval system in this embodiment may specifically be a distributed retrieval system, and when the original document is obtained, an index needs to be created for the incoming original document. Among them, it is first necessary to extract target keywords from the original document.

[0033] In a specific implementation manner, extracting target keywords from the original document includes: using a preset word segmenter to extract corresponding keywords from the original document and use them as target keywords; among them, the preset word segmenter is constructed based on a first word segmenter and a second word segmenter; among them, the first word segmenter is constructed based on a general word segmentation library, and the second word segmenter is constructed based on a custom word segmentation library set by the user side. It can be understood that the present application can directly extract keywords from the original document using the preset word segmenter. The preset word segmenter processing includes not only the first word segmenter constructed based on the general word segmentation library, but also the second word segmenter constructed based on the custom word segmentation library set by the user side. That is, the word segmenter in the present application supports flexible user-defined configuration, so as to be applicable to different application scenarios.

[0034] Among them, the general word segmentation library is a standardized word library designed for basic language processing, aiming to cover the splitting requirements of daily language and general terms, such as word segmentation by ordinary items, species, solid and liquid states, etc., or word segmentation by grammatical structures (such as subject-predicate-object). The custom word segmentation library is to configure and input specific phrases existing in a specific field / industry. Taking some special industries as examples, there are unique words in professional fields such as chemistry, physics, and law, such as "aromatic" chemical substances. This type of segmentation has strong professional characteristics and requires separate configuration of its own unique word segmenter in the business systems of relevant industries.

[0035] In another specific implementation, extracting target keywords from the original document includes: performing sharding processing on the original document to obtain multiple shard fragments; calculating the semantic density of each shard fragment, and screening out target shard fragments with a semantic density greater than a preset density threshold from each shard fragment; using a preset word segmenter to extract corresponding keywords from the target shard fragments and taking them as target keywords. That is to say, this application can first preprocess the original document to filter out some low-value fragments, and then extract keywords from the remaining fragments.

[0036] Specifically, in this embodiment, the original document is sharded to obtain multiple shard fragments, and then the semantic density (Semantic Density, i.e., SD) of each shard fragment is calculated. The semantic density is mainly used to measure the degree of text information concentration of the entire shard fragment, that is, to quantify the information value of each shard fragment. Then, target shard fragments with a semantic density greater than the preset density threshold are screened out from each shard fragment, that is, some fragments with higher information value are screened out, and some low-value shard fragments are filtered out. For example, the preset density threshold can be set to 0.85, then it can be understood that the shard fragments with SD < 0.15 are discarded. Finally, the preset word segmenter is used to extract keywords from these selected target shard fragments to obtain the target keywords. In this way, there is no need to extract keywords for the entire document, which can reduce the overhead generated by inefficient index construction.

[0037] Among them, calculating the semantic density of each fragment includes: for any fragment, calculating the term frequency-inverse document frequency of each word in the fragment, and summing up the term frequency-inverse document frequencies to obtain a summation result; determining the semantic density of the fragment based on the summation result and the length of the fragment. That is to say, the semantic density is mainly calculated based on the term frequency-inverse document frequency (TF-IDF) of each word in a single fragment and the length of the fragment. It can be understood that TF-IDF is a statistical method for evaluating the importance of words in a document, widely used in the fields of text analysis, information retrieval, and natural language processing. Its core idea is to measure the distinctiveness of words by combining the term frequency (TF) and the inverse document frequency (IDF). Among them, the term frequency refers to the number of times a word appears in a document. For example, if the word "apple" appears twice in a document while other words only appear once, the term frequency of "apple" is higher, which may contribute more to the content of the document; the inverse document frequency reflects the universality of a word in a corpus. For example, common words such as "of" and "is" appear frequently in most documents, and their IDF values are relatively low; while professional terms such as "quantum entanglement" only appear in documents in specific fields, and their IDF values are relatively high.

[0038] Therefore, in this application, by calculating the term frequency-inverse document frequency of each word in a single fragment, then summing up all the term frequency-inverse document frequencies of the fragment to obtain a summation result, and then based on the summation result and the length of the fragment, the semantic density can be calculated. The specific calculation formula is as follows: SD = ∑(TF-IDF) / fragment length.

[0039] Step S12: Match the target keyword with the current partitions to determine the first partition that matches the target keyword, store the original document in the first partition, and calculate the document score corresponding to the original document; where the document score is dynamically updated according to a preset rule.

[0040] In this embodiment, multiple partitions have been pre-constructed. After obtaining the target keyword, the target keyword is matched with each current partition to determine the first partition that matches the target keyword, and the original document is stored in the first partition. That is, in this application, the original document is stored by matching the keyword with the partition, realizing the centralized storage of documents classified by semantics, that is, it can be ensured that the documents stored in the same partition have a strong correlation in content semantics. In addition, when storing the original document, the document score corresponding to the original document is calculated, and over time, the document score is dynamically updated according to a preset rule. In this way, after the complete process of keyword extraction, partition matching, document storage, and score calculation described above, the index construction process of the original document is completed.

[0041] In a specific implementation manner, matching the target keyword with each current partition to determine the first partition that matches the target keyword includes: obtaining the keyword list corresponding to each current partition respectively, and determining whether the target keyword is included in each keyword list; if there is a target keyword list that includes the target keyword, the partition corresponding to the target keyword list is used as the first partition that matches the target keyword. That is, each current partition maintains its own corresponding keyword list. When matching the target keyword with each current partition, it is to determine whether the target keyword is included in each keyword list. If there is a target keyword list that includes the target keyword, it means that there is a partition that matches the target keyword. Therefore, the partition corresponding to the target keyword list is used as the first partition that matches the target keyword.

[0042] In another specific implementation manner, matching the target keyword with each current partition to determine the first partition that matches the target keyword includes: obtaining the regular expression corresponding to each current partition respectively; wherein, the regular expression is constructed based on text features and target characters; calculating the similarity between the target keyword and each regular expression respectively; if there is any regular expression whose similarity with the target keyword is greater than a preset similarity threshold, the partition corresponding to any regular expression is used as the first partition that matches the target keyword. It can be understood that considering that the models of some products in the same series may be different, or there are different expressions of some synonyms, in this embodiment, a corresponding regular expression is set for each partition. The regular expression can flexibly match keywords that conform to a specific pattern, especially suitable for dealing with complex, variable, or partially known terms; wherein, the regular expression is mainly constructed based on text features and target characters. In this embodiment, by calculating the similarity between the target keyword and each regular expression, if there is any regular expression whose similarity with the target keyword is greater than a preset similarity threshold, the partition corresponding to any regular expression is used as the first partition that matches the target keyword.

[0043] It should be noted that high / middle / low score matching partitions can also be divided according to the size value of the preset similarity threshold, specifically corresponding to a similarity higher than 80%, a similarity between 50% and 80%, and a similarity between 30% and 50%.

[0044] It should also be noted that in the process of matching the target keyword with the current partitions to determine the first partition that matches the target keyword, it also includes: if there is no partition in the current partitions that matches the target keyword, a new partition is created based on the target keyword, and the new partition is used as the first partition that matches the target keyword. That is to say, if the keyword list of the current partitions does not contain the target keyword, or the similarity between the target keyword and the regular expression is lower than 30%, or even lower than 10%, it means that the current partitions do not match the target keyword. In this case, a new partition can be created for the target keyword, and the new partition is used as the first partition that matches the target keyword.

[0045] Step S13: When a retrieval request from the client is obtained, determine the target partition parameter and the target scoring parameter based on the retrieval request.

[0046] In this embodiment, when a retrieval request from the client is obtained, it is first necessary to determine the target partition parameter and the target scoring parameter based on the retrieval request.

[0047] Step S14: Locate the second partition based on the target partition parameter, and screen out the matching document content from the second partition based on the target scoring parameter, so as to return the document content to the client.

[0048] In this embodiment, after determining the target partition parameter and the target scoring parameter based on the retrieval request, the corresponding second partition can be quickly located by means of directional retrieval based on the target partition parameter and the target scoring parameter, and the matching document content is extracted from the second partition for return, thereby improving the retrieval performance.

[0049] It can be seen that after obtaining the original document, the present application first needs to extract target keywords from the original document. Moreover, the present application has pre-constructed multiple partitions. After the target keywords are extracted, the target keywords are matched with the current partitions to determine the first partition that matches the target keywords, and the original document is stored in the first partition. That is, the present application stores the original document by matching keywords with partitions, realizing the centralized storage of documents classified by semantics, that is, it can be ensured that the documents stored in the same partition have a strong correlation in content semantics. In addition, when storing the original document, the document score corresponding to the original document is calculated, and as time goes by, the document score is dynamically updated according to a preset rule. Further, when a retrieval request from the user side is obtained, it is first necessary to determine the target partition parameter and the target score parameter based on the retrieval request. In this way, the corresponding second partition can be quickly located by means of such directional retrieval based on the target partition parameter and the target score parameter, and the matching document content is extracted from the second partition and returned, thereby improving the retrieval performance. Through the above solution, the retrieval system can ensure better response performance and stability while guaranteeing the retrieval accuracy.

[0050] See Figure 2 As shown, an embodiment of the present application discloses a retrieval optimization method in a specific retrieval system. Compared with the previous embodiment, this embodiment further explains and optimizes the technical solution. Specifically, it includes: Step S21: Obtain the original document and extract the target keywords from the original document.

[0051] Step S22: Match the target keywords with the current partitions to determine the first partition that matches the target keywords, store the original document in the first partition, and score the original document based on a preset multi-dimensional scoring model to obtain the corresponding document score; wherein, the document score is dynamically updated according to a preset rule.

[0052] In this embodiment, after storing the original document in the first partition, it is also necessary to score the document, that is, when the document is first stored in the partition, the corresponding document score needs to be calculated, and then the score will be refreshed opportunistically and irregularly according to the internal preset rules, which is convenient for determining the matching degree of the result according to the score parameter during subsequent retrieval. Documents with a lower score should not be retrieved. Specifically, the present application scores the original document based on a preset multi-dimensional scoring model to obtain the corresponding document score, and records the document score in a pre-configured scoring area.

[0053] Among them, the multi-dimensional scoring model is constructed based on the original score value, the document click volume, the time decay factor, and the corresponding weight coefficients. The original score value is the score value obtained by scoring using the internal scoring mechanism of the distributed retrieval system. The document click volume is the cumulative click volume of the original document. The time decay factor is calculated based on a preset time decay formula. That is, in this embodiment, on the basis of the original internal scoring mechanism of the distributed retrieval system, the document click volume and the time decay factor are further combined and weighted to generate a new document score. The specific expression of the multi-dimensional scoring model is: Score_final = α × ES_score + β × log(1 + click_count) + γ × time_decay; Among them, ES_score is the original score value obtained by scoring using the internal scoring mechanism of the distributed retrieval system; click_count is the cumulative click volume of the original document, which can represent its access popularity; time_decay is the time decay factor, that is, when a term is retrieved, it carries time decay information. For example, a hot word of a social event has a high popularity when the event occurs. As the popularity of the event fades, the popularity of the term will also decay. When the terms attached to the term with a high popularity are retrieved, the term will also be retrieved with a high score with a high probability; α + β + γ = 1. In the specific implementation, α = 0.6, β = 0.3, γ = 0.1, or α = 0.3, β = 0.5, γ = 0.2; among them, the specific weight ratio can be set according to the specific scenario, and this embodiment does not limit this.

[0054] In addition, the specific expression of the preset time decay formula is: ; Among them, t is the document storage time, which can be 0.05.

[0055] In the specific implementation, the document score is dynamically updated based on a preset rule, including: obtaining the latest cumulative click volume and the latest time decay factor; recalculating the score of the original document based on the original score value, the latest cumulative click volume, the latest time decay factor, and the corresponding weight coefficients to update the document score. That is, in this embodiment, in order to refresh the document score, it is necessary to obtain the current latest cumulative click volume and the latest time decay factor, and then recalculate the score of the original document based on the original score value, the latest cumulative click volume, the latest time decay factor, and the corresponding weight coefficients to update the document score.

[0056] Among them, obtaining the latest cumulative click volume and the latest time decay factor includes: obtaining the current latest cumulative click volume and the latest time decay factor at preset time intervals; or, when the current load of the distributed retrieval system is lower than the preset load threshold and the current time is within the preset time range, obtaining the current latest cumulative click volume and the latest time decay factor.

[0057] It can be understood that in a specific implementation, a method of refreshing the document score regularly can be adopted. For example, the current latest cumulative click volume and the latest time decay factor are obtained every 12 hours or every day, and then the document score is recalculated. In another specific implementation, in order to ensure that the system currently has sufficient computing power to complete the refresh of the document score and avoid resource contention during peak hours, resulting in the problem of score refresh failure, this embodiment can perform the score refresh operation only when the current load of the distributed retrieval system is lower than the preset load threshold and the current time is within the preset time range. For example, the score refresh can be performed when the CPU load < 30% and the current time is between UTC 02:00 - 04:00. On the one hand, determining the current load can ensure that the system has sufficient idle resources to handle computationally intensive operations, preventing refresh interruption or timeout due to insufficient resources; on the other hand, considering that real-time retrieval services need to be prioritized during the peak business hours of the day, triggering score refresh at this time may cause CPU / memory competition.

[0058] Step S23: When the retrieval request from the user side is obtained, determine the target partition parameter and the target score parameter based on the retrieval request; the target partition parameter includes the partition retrieval parameter and the target search parameter indicating whether to search for the associated partition; the target score parameter includes the score threshold.

[0059] In this embodiment, when the retrieval request from the user side is obtained, it is necessary to determine the target partition parameter and the target score parameter based on the retrieval request. Among them, the target partition parameter may specifically include, but is not limited to, the partition retrieval parameter and the target search parameter indicating whether to search for the associated partition, while the target score parameter includes, but is not limited to, the score threshold.

[0060] Among them, determining the target partition parameter and the target scoring parameter based on the retrieval request includes: determining whether the retrieval request carries a partition parameter and a scoring parameter; if so, using the partition parameter and the scoring parameter carried in the retrieval request as the target partition parameter and the target scoring parameter respectively. That is to say, the user can choose whether to carry the partition parameter and the scoring parameter in the retrieval request. If the retrieval request carries the partition parameter and the scoring parameter, the partition parameter and the scoring parameter carried in the retrieval request can be directly used as the target partition parameter and the target scoring parameter respectively. For example, when the user conducts a retrieval, the partition retrieval parameter is set to the aromatic partition information and there is no need to search for associated partitions, that is, only the documents within the aromatic partition are retrieved. In addition, the scoring threshold is set to 9.3 (that is, the document score needs to be greater than 9.3). Then, based on the above conditions, the retrieval range can be restricted during the retrieval, thereby improving the query speed and achieving targeted retrieval on the user side. That is to say, this application supports customizing the partition parameter and the scoring threshold during the application-side retrieval, so as to retrieve the desired results more accurately and quickly.

[0061] Further, after determining whether the retrieval request carries a partition parameter and a scoring parameter, it also includes: if the retrieval request does not carry a partition parameter and a scoring parameter, performing word segmentation processing and semantic analysis on the retrieval content in the retrieval request to obtain a partition retrieval parameter; querying whether there are target search parameters and scoring parameters pre-configured by the user side in the preset configuration file; if so, obtaining the target partition parameter based on the target search parameter pre-configured by the user side and the partition retrieval parameter determined by using the retrieval content, and using the scoring parameter pre-configured by the user side as the target scoring parameter; if not, obtaining the target partition parameter based on the default search parameter stored in the preset configuration file and the partition retrieval parameter determined by using the retrieval content, and using the default scoring parameter stored in the preset configuration file as the target scoring parameter.

[0062] That is to say, if the retrieval request does not carry a partition parameter and a scoring parameter, it is necessary to further perform word segmentation processing and semantic analysis on the retrieval content in the retrieval request to obtain a partition retrieval parameter, and then query whether there are target search parameters and scoring parameters pre-configured by the user side in the preset configuration file, that is, whether the user side has pre-configured the target search parameter and the scoring parameter. If so, obtaining the target partition parameter based on the target search parameter pre-configured by the user side and the partition retrieval parameter determined by using the retrieval content, and using the scoring parameter pre-configured by the user side as the target scoring parameter. For example, through word segmentation processing and semantic analysis of the retrieval content, the partition retrieval parameter obtained is the alcohol partition information, and the target search parameter pre-configured by the user side indicates that it is necessary to search for associated partitions. Assuming that the associated partition of the alcohol partition is the aromatic partition information, and the scoring threshold is 9.8, then the retrieval range can also be restricted during the retrieval based on the above conditions, thereby improving the query speed.

[0063] In addition, if the user does not carry relevant parameters in the retrieval request and there is no relevant configuration in the preset configuration file, then after performing word segmentation and semantic parsing on the retrieval content in the retrieval request to obtain partition retrieval parameters, the default search parameters and default scoring parameters stored in the preset configuration file are used for query. Among them, the default search parameter is generally set to not search for associated partitions, and the default scoring parameter can be set to 8.0.

[0064] It can also be seen from the above content that when performing directional retrieval, the priority of using the parameters carried in the retrieval request > the priority of using the parameters configured by the user at the client in the preset configuration file > the priority of using the default parameters in the preset configuration file. That is, if the client actively issues directional retrieval parameters, such as actively setting partition parameters and scoring parameters during query, then the query is preferentially performed according to the partition parameters and scoring parameters set by the user; if the user does not actively bring partition parameters and scoring parameters during the query, it is necessary to check whether there are parameters manually configured by the user in the preset configuration file, such as setting the aromatic class restriction to only query the aromatic class partition, the default score being 9.8, etc., and use them if available; if the user does not actively bring partition parameters and scoring parameters during the query and there are no parameters manually configured by the user in the preset configuration file, then the built-in default search parameters and default scoring parameters need to be used for query.

[0065] Step S24: If the target search parameter indicates not to search for associated partitions, then determine the corresponding original retrieval partition based on the partition retrieval parameters, and use the original retrieval partition as the second partition; if the target search parameter indicates to search for associated partitions, then locate the corresponding original retrieval partition based on the partition retrieval parameters, and determine the associated partitions that have an association relationship with the original retrieval partition, so as to use the original retrieval partition and the associated partitions as the second partition.

[0066] In a specific implementation manner, if the target search parameter indicates not to search for associated partitions, then it is only necessary to directly use the original retrieval partition determined based on the partition retrieval parameters as the second partition. For example, assuming the partition retrieval parameter is the alcohol partition, then only search in the alcohol partition; assuming the partition retrieval parameter is the aromatic partition, then only search in the aromatic partition.

[0067] In another specific implementation manner, if the target search parameter indicates to search for associated partitions, then first it is necessary to locate the corresponding original retrieval partition based on the partition retrieval parameters, and then determine the associated partitions that have an association relationship with the original retrieval partition, so as to use the original retrieval partition and the associated partitions together as the second partition. For example, assuming the partition retrieval parameter is the alcohol partition, since there is an association relationship between the alcohol partition and the aromatic partition, it is necessary to search in both the alcohol partition and the aromatic partition.

[0068] Further, the above method further includes: if the number of target keywords is multiple, semantic analysis is performed on each target keyword to determine whether there is a semantic similarity relationship between the target keywords based on the semantic analysis result; if there are multiple target keywords that satisfy the semantic similarity relationship, an association relationship is established between the first partitions corresponding to the multiple target keywords. It can be understood that if the number of extracted target keywords is multiple, semantic analysis needs to be performed on each target keyword to determine whether there is a semantic similarity relationship between the target keywords. For example, according to a retrieval content "This is a patent for invention", the keywords in it are extracted to obtain three keywords: invention, patent, and patent for invention. After corresponding semantic analysis and other operations, the basic partitions are determined. For example, invention, patent, and patent for invention are three basic partitions, and according to the semantic analysis result, it can be determined that there is a semantic similarity relationship between patent and patent for invention. Therefore, if there are multiple target keywords that satisfy the semantic similarity relationship, an association relationship is established between the first partitions corresponding to the multiple target keywords. For example, an association relationship is established between the partitions corresponding to patent and patent for invention to facilitate associated query during subsequent query.

[0069] For another example, aromatic chemical substances include polycyclic aromatic hydrocarbons, alcohols, etc. It is necessary to establish an association relationship between aromatic and polycyclic aromatic hydrocarbons and between aromatic and alcohols respectively, so that when retrieving aromatic substances, the contents of the two partitions of polycyclic aromatic hydrocarbons and alcohols can be retrieved simultaneously, and vice versa. In addition, for another example, the human partition can be associated with behavioral characteristics, such as taking, grasping, working, etc. for association establishment, forming a multi-dimensional classification model based on entity characteristics (ResNet-50 extracts visual features, and BERT (Bidirectional Encoder Representations from Transformers, a pre-trained language model) extracts text features) storage structure.

[0070] Step S25: Screen out target documents in the second partition whose document scores are higher than the score threshold, and determine the document content from the target documents, so as to return the document content to the client.

[0071] In this embodiment, after the second partition is located based on the target partition parameter, it is necessary to screen out target documents in the second partition whose document scores are higher than the score threshold according to the target score parameter. From the foregoing content, it can be seen that a corresponding document score is calculated when each document is stored in the partition, and the document score will be dynamically refreshed. Therefore, the present application can accurately screen out target documents that meet the user's requirements from the partition according to the score threshold. Further, the document content is determined from the target documents, so as to return the document content to the client.

[0072] In the specific implementation, the location of the underlying data storage can be located to the node according to the partition →, then to the data shard →, and finally to the data segment.

[0073] It should also be noted that the present application discloses an archiving model for processing the document structure during underlying storage. Specifically, during the process of storing the original document in the first partition, it further includes: extracting the global information of the original document to construct the main index; segmenting and storing the original document according to the document size, and constructing a sub-index for each segment; where each sub-index is an associated index of the main index. It can be understood that the present application can first construct the main index of the original document, and the main index can specifically be the title, author, abstract, etc. of the original document; then segment and store the original document, and construct a sub-index for each segment, and the sub-index is specifically used to represent the semantic content of each segment. In the specific implementation, it can be specifically set to store the original document according to the segmented semantics, such as splitting by a default segment length of 50 characters, and splitting and storing if it exceeds 50 characters, so as to reduce the problem of excessive requests occupied during retrieval due to an overly long single document. It should be noted that the length of the segmentation and splitting parameters can be independently set and adjusted in combination with the system business characteristics to achieve a better retrieval effect.

[0074] Specifically, the target partition parameter further includes a document retrieval mode. Determining the document content from the target document includes: if the partition retrieval parameter represents a full-document retrieval, then taking all the content in the target document as the document content; if the partition retrieval parameter does not represent a full-document retrieval, then matching the partition retrieval parameter with the main index and each sub-index respectively; if the partition retrieval parameter matches the main index successfully, then taking the global information corresponding to the main index as the document content; if the partition retrieval parameter matches any sub-index successfully, then taking the segment corresponding to any sub-index as the document content.

[0075] It is understandable that the user can choose whether to perform full-document retrieval. When the partition retrieval parameter represents full-document retrieval, all the content in the target document is returned to the user terminal as the document content. If the partition retrieval parameter does not represent full-document retrieval, the partition retrieval parameter is respectively matched with the main index and each sub-index. If the partition retrieval parameter matches the main index successfully, the global information corresponding to the main index is returned to the user terminal as the document content. If the partition retrieval parameter matches any sub-index successfully, the segment corresponding to any sub-index is returned to the user terminal as the document content. That is to say, the original document is segmented and stored according to the corresponding archiving mode and the document size, and points to the same document index. When the partition retrieval parameter points to the sub-index in the split document, only the relevant document segment is returned to the application terminal. Only when the full-document retrieval mode is retrieved, the full document is returned, thus avoiding the problem that the return of a large document occupies more request space, resulting in slow or even failed retrieval, and improving the performance and efficiency of the system.

[0076] That is to say, the archiving model is to process the document structure during underlying storage. For example, when an article is saved as a single document, due to its long length, the performance during saving and subsequent use is very poor. Therefore, the document will be cut after semantic analysis during saving, such as saving it in sub-indexes paragraph by paragraph. The split main index only saves the global information of the document (such as title, author, abstract) and the position relationship information of the sub-indexes. In this way, when querying, the main index will be queried first. Only when the entire content of the article needs to be queried, all the sub-index information that needs to be spliced into the complete article needs to be returned.

[0077] It is understandable that the implementation of document storage specification optimization can be achieved through corresponding rules and methods. For example, it can be classified according to the semantic segmentation characteristics of the document (such as type, semantics, word meaning, etc.), combined with custom rule library matching. According to the segmentation result, the document is split and stored and sub-indexes are constructed. The sub-indexes are associated with the main index. Finally, during retrieval, the local or all content of the document can be returned according to the split structure relationship of the document, avoiding the phenomenon of low query efficiency or query crash caused by returning all the document content in a single retrieval, and improving the system stability and efficiency. For example: A. Rule library matching: The document can be matched according to the default word segmentation semantic library. The custom rules can be defined and imported through the rule control module. B. The semantic segmentation rule can identify the logical paragraphs of the document through the NLP (Natural Language Processing) model, such as the BERT+CRF (conditional random fields) recognition technology clause boundary. C. The associated index can use the Master Index: storing global document information (such as title, author, abstract) and the segment mapping table, and the Chunk Index: ① The keyword inverted index: recording the keywords and their position weights within the segment. ② And the vector index: storing the chunk semantic vectors (generated by Sentence - BERT); D. Strategies when the index is retrieved: The on - demand retrieval mechanism, such as ① The lightweight retrieval mode: After the user inputs keywords, only the matching segment content and related context abstracts are returned, and the results are sorted according to the segment relevance, supporting paged loading (≤5 segments are transferred each time). ② The full - document mode: Triggered after authorization configuration, the complete document is spliced in the order of segment identifiers, and the streaming transmission technology (such as HTTP Chunked Encoding) is used to avoid memory overflow.

[0078] Among them, for a more specific processing process of the above - mentioned step S21, reference can be made to the corresponding content disclosed in the foregoing embodiments, and details will not be repeated here.

[0079] It can be seen that this application speeds up and optimizes the index and retrieval processes of the distributed retrieval system, enabling the index to perform intelligent matching and partitioned storage according to established rules, and using the directional retrieval method to extract the partition index results, thereby improving the retrieval performance efficiency and optimizing the resource utilization rate. By combining the multi - dimensional scoring model, dynamically refreshing the document score by combining the original score value, document click - through rate, and time decay factor, improving the accuracy of the result relevance ranking, and optimizing the retrieval efficiency and accuracy. The archiving model mechanism effectively avoids the memory overflow risk during large - document processing through intelligent document segmentation and segmented storage, combined with the streaming transmission technology, ensuring the stability and efficiency of the retrieval performance under large documents. The directional retrieval realizes scenario - based precise control while ensuring the basic performance by supporting the priority of parameters carried in the retrieval request > the priority of user - end configuration parameters > the priority of default parameters, optimizing the retrieval efficiency in cooperation with the overall index ability, and improving the overall performance of the retrieval system. The overall process can enable the retrieval system to have better response performance and stability while ensuring the retrieval accuracy in complex multi - dimensional scenarios, and optimize the system resource utilization rate.

[0080] See Figure 3 As shown, the embodiment of this application discloses a retrieval optimization architecture diagram in a retrieval system, which specifically includes four parts: a rule control module, an index aggregation controller, an archiving model, and a positioning retrieval.

[0081] Among them, the rule control module includes parameters and rule management for the overall aggregate index accelerator, such as the aggregate index accelerator switch, aggregation rule configuration, archiving model rules, scoring refresh rules, scoring thresholds, and search targeting words. For example, the enable switch of the aggregate index accelerator is used to set whether the current system enables the aggregate index accelerator. The aggregation rule configuration is used to control the rules for data classification and corresponding partitioning when the index is stored, so that the data can be indexed according to the rules and patterns required by the user. The archiving model rules are used to set the storage method of the underlying data. The scoring refresh rules and scoring thresholds are the scoring algorithm rules and threshold settings for the document data. The search targeting words are the user-side active input of targeting words during the search process to directly narrow the search scope and improve the search accuracy. These rules are mainly used for parameter and rule settings for the index establishment, underlying index shard storage, data retrieval and other process links in the overall effectiveness process of the aggregate index accelerator. It should be noted that in this embodiment, the aggregate index accelerator switch partition_switch=on / off can be manually defined. The default state is off and it will only take effect when manually set when index acceleration is required. That is, the search optimization process steps in the retrieval system disclosed in this application are only executed when the aggregate index accelerator switch is on.

[0082] The index aggregation controller is mainly used for index aggregation control and index data partitioning. After obtaining the aggregation controller switch and aggregation rules from the rule control module, it creates and refreshes the overall aggregate index, and performs corresponding partitioning on the stored documents. Specifically, the index aggregation controller mainly associates partition index features during the overall index aggregation, index partitioning, and index scoring processes during index processing. That is, by associating data links between partitions, for example, human partitions can be associated with behavioral features such as holding, grabbing, and working. This allows for faster retrieval through associated searches. Document storage achieves partition control through segment pointing processing and records the initial document scores.

[0083] The archiving model is designed to process the document structure during the underlying storage process. Its purpose is to avoid directly returning large documents and taking up too much of the response request load, thereby improving system performance and efficiency.

[0084] Targeted retrieval means that when the application sends a retrieval request, the retrieval system can perform document relevance and word meaning classification analysis based on the document word meaning of the retrieval content, generate specific target partition parameters and target scoring parameters, and then search the index content of the partition based on the specific word meaning content and score. The application is allowed to customize the partition parameters and scoring thresholds during the retrieval, so as to retrieve the desired results more accurately and quickly.

[0085] See also Figure 4As shown in the figure, an embodiment of the present application discloses a retrieval optimization device in a retrieval system. The device includes: A keyword extraction module 11, configured to obtain an original document and extract target keywords from the original document; A document storage module 12, configured to match the target keywords with current partitions to determine a first partition that matches the target keywords, store the original document in the first partition, and calculate a document score corresponding to the original document; wherein, the document score is dynamically updated according to a preset rule; A parameter determination module 13, configured to, when receiving a retrieval request from a client, determine target partition parameters and target score parameters based on the retrieval request; A document retrieval module 14, configured to locate a second partition based on the target partition parameters and screen out matching document content from the second partition based on the target score parameters, so as to return the document content to the client.

[0086] It can be seen that after obtaining the original document, the present application first needs to extract target keywords from the original document. And the present application has pre-constructed multiple partitions. After the target keywords are extracted, the target keywords are matched with current partitions to determine a first partition that matches the target keywords, and the original document is stored in the first partition. That is, the present application stores the original document by matching keywords with partitions, realizing the centralized storage of documents classified by semantics, that is, it can be ensured that the documents stored in the same partition have a strong correlation in content semantics. In addition, when storing the original document, a document score corresponding to the original document will be calculated, and as time goes by, the document score will be dynamically updated according to a preset rule. Further, when receiving a retrieval request from a client, it is first necessary to determine target partition parameters and target score parameters based on the retrieval request. In this way, the corresponding second partition can be quickly located by means of directional retrieval based on the target partition parameters and target score parameters, and matching document content is extracted from the second partition and returned, thereby improving the retrieval performance. Through the above solution, the retrieval system can ensure retrieval accuracy while having better response performance and stability.

[0087] Since the embodiments of the device part correspond to the above embodiments, the embodiments of the device part are described with reference to the embodiments of the above method part and will not be elaborated here.

[0088] Figure 5A schematic structural diagram of an electronic device provided by an embodiment of the present application. Specifically, it may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. Among them, the memory 22 is used to store a computer program, and the computer program is loaded and executed by the processor 21 to implement the relevant steps in the retrieval optimization method in the retrieval system executed by the electronic device disclosed in any of the foregoing embodiments.

[0089] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows is any communication protocol applicable to the technical solution of the present application, and no specific limitation is imposed on it here; the input / output interface 25 is used to obtain external input data or output data to the outside, and the specific interface type can be selected according to specific application needs, and no specific limitation is made here.

[0090] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor used to process data in the wake state, also known as the CPU (Central Processing Unit); the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may be integrated with a GPU (Graphics Processing Unit), and the GPU is responsible for the rendering and drawing of the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, and the AI processor is used to process computational operations related to machine learning.

[0091] In addition, as a carrier for resource storage, the memory 22 may be a read-only memory, a random access memory, a disk, or an optical disc, etc. The resources stored thereon include an operating system 221, a computer program 222, and data 223, etc., and the storage method may be temporary storage or permanent storage.

[0092] Among them, the operating system 221 is used to manage and control each hardware device on the electronic device 20 and the computer program 222, so as to implement the operation and processing of the massive data 223 in the memory 22 by the processor 21. It can be Windows, Unix, Linux, etc. In addition to the computer program that can be used to complete the retrieval optimization method in the retrieval system executed by the electronic device 20 disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs that can be used to complete other specific tasks. The data 223 may include not only the data transmitted by external devices received by the electronic device, but also the data collected by its own input / output interface 25, etc.

[0093] Furthermore, the embodiment of the present application also discloses a computer-readable storage medium, in which a computer program is stored. When the computer program is loaded and executed by a processor, the steps of the retrieval optimization method in the retrieval system disclosed in any of the foregoing embodiments are implemented.

[0094] The embodiment of the present invention also discloses a computer program product, including a computer program / instructions. When the computer program / instructions are executed by a processor, the steps of the retrieval optimization method in the retrieval system disclosed in any of the foregoing embodiments are implemented.

[0095] In this specification, each embodiment is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. The same or similar parts among the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple. For the relevant parts, refer to the description in the method part.

[0096] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in this article can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but this implementation should not be considered to exceed the scope of this application.

[0097] The steps of the methods or algorithms described in connection with the embodiments disclosed herein may be implemented directly in hardware, in software modules executed by a processor, or in a combination thereof. The software modules may be located in Random Access Memory (RAM), internal memory, Read-Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, Compact Disc Read-Only Memory (CD-ROM), or any other form of storage medium known in the art.

[0098] Finally, it should also be noted that in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variation thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the phrase "comprising a..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0099] The above has introduced in detail a retrieval optimization method, device, equipment, medium and product in a retrieval system provided by the present invention. Specific examples are used in this document to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A retrieval optimization method in a retrieval system, characterized in that Including: Obtain the original document and extract target keywords from the original document; Match the target keywords with current partitions to determine a first partition that matches the target keywords, store the original document in the first partition, and calculate a document score corresponding to the original document; wherein, the document score is dynamically updated according to a preset rule; When a retrieval request from the client is obtained, determine a target partition parameter and a target score parameter based on the retrieval request; Locate a second partition based on the target partition parameter and screen out matching document content from the second partition based on the target score parameter, so as to return the document content to the client.

2. The retrieval optimization method in the retrieval system according to claim 1, characterized in that, The extracting target keywords from the original document includes: Use a preset word segmenter to extract corresponding keywords from the original document and use them as target keywords; wherein, the preset word segmenter is constructed based on a first word segmenter and a second word segmenter; wherein, the first word segmenter is constructed based on a general word segment library, and the second word segmenter is constructed based on a custom word segment library set by the client.

3. The retrieval optimization method in the retrieval system according to claim 2, characterized in that The extracting target keywords from the original document includes: Perform sharding processing on the original document to obtain multiple shard fragments; Calculate the semantic density of each shard fragment, and screen out target shard fragments with a semantic density greater than a preset density threshold from each shard fragment; Use the preset word segmenter to extract corresponding keywords from the target shard fragments and use them as target keywords.

4. The retrieval optimization method in the retrieval system according to claim 3, characterized in that, The calculating the semantic density of each shard fragment includes: For any shard fragment, calculate the term frequency-inverse document frequency of each word in the shard fragment, and sum up the term frequency-inverse document frequencies to obtain a summation result; Determine the semantic density of the shard fragment based on the summation result and the shard length of the shard fragment.

5. The retrieval optimization method in the retrieval system according to claim 1, wherein, The matching the target keywords with current partitions to determine a first partition that matches the target keywords includes: Obtain keyword lists respectively corresponding to current partitions, and determine whether the target keywords are included in each keyword list; If there is a target keyword list including the target keywords, use the partition corresponding to the target keyword list as the first partition that matches the target keywords.

6. The retrieval optimization method in the retrieval system according to claim 1, characterized in that, The matching the target keywords with current partitions to determine a first partition that matches the target keywords includes: Obtain regular expressions respectively corresponding to current partitions; wherein, the regular expressions are constructed based on text features and target characters; Calculate the similarity between the target keywords and each regular expression respectively; If the similarity between any regular expression and the target keywords is greater than a preset similarity threshold, use the partition corresponding to the regular expression as the first partition that matches the target keywords.

7. The retrieval optimization method in the retrieval system according to claim 1, characterized in that, During the process of matching the target keywords with current partitions to determine a first partition that matches the target keywords, it includes: If there is no partition in the current partitions that matches the target keyword, a new partition is created based on the target keyword, and the new partition is used as the first partition that matches the target keyword.

8. The retrieval optimization method in the retrieval system according to claim 1, wherein The calculating the document score corresponding to the original document includes: Scoring the original document based on a preset multi-dimensional scoring model to obtain the corresponding document score; wherein, the multi-dimensional scoring model is constructed based on the original score value, the document click-through rate, the time decay factor, and the corresponding weight coefficients. The original score value is the score obtained by using the internal scoring mechanism of the distributed retrieval system, the document click-through rate is the cumulative click-through rate of the original document, and the time decay factor is calculated based on a preset time decay formula.

9. The retrieval optimization method in the retrieval system according to claim 8, wherein Dynamically updating the document score based on preset rules includes: Obtaining the latest cumulative click-through rate and the latest time decay factor; Recalculating the score of the original document based on the original score value, the latest cumulative click-through rate, the latest time decay factor, and the corresponding weight coefficients to update the document score.

10. The retrieval optimization method in the retrieval system according to claim 9, characterized in that, The obtaining the latest cumulative click-through rate and the latest time decay factor includes: Obtaining the current latest cumulative click-through rate and the latest time decay factor at preset time intervals; Or, when the current load of the distributed retrieval system is lower than the preset load threshold and the current time is within a preset time range, obtaining the current latest cumulative click-through rate and the latest time decay factor.

11. The retrieval optimization method in the retrieval system according to any one of claims 1 to 10, characterized in that, The target partition parameters include partition retrieval parameters and a target search parameter indicating whether to search for associated partitions; Correspondingly, the locating the second partition based on the target partition parameters includes: If the target search parameter indicates not to search for associated partitions, determining the corresponding original retrieval partition based on the partition retrieval parameters, and using the original retrieval partition as the second partition; If the target search parameter indicates to search for associated partitions, locating the corresponding original retrieval partition based on the partition retrieval parameters, and determining the associated partition that has an association relationship with the original retrieval partition, so as to use the original retrieval partition and the associated partition as the second partition.

12. The retrieval optimization method in the retrieval system according to claim 11, characterized in that, It further includes: If the number of the target keywords is multiple, performing semantic analysis on each of the target keywords to determine whether there is a semantic similarity relationship between the target keywords based on the semantic analysis result; If there are multiple target keywords that satisfy the semantic similarity relationship, establishing the association relationship between the first partitions corresponding to the multiple target keywords.

13. The retrieval optimization method in the retrieval system according to claim 11, characterized in that, The determining the target partition parameters and the target score parameters based on the retrieval request includes: Judging whether the retrieval request carries partition parameters and score parameters; If so, using the partition parameters and the score parameters carried in the retrieval request as the target partition parameters and the target score parameters respectively.

14. The retrieval optimization method in the retrieval system according to claim 13, characterized in that, After judging whether the retrieval request carries partition parameters and score parameters, it further includes: If the retrieval request does not carry partition parameters and score parameters, performing word segmentation processing and semantic parsing on the retrieval content in the retrieval request to obtain partition retrieval parameters; Query whether the target search parameters and scoring parameters pre-configured for the client exist in the preset configuration file; If they exist, obtain target partition parameters based on the target search parameters pre-configured for the client and the partition search parameters determined using the retrieved content, and use the scoring parameters pre-configured for the client as the target scoring parameters; If they do not exist, obtain target partition parameters based on the default search parameters stored in the preset configuration file and the partition search parameters determined using the retrieved content, and use the default scoring parameters stored in the preset configuration file as the target scoring parameters.

15. The retrieval optimization method in the retrieval system according to claim 11, wherein The target scoring parameters include a scoring threshold; Correspondingly, screening out matching document content from the second partition based on the target scoring parameters includes: Screening out target documents with a document score higher than the scoring threshold from the second partition, and determining the document content from the target documents.

16. The retrieval optimization method in the retrieval system according to claim 15, characterized in that, During the process of storing the original document into the first partition, it further includes: Extracting global information of the original document to construct a main index; Segmenting and storing the original document according to the document size, and constructing a sub-index for each segment; wherein, each sub-index is an associated index of the main index.

17. The retrieval optimization method in the retrieval system according to claim 16, characterized in that, The target partition parameters further include a document retrieval mode. Determining the document content from the target documents includes: If the partition search parameter represents full-document retrieval, use all the content in the target document as the document content; If the partition search parameter does not represent full-document retrieval, match the partition search parameter with the main index and each sub-index respectively; If the partition search parameter successfully matches the main index, use the global information corresponding to the main index as the document content; If the partition search parameter successfully matches any sub-index, use the segment corresponding to the any sub-index as the document content.

18. A retrieval optimization device in a retrieval system, characterized in that, It includes: A keyword extraction module, configured to obtain an original document and extract target keywords from the original document; A document storage module, configured to match the target keywords with current partitions to determine a first partition that matches the target keywords, store the original document into the first partition, and calculate a document score corresponding to the original document; wherein, the document score is dynamically updated according to a preset rule; A parameter determination module, configured to, when receiving a retrieval request from a client, determine target partition parameters and target scoring parameters based on the retrieval request; A document retrieval module, configured to locate a second partition based on the target partition parameters, and screen out matching document content from the second partition based on the target scoring parameters, so as to return the document content to the client.

19. An electronic device, characterized in that, It includes: A memory, configured to store a computer program; A processor, configured to execute the computer program to implement the steps of the retrieval optimization method in the retrieval system according to any one of claims 1 to 17.

20. A computer-readable storage medium, characterized in that, For storing a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the retrieval optimization method in the retrieval system according to any one of claims 1 to 17.

21. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by a processor, the steps of the retrieval optimization method in the retrieval system according to any one of claims 1 to 17 are implemented.

Images