Search optimization method, device, equipment, medium and product in search system

Abstract

The present application discloses a retrieval optimization method, apparatus, device, medium, and product in a retrieval system, which relates to the field of information retrieval technology, including: obtaining an original document and extracting target keywords from the original document; matching the target keywords with the current partitions to determine a first partition that matches the target keywords, storing the original document in the first partition, and calculating a document score corresponding to the original document; wherein the document score is dynamically updated according to preset rules; upon obtaining a retrieval request from a user end, determining target partition parameters and target score parameters based on the retrieval request; locating a second partition based on the target partition parameters, and filtering out matching document content from the second partition based on the target score parameters to return the document content to the user end. The present application can optimize the retrieval process of the retrieval system to improve retrieval performance and efficiency.

Description

Technical Field

[0001] The present invention relates to the field of information retrieval technology, and in particular to a retrieval optimization method, device, equipment, medium and product in a retrieval system. Background Art

[0002] The performance bottlenecks of existing distributed retrieval systems are mainly reflected in two aspects: (1) low index construction efficiency: high time and space complexity, using a fixed word segmentation model to build the index, it is very time-consuming to process large amounts of data, and the index structure is only suitable for conventional retrieval; (2) low efficiency of the retrieval process: data transmission and computing efficiency are low, and traditional algorithm optimization based on the scan matching mechanism has great limitations and cannot significantly improve retrieval performance without the original scanning mechanism. The reason is that the existing retrieval and index optimization methods are based on a single index structure, which is difficult to meet the diverse retrieval needs and cannot efficiently handle the complex needs of large-scale distributed retrieval scenarios.

[0003] In summary, how to optimize the retrieval process of the retrieval system to improve retrieval performance and efficiency is a problem that needs to be solved. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a search optimization method, device, equipment, medium and product that can optimize the search process of the search system to improve search performance and efficiency. The specific solution is as follows:

[0005] In a first aspect, the present application discloses a search optimization method in a search system, comprising:

[0006] Obtain original documents and extract target keywords from the original documents;

[0007] Matching the target keyword with each current partition to determine the first partition that matches the target keyword, storing the original document in the first partition, and calculating a document score corresponding to the original document; wherein the document score is dynamically updated according to preset rules;

[0008] When a search request is received from a user, target partition parameters and target scoring parameters are determined based on the search request;

[0009] The second partition is located based on the target partition parameter, and matching document content is filtered out from the second partition based on the target score parameter, so as to return the document content to the user end.

[0010] Optionally, extract target keywords from the original document, including:

[0011] A preset word segmenter is used to extract corresponding keywords from the original document and use them as target keywords; wherein, the preset word segmenter is constructed based on the first word segmenter and the second word segmenter; wherein, the first word segmenter is constructed based on a general word segmentation library, and the second word segmenter is constructed based on a custom word segmentation library set by the user end.

[0012] Optionally, extract target keywords from the original document, including:

[0013] Segment the original document to obtain multiple segment fragments;

[0014] Calculate the semantic density of each fragment and select the target fragment whose semantic density is greater than a preset density threshold from each fragment;

[0015] Use the preset word segmenter to extract the corresponding keywords from the target segment and use them as target keywords.

[0016] Optionally, calculate the semantic density of each fragment, including:

[0017] For any fragment, calculate the word frequency-inverse document frequency of each word in any fragment, and sum the word frequency-inverse document frequency to obtain the sum result;

[0018] The semantic density of any slice segment is determined based on the summation result and the slice length of any slice segment.

[0019] Optionally, matching the target keyword with each current partition to determine the first partition matching the target keyword includes:

[0020] Obtain the keyword lists corresponding to each current partition and determine whether each keyword list contains the target keyword;

[0021] If there is a target keyword list including the target keyword, the partition corresponding to the target keyword list is used as the first partition matching the target keyword.

[0022] Optionally, matching the target keyword with each current partition to determine the first partition matching the target keyword includes:

[0023] Get the regular expressions corresponding to each current partition; the regular expressions are constructed based on text features and target characters;

[0024] Calculate the similarity between the target keyword and each regular expression respectively;

[0025] If the similarity between any regular expression and the target keyword is greater than a preset similarity threshold, the partition corresponding to any regular expression is used as the first partition matching the target keyword.

[0026] Optionally, the process of matching the target keyword with the current partitions to determine the first partition matching the target keyword includes:

[0027] If no partition matching the target keyword exists in the current partitions, a new partition is created based on the target keyword, and the new partition is used as the first partition matching the target keyword.

[0028] Optionally, calculate the document score corresponding to the original document, including:

[0029] The original document is scored based on a preset multi-dimensional scoring model to obtain a corresponding document score; wherein, the multi-dimensional scoring model is constructed based on the original score value, the document click volume, the time decay factor and the corresponding weight coefficient, the original score value is the score value obtained by scoring using the internal scoring mechanism of the distributed retrieval system, the document click volume is the cumulative click volume of the original document, and the time decay factor is calculated based on the preset time decay formula.

[0030] Optionally, dynamically update document scores based on pre-set rules, including:

[0031] Get the latest cumulative click volume and the latest time decay factor;

[0032] The score of the original document is recalculated based on the original score value, the latest cumulative click volume, the latest time decay factor and the corresponding weight coefficient to update the document score.

[0033] Optionally, get the latest cumulative click count and the latest time decay factor, including:

[0034] Obtain the latest cumulative click volume and the latest time decay factor at every preset time interval;

[0035] Alternatively, when the current load of the distributed retrieval system is lower than a preset load threshold and the current time is within a preset time range, the latest cumulative click volume and the latest time decay factor are obtained.

[0036] Optionally, the target partition parameters include a partition search parameter and a target search parameter indicating whether to search for associated partitions;

[0037] Accordingly, locating the second partition based on the target partition parameters includes:

[0038] If the target search parameter indicates that the associated partition is not to be searched, then the corresponding original search partition is determined based on the partition search parameter, and the original search partition is used as the second partition;

[0039] If the target search parameter represents searching for an associated partition, the corresponding original search partition is located based on the partition search parameter, and an associated partition associated with the original search partition is determined, so that the original search partition and the associated partition are used as the second partition.

[0040] Optionally, the search optimization method in the search system of the present application further includes:

[0041] If there are multiple target keywords, semantic analysis is performed on each target keyword to determine whether there is a semantic similarity relationship between the target keywords based on the semantic analysis result;

[0042] If there are multiple target keywords that satisfy a semantic similarity relationship, an association relationship is established between the first partitions corresponding to the multiple target keywords.

[0043] Optionally, target partition parameters and target scoring parameters are determined based on the retrieval request, including:

[0044] Determine whether the search request carries partition parameters and scoring parameters;

[0045] If so, the partition parameter and score parameter carried in the retrieval request are used as the target partition parameter and target score parameter respectively.

[0046] Optionally, after determining whether the search request carries the partition parameter and the scoring parameter, the following steps are also performed:

[0047] If the search request does not carry partition parameters and scoring parameters, the search content in the search request is segmented and semantically analyzed to obtain the partition search parameters;

[0048] Querying the preset configuration file to determine whether the target search parameters and scoring parameters pre-configured by the user end exist;

[0049] If so, the target partition parameter is obtained based on the target search parameter pre-configured by the user terminal and the partition search parameter determined by the search content, and the scoring parameter pre-configured by the user terminal is used as the target scoring parameter;

[0050] If not, the target partition parameters are obtained based on the default search parameters stored in the preset configuration file and the partition search parameters determined using the search content, and the default scoring parameters stored in the preset configuration file are used as the target scoring parameters.

[0051] Optionally, the target scoring parameters include a scoring threshold;

[0052] Accordingly, matching document content is filtered out from the second partition based on the target scoring parameter, including:

[0053] Target documents with a document score higher than a score threshold are screened out from the second partition, and document content is determined from the target documents.

[0054] Optionally, the process of storing the original document in the first partition further includes:

[0055] Extract global information of original documents to build the main index;

[0056] The original document is stored in segments according to the document size, and a sub-index is constructed for each segment; each sub-index is an associated index of the main index.

[0057] Optionally, the target partition parameters also include a document retrieval mode to determine the document content from the target document, including:

[0058] If the partitioned search parameter indicates a full document search, the entire content of the target document is used as the document content;

[0059] If the partitioned search parameter does not represent a full document search, the partitioned search parameter is matched against the main index and each sub-index respectively;

[0060] If the partition search parameters successfully match the primary index, the global information corresponding to the primary index is used as the document content;

[0061] If the partition search parameter successfully matches any sub-index, the segment corresponding to any sub-index will be used as the document content.

[0062] In a second aspect, the present application discloses a search optimization device in a search system, comprising:

[0063] A keyword extraction module is used to obtain original documents and extract target keywords from the original documents;

[0064] A document storage module is configured to match the target keyword with the current partitions to determine a first partition that matches the target keyword, store the original document in the first partition, and calculate a document score corresponding to the original document; wherein the document score is dynamically updated according to preset rules;

[0065] A parameter determination module, configured to determine target partition parameters and target scoring parameters based on a search request received from a user terminal;

[0066] The document retrieval module is used to locate the second partition based on the target partition parameter, and filter out matching document content from the second partition based on the target scoring parameter, so as to return the document content to the user end.

[0067] In a third aspect, the present application discloses an electronic device, comprising:

[0068] Memory, used to store computer programs;

[0069] The processor is used to execute a computer program to implement the steps of the search optimization method in the aforementioned disclosed search system.

[0070] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the steps of the retrieval optimization method in the aforementioned disclosed retrieval system are implemented.

[0071] In a fifth aspect, the present application discloses a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the retrieval optimization method in the aforementioned disclosed retrieval system.

[0072] It can be seen that the present application obtains the original document and extracts the target keyword from the original document; matches the target keyword with the current partitions to determine the first partition that matches the target keyword, stores the original document in the first partition, and calculates the document score corresponding to the original document; wherein the document score is dynamically updated according to preset rules; when the retrieval request from the user end is obtained, the target partition parameters and target score parameters are determined based on the retrieval request; the second partition is located based on the target partition parameters, and the matching document content is filtered out from the second partition based on the target score parameters to return the document content to the user end.

[0073] Beneficial effects: After obtaining the original document, the present application first needs to extract the target keywords from the original document, and the present application has pre-built multiple partitions. After extracting the target keywords, the target keywords are matched with the current partitions to determine the first partition that matches the target keywords, and the original document is stored in the first partition. That is, the present application stores the original documents by matching keywords with partitions, realizing centralized storage of documents by semantic classification, that is, it can achieve a strong correlation between the content semantics of documents stored in the same partition. In addition, while storing the original document, the document score corresponding to the original document is also calculated, and over time, the document score is dynamically updated according to preset rules. Furthermore, when a retrieval request from the user end is obtained, it is first necessary to determine the target partition parameters and target score parameters based on the retrieval request. In this way, the corresponding second partition can be quickly located based on the targeted retrieval method of the target partition parameters and target score parameters, and the matching document content can be extracted from the second partition and returned, thereby improving the retrieval performance. The above solution can ensure that the retrieval system has better response performance and stability while ensuring retrieval accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0075] Figure 1 A flowchart of a search optimization method in a search system disclosed in this application;

[0076] Figure 2 A flowchart of a specific search optimization method in a search system disclosed in this application;

[0077] Figure 3 A diagram of a search optimization architecture in a search system disclosed in this application;

[0078] Figure 4 This is a schematic diagram of the structure of a search optimization device in a search system disclosed in this application;

[0079] Figure 5 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0080] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0081] The performance bottlenecks of existing distributed retrieval systems are mainly reflected in two aspects: (1) low efficiency of index construction: high time and space complexity, using a fixed word segmentation pattern to build an index, it takes a lot of time to process a large amount of data, and the index structure is only suitable for conventional retrieval; (2) low efficiency of the retrieval process: data transmission and computing efficiency are low, and the traditional algorithm optimization based on the scan matching mechanism has great limitations, and it is impossible to achieve a significant improvement in retrieval performance without breaking away from the original scanning mechanism. The reason is that the existing retrieval and index optimization methods are all based on a single index structure, which makes it difficult to meet diversified retrieval needs and cannot efficiently handle the complex needs in large-scale distributed retrieval scenarios. To this end, the embodiments of the present application disclose a retrieval optimization method, device, equipment, medium and product in a retrieval system, which can optimize the retrieval process of the retrieval system to improve retrieval performance and efficiency.

[0082] See also Figure 1 As shown, the embodiment of the present application discloses a search optimization method in a search system, the method comprising:

[0083] Step S11: Obtain an original document and extract target keywords from the original document.

[0084] The retrieval system in this embodiment may be a distributed retrieval system, and after obtaining the original document, it is necessary to create an index for the original document stored in the database.

[0085] In a specific embodiment, extracting target keywords from an original document includes: extracting corresponding keywords from the original document using a preset word segmenter and using them as target keywords; wherein the preset word segmenter is constructed based on a first word segmenter and a second word segmenter; wherein the first word segmenter is constructed based on a general word segmentation library, and the second word segmenter is constructed based on a custom word segmentation library set by the user end. It can be understood that the present application can use the preset word segmenter to directly extract keywords from the original document. In addition to including the first word segmenter constructed based on the general word segmentation library, the preset word segmenter processing can also include a second word segmenter constructed based on the custom word segmentation library set by the user end. That is, the word segmenter in the present application supports flexible user-defined configuration, so that it can be applied to different application scenarios.

[0086] Among them, the general word segmentation library is a standardized word library designed for basic language processing, aiming to cover the needs of segmenting everyday language and general terminology, such as word segmentation by common objects, species, solid and liquid states, or word segmentation by grammatical structure (such as subject, predicate, and object). Custom word segmentation libraries are configured to enter specific phrases that exist in specific fields or industries. For example, some specialized industries, such as chemistry, physics, and law, have industry-specific vocabulary, such as the term "aromatic chemical substances." This type of segmentation has highly specialized characteristics and requires the configuration of its own unique word segmenter in the business system of the relevant industry.

[0087] In another specific embodiment, extracting target keywords from an original document includes: segmenting the original document to obtain multiple fragments; calculating the semantic density of each fragment, and selecting target fragments from each fragment whose semantic density exceeds a preset density threshold; and extracting corresponding keywords from the target fragments using a preset word segmenter as target keywords. In other words, the present application can first pre-process the original document to filter out some low-value fragments, and then extract keywords from the remaining fragments.

[0088] Specifically, in this embodiment, the original document is segmented to obtain multiple segmented fragments, and then the semantic density (Semantic Density, SD) of each segmented fragment is calculated. The semantic density is mainly used to measure the degree of text information concentration of the entire segmented fragment, that is, to quantify the information value of each segmented fragment. Then, the target segmented fragments with semantic density greater than the preset density threshold are screened out from each segmented fragment, that is, some fragments with higher information value are screened out, and some low-value segmented fragments are filtered out. For example, the preset density threshold can be set to 0.85, then it can be understood that the segmented fragments with SD < 0.15 are discarded. Finally, the preset tokenizer is used to extract keywords from these screened target segmented fragments to obtain target keywords. In this way, there is no need to extract keywords for the entire document, which can reduce the overhead caused by inefficient index construction.

[0089] Among them, calculating the semantic density of each segmented fragment includes: for any segmented fragment, calculating the term frequency-inverse document frequency of each word in the segmented fragment, and summing up the term frequency-inverse document frequencies to obtain a summation result; determining the semantic density of the segmented fragment based on the summation result and the length of the segmented fragment. That is, the semantic density is mainly calculated based on the term frequency-inverse document frequency (Term Frequency-Inverse Document Frequency, TF-IDF) of each word in a single segmented fragment and the length of the segmented fragment. It can be understood that TF-IDF is a statistical method used to evaluate the importance of words in a document, which is widely used in the fields of text analysis, information retrieval, and natural language processing. Its core idea is to measure the discrimination of words by combining the term frequency (TF) and the inverse document frequency (IDF). Among them, the term frequency refers to the number of times a word appears in a document. For example, if the word "apple" appears twice in a document, while other words only appear once, then the term frequency of "apple" is higher, and it may contribute more to the content of the document; the inverse document frequency reflects the universality of words in the corpus. For example, common words such as "of" and "is" appear frequently in most documents, and their IDF values are relatively low; while professional terms such as "quantum entanglement" only appear in documents in specific fields, and their IDF values are relatively high.

[0090] Therefore, in this application, the term frequency-inverse document frequency of each word in a single segmented fragment is calculated, and then the summation result is obtained by summing up all the term frequency-inverse document frequencies of the segmented fragment. Then, the semantic density can be calculated based on the summation result and the length of the segmented fragment. The specific calculation formula is as follows:

[0091] SD = ∑(TF-IDF) / fragment length.

[0092] Step S12: Match the target keyword with the current partitions to determine the first partition that matches the target keyword, store the original document in the first partition, and calculate the document score corresponding to the original document; wherein the document score is dynamically updated according to preset rules.

[0093] In this embodiment, multiple partitions have been pre-built. After the target keyword is extracted, the target keyword is matched with each current partition to determine the first partition that matches the target keyword, and the original document is stored in the first partition. That is, the present application stores the original document by matching keywords with partitions, and realizes the centralized storage of documents by semantic classification, that is, it can achieve a strong correlation between the content semantics of documents stored in the same partition. In addition, while storing the original document, the document score corresponding to the original document is also calculated, and as time goes by, the document score will be dynamically updated according to the preset rules. In this way, after the complete process of keyword extraction, partition matching, document storage and score calculation mentioned above, the index construction process of the original document is completed.

[0094] In one specific embodiment, matching the target keyword with each current partition to determine the first partition that matches the target keyword includes: obtaining keyword lists corresponding to each current partition, and determining whether each keyword list includes the target keyword; if a target keyword list exists that includes the target keyword, then determining the partition corresponding to the target keyword list as the first partition that matches the target keyword. In other words, each current partition maintains its own corresponding keyword list, and when matching the target keyword with each current partition, determining whether each keyword list includes the target keyword. If a target keyword list exists that includes the target keyword, then a partition that matches the target keyword exists, and therefore the partition corresponding to the target keyword list is determined as the first partition that matches the target keyword.

[0095] In another specific embodiment, the target keyword is matched with each current partition to determine the first partition that matches the target keyword, including: obtaining regular expressions corresponding to each current partition; wherein the regular expressions are constructed based on text features and target characters; respectively calculating the similarity between the target keyword and each regular expression; if the similarity between any regular expression and the target keyword is greater than a preset similarity threshold, then the partition corresponding to any regular expression is used as the first partition that matches the target keyword. It is understandable that, considering that the models of some products in the same series may be different, or some synonyms are expressed differently, this embodiment sets a corresponding regular expression for each partition. The regular expression can flexibly match keywords that meet specific patterns, and is particularly suitable for processing complex, variable, or partially known terms; wherein the regular expression is mainly constructed based on text features and target characters. In this embodiment, by calculating the similarity between the target keyword and each regular expression, if the similarity between any regular expression and the target keyword is greater than a preset similarity threshold, then the partition corresponding to any regular expression is used as the first partition that matches the target keyword.

[0096] It should be noted that high / medium / low score matching partitions can also be divided according to the size of the preset similarity threshold, which can specifically correspond to a similarity higher than 80%, a similarity between 50% and 80%, and a similarity between 30% and 50%.

[0097] It should also be noted that the process of matching the target keyword with the current partitions to determine the first partition that matches the target keyword also includes: if there is no partition matching the target keyword in the current partitions, then a new partition is created based on the target keyword, and the new partition is used as the first partition that matches the target keyword. In other words, if the keyword lists of the current partitions do not contain the target keyword, or if the similarity between the target keyword and the regular expression is less than 30% or even less than 10%, then it means that the current partitions do not match the target keyword. In this case, a new partition can be created for the target keyword, and the new partition can be used as the first partition that matches the target keyword.

[0098] Step S13: When a search request is received from the user, target partition parameters and target scoring parameters are determined based on the search request.

[0099] In this embodiment, when a search request is received from a user, it is first necessary to determine a target partition parameter and a target scoring parameter based on the search request.

[0100] Step S14: Locate the second partition based on the target partition parameter, and filter out matching document content from the second partition based on the target score parameter, so as to return the document content to the user end.

[0101] In this embodiment, after determining the target partition parameters and target scoring parameters based on the retrieval request, the corresponding second partition can be quickly located based on the targeted retrieval method of the target partition parameters and target scoring parameters, and the matching document content can be extracted from the second partition and returned, thereby improving the retrieval performance.

[0102] As can be seen, after obtaining the original document, the present application first needs to extract the target keywords from the original document, and the present application has pre-built multiple partitions. After extracting the target keywords, the target keywords are matched with the current partitions to determine the first partition that matches the target keywords, and the original document is stored in the first partition. In other words, the present application stores the original documents by matching keywords with partitions, realizing centralized storage of documents according to semantic classification, that is, it can achieve a strong correlation between the content semantics of documents stored in the same partition. In addition, while storing the original document, the document score corresponding to the original document is also calculated, and the document score is dynamically updated according to preset rules over time. Furthermore, when a search request from the user end is obtained, it is first necessary to determine the target partition parameters and target score parameters based on the search request. In this way, the corresponding second partition can be quickly located based on the target partition parameters and target score parameters in a targeted search method, and the matching document content can be extracted from the second partition and returned, thereby improving the search performance. The above solution can ensure that the retrieval system has better response performance and stability while ensuring retrieval accuracy.

[0103] See also Figure 2 As shown, the embodiment of the present application discloses a specific search optimization method in a search system. Compared with the previous embodiment, this embodiment further explains and optimizes the technical solution. Specifically, it includes:

[0104] Step S21: Obtain an original document and extract target keywords from the original document.

[0105] Step S22: Match the target keyword with the current partitions to determine the first partition that matches the target keyword, store the original document in the first partition, and score the original document based on a preset multi-dimensional scoring model to obtain a corresponding document score; wherein the document score is dynamically updated according to preset rules.

[0106] In this embodiment, after the original document is stored in the first partition, it is necessary to score the document. That is, when the document is first stored in the partition, a corresponding document score is calculated. The score is then refreshed periodically based on internal preset rules. This facilitates determining the degree of match of the results based on the scoring parameters during subsequent searches. Documents with low scores are excluded from search results. Specifically, this application scores the original document based on a preset multi-dimensional scoring model to obtain a corresponding document score, and records the document score in a pre-configured scoring area.

[0107] The multi-dimensional scoring model is constructed based on the original score value, document click volume, time decay factor, and corresponding weight coefficients. The original score value is the score value obtained by scoring using the internal scoring mechanism of the distributed retrieval system, the document click volume is the cumulative click volume of the original document, and the time decay factor is calculated based on a preset time decay formula. In other words, this embodiment further combines the document click volume and the time decay factor to generate a new document score based on the original internal scoring mechanism of the distributed retrieval system. The specific expression of the multi-dimensional scoring model is:

[0108] Score_final=α×ES_score+β×log(1+click_count)+γ×time_decay;

[0109] Among them, ES_score is the original score value obtained by scoring using the internal scoring mechanism of the distributed retrieval system; click_count is the cumulative number of clicks on the original document, which can represent its access popularity; time_decay is the time decay factor, that is, the term carries time decay information when it is retrieved. For example, the hot words of a social event have high popularity when the event occurs. As the popularity of the event decreases, the popularity of the term will also decay. When the words accompanying the term are retrieved when the popularity is high, the term will also be retrieved with a high probability; α+β+γ=1, in a specific implementation method, α=0.6, β=0.3, γ=0.1, or α=0.3, β=0.5, γ=0.2; the specific weight ratio can be set according to the specific scenario, and this embodiment does not limit this.

[0110] In addition, the expression of the preset time attenuation formula is specifically:

[0111] ;

[0112] Among them, t is the time when the document is stored, It can be 0.05.

[0113] In a specific embodiment, dynamically updating a document score based on preset rules includes: obtaining the latest cumulative click count and the latest time decay factor; and recalculating the original document score based on the original score value, the latest cumulative click count, the latest time decay factor, and the corresponding weight coefficient to update the document score. That is, in order to refresh the document score, this embodiment requires obtaining the latest cumulative click count and the latest time decay factor, and then recalculating the original document score based on the original score value, the latest cumulative click count, the latest time decay factor, and the corresponding weight coefficient to update the document score.

[0114] Among them, obtaining the latest cumulative clicks and the latest time attenuation factor includes: obtaining the current latest cumulative clicks and the latest time attenuation factor at every preset time interval; or, when the current load of the distributed retrieval system is lower than the preset load threshold and the current time is within the preset time range, obtaining the current latest cumulative clicks and the latest time attenuation factor.

[0115] It is understandable that, in one specific embodiment, a method of regularly refreshing the document score can be adopted, for example, obtaining the current latest cumulative click volume and the latest time decay factor every 12 hours or every other day, and then recalculating the document score. In another specific embodiment, in order to ensure that the system currently has sufficient computing power to complete the refresh of the document score and avoid resource competition during peak hours, which may lead to the failure of the score refresh, this embodiment can only perform the score refresh operation when the current load of the distributed retrieval system is lower than the preset load threshold and the current time is within the preset time range. For example, the score refresh can be performed when the CPU load is <30% and the current time is UTC 02:00-04:00. On the one hand, by determining the current load, it can ensure that the system has sufficient idle resources to handle computationally intensive operations, and prevent refresh interruptions or timeouts due to insufficient resources; on the other hand, considering that real-time retrieval services need to be prioritized during the daytime business peak, triggering the score refresh at this time may cause CPU / memory competition.

[0116] Step S23: upon receiving a search request from the user, determining target partition parameters and target scoring parameters based on the search request; the target partition parameters include partition search parameters and target search parameters indicating whether to search for associated partitions; and the target scoring parameters include a scoring threshold.

[0117] In this embodiment, when a search request is received from a user, target partition parameters and target scoring parameters need to be determined based on the search request. Specifically, the target partition parameters may include, but are not limited to, partition search parameters and target search parameters indicating whether to search for associated partitions, while the target scoring parameters may include, but are not limited to, scoring thresholds.

[0118] Among them, the target partition parameters and target scoring parameters are determined based on the retrieval request, including: judging whether the retrieval request carries partition parameters and scoring parameters; if so, the partition parameters and scoring parameters carried in the retrieval request are used as target partition parameters and target scoring parameters respectively. That is, the user can choose whether to carry partition parameters and scoring parameters in the retrieval request. If the retrieval request carries partition parameters and scoring parameters, the partition parameters and scoring parameters carried in the retrieval request can be directly used as target partition parameters and target scoring parameters respectively. For example, when the user is searching, the partition retrieval parameter is set to the aroma partition information and there is no need to search for associated partitions, that is, only documents in the aroma partition are retrieved, and the scoring threshold is set to 9.3 (that is, the document score must be greater than 9.3). Then, based on the above conditions, the search scope can be limited during retrieval, thereby improving the query speed and realizing targeted retrieval on the user side. That is, this application supports customized partition parameters and scoring thresholds when searching on the application side, so as to retrieve the desired results more accurately and quickly.

[0119] Furthermore, after determining whether the retrieval request carries partition parameters and scoring parameters, it also includes: if the retrieval request does not carry partition parameters and scoring parameters, performing word segmentation processing and semantic analysis on the retrieval content in the retrieval request to obtain partition retrieval parameters; querying the preset configuration file whether there are target search parameters and scoring parameters pre-configured by the user end; if so, obtaining the target partition parameters based on the target search parameters pre-configured by the user end and the partition retrieval parameters determined using the retrieval content, and using the scoring parameters pre-configured by the user end as the target scoring parameters; if not, obtaining the target partition parameters based on the default search parameters stored in the preset configuration file and the partition retrieval parameters determined using the retrieval content, and using the default scoring parameters stored in the preset configuration file as the target scoring parameters.

[0120] That is, if the search request does not carry the partition parameters and scoring parameters, it is necessary to further perform word segmentation processing and semantic analysis on the search content in the search request to obtain the partition search parameters, and then query the preset configuration file to see whether the target search parameters and scoring parameters pre-configured by the user end exist, that is, whether the user end has pre-configured the target search parameters and scoring parameters. If so, the target partition parameters are obtained based on the target search parameters pre-configured by the user end and the partition search parameters determined using the search content, and the scoring parameters pre-configured by the user end are used as the target scoring parameters. For example, if the partition search parameters obtained by performing word segmentation processing and semantic analysis on the search content are alcohol partition information, and the target search parameters pre-configured by the user end indicate that it is necessary to search for associated partitions, assuming that the associated partition of the alcohol partition is the aromatic partition information, and the scoring threshold is 9.8, then the search scope can also be limited according to the above conditions during the search, thereby improving the query speed.

[0121] In addition, if the user does not include relevant parameters in the search request and the user does not have relevant configuration in the preset configuration file, after performing word segmentation and semantic analysis on the search content in the search request to obtain the partition search parameters, the default search parameters and default scoring parameters stored in the preset configuration file are used for the query. The default search parameters are generally set to not search related partitions, and the default scoring parameter can be set to 8.0.

[0122] It can also be seen from the above content that when performing a targeted search, the priority of using the parameters carried in the search request is greater than the priority of using the parameters configured by the user in the preset configuration file, and the priority of using the default parameters in the preset configuration file. That is, if the user actively sends down the targeted search parameters, such as actively setting the partition parameters and scoring parameters when querying, the query will be based on the partition parameters and scoring parameters set by the user first; if the user query does not actively include the partition parameters and scoring parameters, it is necessary to check whether there are any parameters manually configured by the user in the preset configuration file, such as limiting the query to the aroma partition only, and setting the score to 9.8 by default. If so, use them; if the user query does not actively include the partition parameters and scoring parameters, and there are no parameters manually configured by the user in the preset configuration file, it is necessary to use the built-in default search parameters and default scoring parameters for the query.

[0123] Step S24: If the target search parameter indicates that no associated partition is to be searched, the corresponding original search partition is determined based on the partition search parameter, and the original search partition is used as the second partition; if the target search parameter indicates that an associated partition is to be searched, the corresponding original search partition is located based on the partition search parameter, and an associated partition having an associated relationship with the original search partition is determined, so that the original search partition and the associated partition are used as the second partition.

[0124] In a specific embodiment, if the target search parameter indicates that no associated partitions should be searched, the original search partition determined based on the partition search parameter can be directly used as the second partition. For example, if the partition search parameter is the alcohol partition, the search only needs to be performed within the alcohol partition. If the partition search parameter is the aromatic partition, the search only needs to be performed within the aromatic partition.

[0125] In another specific embodiment, if the target search parameter represents a search for associated partitions, the corresponding original search partition is first located based on the partition search parameter. Then, an associated partition associated with the original search partition is determined, and the original search partition and the associated partition are combined as the second partition. For example, if the partition search parameter is the alcohols partition, since the alcohols partition is associated with the aromatics partition, the search needs to be performed in both the alcohols partition and the aromatics partition.

[0126] Furthermore, the above method also includes: if there are multiple target keywords, performing semantic analysis on each target keyword to determine whether there is a semantic similarity relationship between the target keywords based on the semantic analysis results; if there are multiple target keywords that satisfy the semantic similarity relationship, establishing an association relationship between the first partitions corresponding to the multiple target keywords. It is understandable that if there are multiple target keywords extracted, it is necessary to perform semantic analysis on each target keyword to determine whether there is a semantic similarity relationship between the target keywords. For example, based on the search content "This is an invention patent", the keywords therein are extracted to obtain the three keywords "invention", "patent", and "invention patent". After performing corresponding semantic analysis and other actions, basic partitions are determined, such as "invention", "patent", and "invention patent" as the three basic partitions. Based on the semantic analysis results, it can be determined that there is a semantic similarity relationship between patents and invention patents. Therefore, if there are multiple target keywords that satisfy the semantic similarity relationship, an association relationship is established between the first partitions corresponding to the multiple target keywords. For example, an association relationship is established between the partitions corresponding to patents and invention patents to facilitate subsequent related queries.

[0127] For example, aromatic chemicals include polycyclic aromatic hydrocarbons (PAHs) and alcohols. It's necessary to establish associations between aromatics and PAHs, and between aromatics and alcohols, respectively. This allows for simultaneous searches for PAHs and alcohols when searching for aromatics, and vice versa. Furthermore, for example, human partitions can be associated with behavioral features such as "holding," "grasping," and "working," forming a multidimensional classification model based on entity features (ResNet-50 extracts visual features, and BERT (Bidirectional Encoder Representations from Transformers, a pre-trained language model) extracts text features) and a storage structure.

[0128] Step S25: Filter out target documents with a document score higher than a score threshold from the second partition, and determine the document content from the target document to return the document content to the user end.

[0129] In this embodiment, after locating the second partition based on the target partition parameters, target documents with a document score above a score threshold are filtered from the second partition based on the target score parameters. As previously mentioned, each document has a corresponding document score calculated when it is stored in the partition, and the document score is dynamically updated. Therefore, this application can accurately filter target documents from the partition that meet user requirements based on the score threshold. Furthermore, the document content is determined from the target document and returned to the user.

[0130] In a specific implementation, the location of the underlying data storage can be located from partition → to node → to data shard → to data segment.

[0131] It should also be noted that the present application discloses an archiving model for processing document structure during underlying storage. Specifically, the process of storing the original document in the first partition also includes: extracting the global information of the original document to construct a main index; storing the original document in segments according to the document size, and constructing a sub-index for each segment; wherein each sub-index is an associated index of the main index. It is understandable that the present application can first construct a main index for the original document, which can specifically be the title, author, abstract, and other content of the original document; then, the original document can be stored in segments, and a sub-index can be constructed for each segment, where the sub-index is specifically used to represent the semantic content of each segment. In a specific embodiment, the original document storage can be specifically set according to the segment semantics, such as the default segment length of 50 characters, and the segment length exceeding 50 characters is split and stored to reduce the problem of a single document being too long and taking up too many requests during retrieval. It should be noted that the length of the segment splitting parameters can be adjusted independently based on the business characteristics of the system to achieve better retrieval results.

[0132] Specifically, the target partition parameters also include a document retrieval mode, which determines the document content from the target document, including: if the partition retrieval parameter represents a full document retrieval, then all the content in the target document is used as the document content; if the partition retrieval parameter does not represent a full document retrieval, then the partition retrieval parameter is matched with the main index and each sub-index respectively; if the partition retrieval parameter successfully matches the main index, then the global information corresponding to the main index is used as the document content; if the partition retrieval parameter successfully matches any sub-index, then the segment corresponding to any sub-index is used as the document content.

[0133] It is understandable that the user can choose whether to perform a full document search. When the partitioned search parameter indicates a full document search, the entire content of the target document is returned to the user end as the document content. If the partitioned search parameter does not indicate a full document search, the partitioned search parameter is matched with the main index and each sub-index respectively. If the partitioned search parameter successfully matches the main index, the global information corresponding to the main index is returned to the user end as the document content; if the partitioned search parameter successfully matches any sub-index, the segment corresponding to any sub-index is returned to the user end as the document content. That is, the original document is segmented and stored according to the corresponding archiving mode and document size, and points to the same document index. When the partitioned search parameter points to a sub-index in the split document, only the relevant document segment is returned to the application end. The full document is returned only when the full document search mode is searched, thereby avoiding the problem that the return of large documents takes up more request space, resulting in slow retrieval or even failure, thereby improving the performance and efficiency of the system.

[0134] That is, the archiving model is designed to process the document structure during the underlying storage. For example, when an article is saved as a document, because the article is too long, the performance during saving and subsequent use is very poor. Therefore, the document will be segmented after semantic analysis when saving, such as saving sub-indexes one by one. The segmented main index only saves the document's global information (such as title, author, abstract) and the positional relationship information of the sub-index. In this way, when querying, the main index will be queried first. Only when the entire content of the article needs to be queried, all the sub-index information spliced ​​into a complete article will be returned.

[0135] It is understandable that document storage specification optimization can be achieved through corresponding rules and methods, such as classifying documents based on their semantic segmentation features (such as type, semantics, word meaning, etc.), combining them with custom rule base matching, splitting and storing documents based on the segmentation results, and building sub-indexes. The sub-indexes are associated with the main index, and ultimately, during retrieval, partial or full content of the document can be returned based on the document's split structure relationship, avoiding the phenomenon of low query efficiency or query crash caused by returning all document content in a single search, thereby improving system stability and efficiency. For example:

[0136] A. Rule base matching: Document matching can be performed based on the default word segmenter semantic library. Custom rules can be defined and imported through the rule control module.

[0137] B. Semantic segmentation rules can be used to identify logical paragraphs in documents using NLP (natural language processing) models, such as BERT + CRF (conditional random fields) to identify technical clause boundaries.

[0138] C. The associated index can use the master index: storing global document information (such as title, author, abstract) and segment mapping table, the sub-index (Chunk Index): ① Keyword inverted index: recording keywords within the segment and their position weights. ② Vector index: storing chunk semantic vectors (generated by Sentence-BERT).

[0139] D. Index Search Strategy: On-demand search mechanism, such as ① Lightweight Search Mode: After a user enters a keyword, only matching segments and related context summaries are returned. Results are sorted by segment relevance and support paginated loading (≤ 5 segments per transfer). ② Full Document Mode: This requires configuration and authorization to trigger, concatenates the entire document in the order of segment identifiers, and uses streaming technologies (such as HTTP Chunked Encoding) to avoid memory overflows.

[0140] For a more specific processing procedure of the above step S21, reference may be made to the corresponding contents disclosed in the above embodiments, which will not be described again here.

[0141] It can be seen that this application speeds up and optimizes the indexing and retrieval process of the distributed retrieval system, so that the index can be intelligently matched and partitioned according to established rules, and the partitioned index results are extracted by targeted retrieval, thereby improving the retrieval performance efficiency and optimizing resource utilization. By combining a multi-dimensional scoring model, the document score is dynamically refreshed in combination with the original score value, the document click volume, and the time decay factor, the result relevance sorting accuracy is improved, and the retrieval efficiency and precision optimization are improved. The archiving model mechanism effectively avoids the risk of memory overflow when processing large documents through intelligent document segmentation and segmented storage, combined with streaming technology, to ensure stable and efficient retrieval performance under large documents. Targeted retrieval achieves scenario-based precise control while ensuring basic performance by supporting the priority of parameters carried in the retrieval request > the priority of user-side configuration parameters > the priority of default parameters, and optimizes retrieval efficiency with the overall indexing capability, which can improve the overall performance of the retrieval system. The overall process can enable the retrieval system to have better response performance and stability while ensuring retrieval accuracy in complex and multi-dimensional scenarios, and optimize system resource utilization.

[0142] See also Figure 3 As shown, an embodiment of the present application discloses a retrieval optimization architecture diagram in a retrieval system, which specifically includes four parts: a rule control module, an index aggregation controller, an archiving model, and a positioning retrieval.

[0143] Among them, the rule control module includes parameters and rule management for the overall aggregate index accelerator, such as the aggregate index accelerator switch, aggregation rule configuration, archiving model rules, scoring refresh rules, scoring thresholds, and search targeting words. For example, the enable switch of the aggregate index accelerator is used to set whether the current system enables the aggregate index accelerator. The aggregation rule configuration is used to control the rules for data classification and corresponding partitioning when the index is stored, so that the data can be indexed according to the rules and patterns required by the user. The archiving model rules are used to set the storage method of the underlying data. The scoring refresh rules and scoring thresholds are the scoring algorithm rules and threshold settings for the document data. The search targeting words are the user-side active input of targeting words during the search process to directly narrow the search scope and improve the search accuracy. These rules are mainly used for parameter and rule settings for the index establishment, underlying index shard storage, data retrieval and other process links in the overall effectiveness process of the aggregate index accelerator. It should be noted that in this embodiment, the aggregate index accelerator switch partition_switch=on / off can be manually defined. The default state is off and it will only take effect when manually set when index acceleration is required. That is, the search optimization process steps in the retrieval system disclosed in this application are only executed when the aggregate index accelerator switch is on.

[0144] The index aggregation controller is mainly used for index aggregation control and index data partitioning. After obtaining the aggregation controller switch and aggregation rules from the rule control module, it creates and refreshes the overall aggregate index, and performs corresponding partitioning on the stored documents. Specifically, the index aggregation controller mainly associates partition index features during the overall index aggregation, index partitioning, and index scoring processes during index processing. That is, by associating data links between partitions, for example, human partitions can be associated with behavioral features such as holding, grabbing, and working. This allows for faster retrieval through associated searches. Document storage achieves partition control through segment pointing processing and records the initial document scores.

[0145] The archiving model is designed to process the document structure during the underlying storage process. Its purpose is to avoid directly returning large documents and taking up too much of the response request load, thereby improving system performance and efficiency.

[0146] Targeted retrieval means that when the application sends a retrieval request, the retrieval system can perform document relevance and word meaning classification analysis based on the document word meaning of the retrieval content, generate specific target partition parameters and target scoring parameters, and then search the index content of the partition based on the specific word meaning content and score. The application is allowed to customize the partition parameters and scoring thresholds during the retrieval, so as to retrieve the desired results more accurately and quickly.

[0147] See also Figure 4As shown, the embodiment of the present application discloses a search optimization device in a search system, the device comprising:

[0148] The keyword extraction module 11 is used to obtain the original document and extract the target keywords from the original document;

[0149] The document storage module 12 is configured to match the target keyword with the current partitions to determine a first partition that matches the target keyword, store the original document in the first partition, and calculate a document score corresponding to the original document; wherein the document score is dynamically updated according to preset rules;

[0150] The parameter determination module 13 is used to determine the target partition parameter and the target scoring parameter based on the search request when obtaining the search request from the user terminal;

[0151] The document retrieval module 14 is configured to locate the second partition based on the target partition parameter, and filter out matching document content from the second partition based on the target scoring parameter, so as to return the document content to the user terminal.

[0152] As can be seen, after obtaining the original document, the present application first needs to extract the target keywords from the original document, and the present application has pre-built multiple partitions. After extracting the target keywords, the target keywords are matched with the current partitions to determine the first partition that matches the target keywords, and the original document is stored in the first partition. In other words, the present application stores the original documents by matching keywords with partitions, realizing centralized storage of documents according to semantic classification, that is, it can achieve a strong correlation between the content semantics of documents stored in the same partition. In addition, while storing the original document, the document score corresponding to the original document is also calculated, and the document score is dynamically updated according to preset rules over time. Furthermore, when a search request from the user end is obtained, it is first necessary to determine the target partition parameters and target score parameters based on the search request. In this way, the corresponding second partition can be quickly located based on the target partition parameters and target score parameters in a targeted search method, and the matching document content can be extracted from the second partition and returned, thereby improving the search performance. The above solution can ensure that the retrieval system has better response performance and stability while ensuring retrieval accuracy.

[0153] Since the embodiments of the device part correspond to the above embodiments, the embodiments of the device part please refer to the description of the embodiments of the method part, which will not be repeated here.

[0154] Figure 5This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Specifically, the device may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps of the search optimization method in the search system performed by the electronic device as disclosed in any of the aforementioned embodiments.

[0155] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0156] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0157] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon include an operating system 221, a computer program 222 and data 223, etc. The storage method can be temporary storage or permanent storage.

[0158] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, so as to enable the processor 21 to calculate and process the massive amount of data 223 in the memory 22. It can be Windows, Unix, Linux, etc. In addition to including computer programs that can be used to implement the search optimization method in the search system executed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include computer programs that can be used to perform other specific tasks. In addition to data received by the electronic device and transmitted from external devices, the data 223 can also include data collected by its own input and output interface 25.

[0159] Furthermore, an embodiment of the present application also discloses a computer-readable storage medium, in which a computer program is stored. When the computer program is loaded and executed by a processor, the steps of the retrieval optimization method in the retrieval system disclosed in any of the aforementioned embodiments are implemented.

[0160] An embodiment of the present invention further discloses a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the search optimization method in the search system disclosed in any of the aforementioned embodiments.

[0161] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0162] Those skilled in the art may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the composition and steps of each example according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0163] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a compact disc read-only memory (CD-ROM), or any other form of storage medium known in the art.

[0164] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0165] The above is a detailed introduction to the retrieval optimization method, device, equipment, medium and product in a retrieval system provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A search optimization method in a search system, characterized in that: include: Obtaining an original document and extracting target keywords from the original document; Matching the target keyword with each current partition to determine a first partition that matches the target keyword, storing the original document in the first partition, and calculating a document score corresponding to the original document; wherein the document score is dynamically updated according to a preset rule; When a search request is received from a user, a target partition parameter and a target scoring parameter are determined based on the search request; Locating a second partition based on the target partition parameter, and filtering matching document content from the second partition based on the target scoring parameter, so as to return the document content to the user terminal; Wherein, the target partition parameters include partition search parameters and target search parameters indicating whether to search for associated partitions; Accordingly, locating the second partition based on the target partition parameter includes: If the target search parameter indicates that the associated partition is not to be searched, determining the corresponding original search partition based on the partition search parameter, and using the original search partition as the second partition; If the target search parameter represents searching for an associated partition, the corresponding original search partition is located based on the partition search parameter, and an associated partition associated with the original search partition is determined, so that the original search partition and the associated partition are used as the second partition.

2. The search optimization method in the search system according to claim 1, characterized in that: The extracting target keywords from the original document includes: A preset word segmenter is used to extract corresponding keywords from the original document and use them as target keywords; wherein, the preset word segmenter is constructed based on a first word segmenter and a second word segmenter; wherein, the first word segmenter is constructed based on a general word segmentation library, and the second word segmenter is constructed based on a custom word segmentation library set by the user end.

3. The search optimization method in the search system according to claim 2, characterized in that: The extracting target keywords from the original document includes: Segmenting the original document to obtain a plurality of segment fragments; Calculating the semantic density of each of the slice segments, and selecting target slice segments having a semantic density greater than a preset density threshold from the slice segments; The preset word segmenter is used to extract corresponding keywords from the target segment segments and use them as target keywords.

4. The search optimization method in the search system according to claim 3, characterized in that: The calculating of the semantic density of each of the slice segments includes: For any segment segment, calculate the word frequency-inverse document frequency of each word in the segment segment, and sum the word frequency-inverse document frequency to obtain a sum result; The semantic density of any one of the slice segments is determined based on the summation result and the slice length of any one of the slice segments.

5. The search optimization method in the search system according to claim 1, characterized in that: The step of matching the target keyword with each current partition to determine a first partition matching the target keyword includes: Obtaining keyword lists corresponding to each current partition, and determining whether each keyword list includes the target keyword; If there is a target keyword list including the target keyword, the partition corresponding to the target keyword list is used as the first partition matching the target keyword.

6. The search optimization method in the search system according to claim 1, characterized in that: The step of matching the target keyword with each current partition to determine a first partition matching the target keyword includes: Obtaining regular expressions corresponding to each current partition; wherein the regular expressions are constructed based on text features and target characters; Calculating the similarity between the target keyword and each of the regular expressions respectively; If the similarity between any regular expression and the target keyword is greater than a preset similarity threshold, the partition corresponding to the regular expression is used as the first partition matching the target keyword.

7. The search optimization method in the search system according to claim 1, characterized in that: The process of matching the target keyword with each current partition to determine the first partition matching the target keyword includes: If no partition matching the target keyword exists in the current partitions, a new partition is created based on the target keyword, and the new partition is used as the first partition matching the target keyword.

8. The search optimization method in a search system according to claim 1, characterized in that: Calculating the document score corresponding to the original document includes: The original document is scored based on a preset multidimensional scoring model to obtain a corresponding document score; wherein, the multidimensional scoring model is constructed based on the original score value, the document click volume, the time decay factor and the corresponding weight coefficient, the original score value is the score value obtained by scoring using the internal scoring mechanism of the distributed retrieval system, the document click volume is the cumulative click volume of the original document, and the time decay factor is calculated based on a preset time decay formula.

9. The search optimization method in the search system according to claim 8, characterized in that: Dynamically updating the document score based on preset rules includes: Get the latest cumulative click volume and the latest time decay factor; The score of the original document is recalculated based on the original score value, the latest cumulative click volume, the latest time decay factor and the corresponding weight coefficient to update the document score.

10. The search optimization method in the search system according to claim 9, characterized in that: The obtaining of the latest cumulative click volume and the latest time decay factor includes: Obtain the latest cumulative click volume and the latest time decay factor at every preset time interval; Alternatively, when the current load of the distributed retrieval system is lower than a preset load threshold and the current time is within a preset time range, the latest cumulative click volume and the latest time decay factor are obtained.

11. The search optimization method in a search system according to claim 1, characterized in that: Also includes: If there are multiple target keywords, semantic analysis is performed on each target keyword to determine whether there is a semantic similarity relationship between the target keywords based on the semantic analysis result; If there are multiple target keywords that satisfy the semantic similarity relationship, the association relationship is established between the first partitions corresponding to the multiple target keywords.

12. The search optimization method in a search system according to claim 1, characterized in that: The determining of target partition parameters and target scoring parameters based on the search request includes: Determining whether the search request carries a partition parameter and a scoring parameter; If so, the partition parameter and the scoring parameter carried in the search request are used as the target partition parameter and the target scoring parameter respectively.

13. The search optimization method in the search system according to claim 12, characterized in that: After determining whether the search request carries the partition parameter and the scoring parameter, the method further includes: If the search request does not carry partition parameters and scoring parameters, performing word segmentation processing and semantic analysis on the search content in the search request to obtain partition search parameters; Querying the preset configuration file to determine whether the target search parameters and scoring parameters pre-configured by the user terminal exist; If so, obtaining target partition parameters based on the target search parameters pre-configured by the user terminal and the partition search parameters determined using the search content, and using the scoring parameters pre-configured by the user terminal as target scoring parameters; If not, the target partition parameters are obtained based on the default search parameters stored in the preset configuration file and the partition search parameters determined using the search content, and the default scoring parameters stored in the preset configuration file are used as the target scoring parameters.

14. The search optimization method in a search system according to claim 1, characterized in that: The target scoring parameters include a scoring threshold; Accordingly, filtering out matching document content from the second partition based on the target scoring parameter includes: Target documents having a document score higher than the score threshold are screened out from the second partition, and document content is determined from the target documents.

15. The search optimization method in the search system according to claim 14, characterized in that: The process of storing the original document in the first partition further includes: Extracting global information of the original document to construct a primary index; The original document is stored in segments according to the document size, and a sub-index is constructed for each segment; wherein each sub-index is an associated index of the main index.

16. The search optimization method in the search system according to claim 15, characterized in that: The target partition parameter also includes a document retrieval mode, and determining the document content from the target document includes: If the partitioned search parameter indicates a full document search, then all the contents of the target document are used as the document content; If the partitioned search parameter does not represent a full document search, matching the partitioned search parameter with the main index and each of the sub-indexes respectively; If the partition search parameter successfully matches the primary index, the global information corresponding to the primary index is used as the document content; If the partition search parameter successfully matches any sub-index, the segment corresponding to the sub-index is used as the document content.

17. A search optimization device in a search system, characterized in that: include: A keyword extraction module, configured to obtain an original document and extract target keywords from the original document; a document storage module, configured to match the target keyword with each current partition to determine a first partition that matches the target keyword, store the original document in the first partition, and calculate a document score corresponding to the original document; wherein the document score is dynamically updated according to preset rules; A parameter determination module, configured to determine target partition parameters and target score parameters based on a search request received from a user terminal; a document retrieval module, configured to locate a second partition based on the target partition parameter, and filter out matching document content from the second partition based on the target scoring parameter, so as to return the document content to the user terminal; Wherein, the target partition parameters include partition search parameters and target search parameters indicating whether to search for associated partitions; Correspondingly, the document retrieval module is specifically used to determine the corresponding original retrieval partition based on the partition retrieval parameters if the target search parameter indicates not to search for the associated partition, and use the original retrieval partition as the second partition; if the target search parameter indicates to search for the associated partition, locate the corresponding original retrieval partition based on the partition retrieval parameters, and determine the associated partition that has an associated relationship with the original retrieval partition, so as to use the original retrieval partition and the associated partition as the second partition.

18. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the retrieval optimization method in the retrieval system according to any one of claims 1 to 16.

19. A computer-readable storage medium, characterized in that Used to store computer programs; wherein, when the computer program is executed by a processor, the steps of the retrieval optimization method in the retrieval system according to any one of claims 1 to 16 are implemented.

20. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the search optimization method in the search system according to any one of claims 1 to 16 are implemented.

Images