File Quick Retrieval Method, Product, Computer Device, and Storage Medium

Abstract

The present application relates to a method, product, computer device and storage medium for fast file retrieval. The method includes: vectorizing the text content in the original document to generate a document vector; clustering and grouping the original documents according to the similarity between the document vectors to form multiple document clusters; setting the dynamic priorities of the original documents in each document cluster according to the update frequency and / or query frequency of the original documents in the document cluster; establishing an index structure for the document vectors, and adjusting the sorting of the document vectors in the index structure according to the dynamic priorities of the original documents; when a user initiates a query, converting the user's query into a query vector, and obtaining the original documents related to the user's query by calculating the similarity between the query vector and the document vectors in the sorted index structure of the document vectors. It can effectively improve the retrieval efficiency, quickly find the documents with high dynamic priorities, and reduce the system response delay or resource consumption.

Description

Technical Field

[0001] The present application relates to the technical field of servers, and particularly to a method for quickly retrieving files, a product, a computer device, and a storage medium. Background Art

[0002] Existing file retrieval technologies mainly rely on traditional algorithms such as inverted indexes and keyword matching, combined with natural language processing (NLP) technologies, to achieve the matching of documents and queries through vectorized representations. At the same time, based on the feedback mechanism of user behavior and machine learning technologies, existing systems can optimize document sorting through indicators such as click-through rate and access frequency. However, as the data scale continues to increase, existing technologies often need to rely on a distributed architecture for processing massive data, and in the case of high concurrency, resource scheduling and load balancing become key challenges. Summary of the Invention

[0003] Based on this, a method, device, computer device, and storage medium for quickly retrieving files are provided, aiming to solve the technical problems that in a traditional file retrieval system, the index construction and retrieval efficiency gradually decrease when processing a large number of documents, and especially in the case of high concurrency or high load, system response latency or resource exhaustion is likely to occur.

[0004] On the one hand, a method for quickly retrieving files is provided, and the method includes:

[0005] Performing vectorized representation on the text content in the original document to generate a document vector;

[0006] Clustering and grouping the original documents according to the similarity between the document vectors to form a plurality of document clusters;

[0007] Setting dynamic priorities for the original documents in each document cluster according to the update frequency and / or query frequency of the original documents in the document cluster;

[0008] Establishing an index structure for the document vectors, and adjusting the sorting of the document vectors in the index structure according to the dynamic priorities of the original documents;

[0009] In response to a user's query, converting the user's query into a query vector, and obtaining the original documents related to the user's query by calculating the similarity between the query vector and the document vectors in the sorted index structure of the document vectors.

[0010] In one embodiment, the performing vectorized representation on the text content in the original document to generate a document vector includes:

[0011] Obtain the weighted value of each target word in the original document according to the word frequency of the target word in the target document, the total number of documents in the original document, and the number of documents containing the target word, where the target word is a word in the original document and the target document is a document in the original document;

[0012] Obtain the weighted value of each target word in each target document in the original document and convert it into a vector representation to form the document vector of the target document.

[0013] In one embodiment, the setting of the dynamic priority of the original documents in each document cluster according to the update frequency and / or query frequency of the original documents in the document cluster includes:

[0014] Obtain the update frequency and query frequency of the original document, and assign dynamic priority to the original document by weighted summation of the update frequency and the query frequency.

[0015] In one embodiment, when responding to a user's query, converting the user's query into a query vector and obtaining the original document related to the user's query by calculating the similarity between the query vector and the document vectors in the sorted index structure of the document vectors includes:

[0016] Obtain the user's query and convert the user's query into a query vector;

[0017] Obtain the document vectors in the index structure, and calculate the similarity between the query vector and the document vectors in the sorted index structure of the document vectors by cosine similarity to measure the relevance between the user's query and the original document.

[0018] In one embodiment, the file fast retrieval method further includes:

[0019] Obtain the ideal sorting after the user's click feedback, and adjust the sorting of the document vectors in the index structure in real time by minimizing the error of the document ranking.

[0020] In one embodiment, the file fast retrieval method further includes:

[0021] Obtain the total computing resources for executing the index structure and the computing resources required for executing each retrieval task, and set the sum of the computing resources consumed by executing the retrieval tasks to be less than or equal to the total computing resources for executing the index structure;

[0022] Obtain the remaining computing resources;

[0023] When the remaining computing resources are less than or equal to the first threshold, set the priority weight of the corresponding retrieval task according to the dynamic priority of the original document, and allocate computing resources in descending order of the priority weight of the retrieval task;

[0024] In response to the remaining computing resources being greater than the first threshold, the retrieval tasks whose priority weights are less than the second threshold are deleted to obtain the computing resources remaining after trimming the low-priority tasks, so that the retrieval tasks whose priority weights are greater than or equal to the second threshold are allocated to the computing resources remaining after trimming the low-priority tasks.

[0025] In one embodiment, deleting the search tasks whose priority weight is less than the second threshold to obtain the remaining computing resources after trimming the low-priority tasks includes:

[0026] Obtain computing resources corresponding to retrieval tasks whose priority weights are less than a second threshold;

[0027] The low priority tasks are trimmed by subtracting the computing resources corresponding to the retrieval tasks whose priority weights are less than the second threshold from the total computing resources of the executing index structure, thereby obtaining the remaining computing resources after trimming the low priority tasks.

[0028] In one embodiment, the method for rapid file retrieval further includes:

[0029] Get the average query time T for retrieving documents query ;

[0030] Get the document d that the user clicked i The probability P(c i ),in rel(d i ) represents the target document d i The correlation score of , α is the weight coefficient for adjusting the update frequency;

[0031] Get the total computing resources R and resource utilization R for executing the index structure used ;

[0032] The index structure is optimized by maximizing the system performance index M. The maximizing system performance index M is: Where α1 is the query frequency weight parameter, and α2 is the call rate weight parameter.

[0033] In one embodiment, the method for rapid file retrieval further includes:

[0034] Setting the update frequency of the index in the index structure to be positively correlated with the dynamic priority of the original document corresponding to the index;

[0035] The priority threshold for index updates is obtained by dividing the document weight by the current node load.

[0036] Performing index update on original documents whose dynamic priority is greater than the priority threshold of index update;

[0037] The update constraint condition of the index in the index structure is where P(d i ) > θ, T update (d i ) is the index update time, T max is the total update time, n is the number of retrieval tasks, and θ is the priority threshold for index update.

[0038] In one embodiment, the file fast retrieval method further includes:

[0039] Obtaining the click-through rate P(a i ) of the document by dividing the number of times the document is clicked by the number of times the document is displayed;

[0040] Obtaining the priority threshold θ for index update, and when P(a i ) < θ and / or when , adjusting the sorting of the documents in the index structure; where N is the total number of documents.

[0041] In one embodiment, the file fast retrieval method further includes:

[0042] Setting multiple nodes in the distributed file retrieval system to jointly execute the document retrieval task for the index structure;

[0043] Adjusting the load of the target node by minimizing the load variance.

[0044] In one embodiment, the file fast retrieval method further includes:

[0045] Monitoring the load of the nodes in the distributed file retrieval system;

[0046] In response to the load of the target node being greater than the maximum load threshold, reallocating the retrieval task and the load of the target node after the task reallocation.

[0047] On the other hand, a computer program product is provided, including a computer program, which when executed by a processor implements the following steps:

[0048] Performing vectorization representation on the text content in the original document to generate a document vector;

[0049] Clustering and grouping the original documents according to the similarity between the document vectors to form multiple document clusters;

[0050] Setting the dynamic priority of the original documents in each document cluster according to the update frequency and / or query frequency of the original documents in the document cluster;

[0051] An index structure is established for the document vectors, and the sorting of the document vectors in the index structure is adjusted according to the dynamic priority of the original document;

[0052] In response to a user's query, the user's query is converted into a query vector, and the original documents related to the user's query are obtained by calculating the similarity between the query vector and the document vectors in the index structure after sorting the document vectors.

[0053] On the other hand, a computer device is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented:

[0054] The text content in the original document is vectorized to generate document vectors;

[0055] The original documents are clustered and grouped into multiple document clusters according to the similarity between the document vectors;

[0056] The dynamic priorities of the original documents in each document cluster are set according to the update frequency and / or query frequency of the original documents in the document cluster;

[0057] An index structure is established for the document vectors, and the sorting of the document vectors in the index structure is adjusted according to the dynamic priority of the original document;

[0058] In response to a user's query, the user's query is converted into a query vector, and the original documents related to the user's query are obtained by calculating the similarity between the query vector and the document vectors in the index structure after sorting the document vectors.

[0059] On another hand, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0060] The text content in the original document is vectorized to generate document vectors;

[0061] The original documents are clustered and grouped into multiple document clusters according to the similarity between the document vectors;

[0062] The dynamic priorities of the original documents in each document cluster are set according to the update frequency and / or query frequency of the original documents in the document cluster;

[0063] An index structure is established for the document vectors, and the sorting of the document vectors in the index structure is adjusted according to the dynamic priority of the original document;

[0064] When a user initiates a query, the user's query is converted into a query vector, and the original documents relevant to the user's query are obtained by calculating the similarity between the query vector and the document vectors in the sorted index structure of the document vectors.

[0065] The above file fast retrieval method, product, computer device and storage medium generate document vectors by vectorizing the text content in the original documents, then cluster to form multiple document clusters, and set dynamic priorities for the original documents in each document cluster to adjust the document sorting of the index structure, which can effectively improve the retrieval efficiency. In the case of high concurrency or high load, it can quickly find documents with high dynamic priorities, reducing system response latency or resource consumption. Brief Description of the Drawings

[0066] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0067] Figure 1 It is an application environment diagram of the file fast retrieval method in an embodiment of the present application;

[0068] Figure 2 It is a flowchart of the file fast retrieval method in an embodiment of the present application;

[0069] Figure 3 It is a structural block diagram of the file fast retrieval device in an embodiment of the present application;

[0070] Figure 4 It is an internal structure diagram of a computer device in an embodiment of the present application. Detailed Description of the Embodiments

[0071] In order to make the purpose, technical solutions and advantages of the present application more clear, the following further details the present application in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0072] As described in the background art, with the acceleration of the digitalization process, the massive data generated by various industries has put forward higher requirements for file retrieval technology. Whether in the fields of e-commerce, healthcare, finance, etc., or in applications such as academic research and social media, users' demand for fast, efficient, and accurate file retrieval is increasing day by day. File retrieval technology not only needs to process a large amount of data but also deal with diverse and unstructured data sources such as text, images, audio, etc. To ensure that users can quickly obtain the required information, the retrieval system must have an efficient indexing and sorting mechanism, be able to adapt to the dynamically changing data environment, and continuously optimize the user experience. This makes building a file retrieval technology solution with intelligent optimization and dynamic adjustment capabilities a key research direction in the current information technology field.

[0073] Existing file retrieval technologies mainly rely on traditional algorithms such as inverted indexing and keyword matching, combined with natural language processing (NLP) technology, to achieve the matching of documents and queries through vectorized representation. At the same time, based on the feedback mechanism of user behavior and machine learning technology, existing systems can optimize document sorting through indicators such as click-through rate and access frequency. However, as the data scale continues to increase, existing technologies often need to rely on a distributed architecture to process massive data, and in the case of high concurrency, resource scheduling and load balancing become key challenges. Some modern systems have introduced deep learning models to enhance the accuracy of file retrieval through semantic understanding and context association, providing a more intelligent search experience.

[0074] Although existing technologies have made significant progress in the field of file retrieval, there are still many deficiencies in dealing with dynamic data, coping with resource-constrained or high-load environments. When faced with complex queries and large-scale data, existing systems are difficult to balance the timeliness of index updates and the accuracy of retrieval results. In addition, existing load balancing strategies often handle node overloads in a timely manner, resulting in system performance bottlenecks. The sorting optimization mechanism based on user feedback also lacks sufficient flexibility and is difficult to dynamically adjust the model to adapt to the changing user needs.

[0075] To solve the above problems, a file fast retrieval method is creatively proposed in the embodiments of the present invention. The improvement goal is to optimize the retrieval and sorting process of documents by introducing intelligent weight regulation, task reallocation, and dynamic resource adjustment mechanisms. By combining a logical judgment mechanism with a mathematical model, the system can not only adapt to resource limitations in different scenarios but also prioritize the completion of important tasks in a high-load environment, improving the overall system response speed and user experience.

[0076] The file fast retrieval method provided in this application can be applied to such as Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. Among them, the terminal 102 can be but is not limited to various personal computers, laptop computers, smart phones, tablet computers, and portable wearable devices, and the server 104 can be implemented by an independent server or a server cluster composed of multiple servers. The server 104 provides a file quick retrieval service, and the terminal 102 initiates a query to find the original document.

[0077] In one embodiment, as Figure 2 shown, a file quick retrieval method is provided. Taking the server 104 in Figure 1 as an example, the method includes the following steps:

[0078] Step S1, vectorize the text content in the original document to generate a document vector;

[0079] Step S2, cluster and group the original documents according to the similarity between the document vectors to form multiple document clusters;

[0080] Step S3, set the dynamic priority of the original documents in each document cluster according to the update frequency and / or query frequency of the original documents in the document cluster;

[0081] Step S4, establish an index structure for the document vectors, and adjust the sorting of the document vectors in the index structure according to the dynamic priority of the original documents;

[0082] Step S5, when responding to a user's query, convert the user's query into a query vector, and obtain the original documents related to the user's query by calculating the similarity between the query vector and the document vectors in the sorted index structure of the document vectors.

[0083] Among them, by vectorizing the text content in the original document to generate a document vector, then clustering to form multiple document clusters, and setting the dynamic priority for the original documents in each document cluster to adjust the document sorting of the index structure, the retrieval efficiency can be effectively improved. In the case of high concurrency or high load, documents with high dynamic priority can be quickly found, reducing system response latency or resource consumption.

[0084] In one of the embodiments, vectorizing the text content in the original document to generate a document vector includes:

[0085] Obtain the weighted value of each target word in the original document according to the word frequency of the target word in the target document, the total number of documents in the original document, and the number of documents containing the target word. The target word is a word in the original document, and the target document is a document in the original document;

[0086] Obtain the weighted value of each word in the original document and convert it into a vector representation to form a document vector.

[0087] Specifically, the TF-IDF algorithm is used to weight each word in the original document. The weighting method is: Where TF(t,d) is the frequency of word t in document d, N is the total number of documents, |{d:t∈d}| is the number of documents containing word t; d =(w(t1,d),w(t2,d),…,w(t n ,d)) Convert each original document into a vector representation and generate a document vector v d .

[0088] In one embodiment, clustering original documents to form multiple document clusters based on similarities between document vectors includes:

[0089] The K-means clustering algorithm is used to cluster the original documents according to the similarity between the document vectors. The goal of clustering the original documents is to minimize the size of each document vector v d Distance to the cluster center Where S represents the document cluster, S i represents the i-th document cluster, μ i is the center of the cluster.

[0090] After clustering, documents are classified by similarity, providing a foundation for subsequent indexing and query optimization. Of course, when document vectors cannot be well classified or system resources are limited, the system will adjust the processing based on the document weight and clustering results, prioritizing frequently accessed or more relevant documents.

[0091] Specifically, in a document retrieval system, documents are first vectorized for clustering and indexing. However, in some cases, document vectors may not be well classified, or system resources may be insufficient to process all documents. Therefore, the system prioritizes frequently accessed documents or documents highly relevant to the query based on their weights and clustering results.

[0092] In one embodiment, setting the dynamic priority of the original documents in each document cluster according to the update frequency and / or query frequency of the original documents in the document cluster includes:

[0093] The update frequency and query frequency of the original document are obtained, and a dynamic priority is assigned to the original document by weighted summation of the update frequency and the query frequency.

[0094] Specifically, obtain the update frequency μ of the original document i and query frequency q i, a dynamic priority P(d i ) is assigned to the original document, where P(d i ) = αμ i +βq i . α is a weight coefficient for adjusting the update frequency, and β is a weight coefficient for adjusting the influence of the query frequency.

[0095] In one embodiment, when a user initiates a query, the user's query is converted into a query vector, and the original documents relevant to the user's query are obtained by calculating the similarity between the query vector and the document vectors in the sorted index structure of the document vectors, including:

[0096] Obtain the user's initiated query and convert the user's query into a query vector;

[0097] Obtain the document vectors in the index structure, and calculate the similarity between the query vector and the document vectors in the sorted index structure of the document vectors by the cosine similarity method to measure the relevance between the user's query and the original documents.

[0098] Specifically, the similarity between the query vector v and the document vector v Q in the index structure is calculated by the cosine similarity d to measure the relevance between the user's query and the original documents.

[0099] Among them, the cosine similarity can accurately measure the similarity between the user's query and the document. The higher the similarity of the document, the retrieval system will return it first.

[0100] In one embodiment, the file fast retrieval method further includes:

[0101] Obtain the ideal sorting after the user's click feedback, and adjust the sorting of the document vectors in the index structure in real time by minimizing the error of the document ranking.

[0102] Specifically, the sorting of the document vectors in the index structure is adjusted in real time by minimizing the error of the document ranking ; where r i is the current sorting, is the ideal sorting after the user's click feedback, and n is the number of retrieval tasks.

[0103] In one embodiment, the file fast retrieval method further includes:

[0104] Obtain the total computing resources for executing the index structure and the computing resources required for each retrieval task, and set the sum of the computing resources consumed for executing the retrieval tasks to be less than or equal to the total computing resources for executing the index structure;

[0105] Obtain the remaining computing resources;

[0106] When the remaining computing resources are less than or equal to the first threshold, set the priority weight of the corresponding retrieval task according to the dynamic priority of the original document, and allocate computing resources in descending order of the priority weight of the retrieval task;

[0107] When the remaining computing resources are greater than the first threshold, delete the retrieval tasks with priority weights less than the second threshold to obtain the remaining computing resources after trimming low-priority tasks, so that the retrieval tasks with priority weights greater than or equal to the second threshold are allocated the remaining computing resources after trimming low-priority tasks.

[0108] Specifically, set the total computing resources for executing the index structure as R, and the computing resources required for executing each retrieval task Q i is R i , set the computing resource allocation to satisfy where n is the number of retrieval tasks;

[0109] Obtain the remaining computing resources

[0110] When the remaining computing resources are less than or equal to the first threshold, set the priority weight p of the corresponding retrieval task Q according to the dynamic priority P(d i ) i , and allocate computing resources in descending order of the priority weight p of the retrieval task Q i , according to the retrieval task Q i 's priority weight p i ;

[0111] When the remaining computing resources are greater than the first threshold, delete the retrieval tasks with priority weight p i less than the second threshold to obtain the remaining computing resources after trimming low-priority tasks, so that the retrieval tasks with priority weight p i greater than or equal to the second threshold are allocated the remaining computing resources after trimming low-priority tasks.

[0112] It can be understood that n is the number of retrieval tasks executed, corresponding to the total resources consumed for executing the retrieval tasks. In the case of limited resources, the system will give priority to processing those retrieval tasks with higher priorities. By optimizing resource allocation through linear programming, it can be ensured that higher-priority tasks are allocated more resources: Among them, p i represents the priority weight of the task. The higher the priority of the task, the more computing resources will be allocated to ensure a faster response speed. When the resources are exhausted, the system will trim low-priority tasks to ensure that high-priority tasks can be executed smoothly under limited resources. Among them, the priority weight p of the task i and the dynamic priority P(d of the original documenti ) is positively correlated, and p can be preferably set i = P(d i ).

[0113] In one embodiment, the computing resources remaining after obtaining the cropped low-priority tasks by deleting the retrieval tasks with a priority weight less than the second threshold include:

[0114] Obtain the computing resources corresponding to the retrieval tasks with a priority weight less than the second threshold;

[0115] By performing the total computing resources of the index structure minus the computing resources corresponding to the retrieval tasks with a priority weight less than the second threshold to crop the low-priority tasks, obtain the computing resources remaining after cropping the low-priority tasks.

[0116] Specifically, obtain the computing resources corresponding to the retrieval tasks with a priority weight p i less than the second threshold where N low is the total computing resources corresponding to the retrieval tasks with a priority weight p i less than the second threshold, and R low,i is the computing resources corresponding to each retrieval task with a priority weight p i less than the second threshold;

[0117] Through Obtain the computing resources remaining after cropping the low-priority tasks.

[0118] Among them, when the resources are exhausted, the system will crop the low-priority tasks to ensure that the high-priority tasks can be successfully executed under limited resources.

[0119] Specifically, in the case of extremely tight resources, the system may not be able to complete all tasks. At this time, the system will crop the low-priority tasks to ensure that the high-priority tasks can be successfully executed.

[0120] Assume that the system resources (such as CPU, memory) are exhausted and cannot process all query tasks at the same time. The system crops the low-priority tasks to ensure that the high-priority tasks are processed. The task cropping strategy can be achieved through so that the system can ensure that the critical tasks can still be successfully completed when the resources are limited.

[0121] In other embodiments, in order to avoid the problems of low retrieval efficiency and insufficient computing resources caused by too many document vectors in the index structure. Among them, the file fast retrieval method further includes:

[0122] The index structure is divided into a first sub-search structure and a second sub-search structure according to dynamic priorities. The dynamic priority of the original document corresponding to the document vector in the first sub-search structure is greater than that of the original document corresponding to the document vector in the second sub-search structure;

[0123] In response to a user initiating a query, retrieve and obtain the original documents related to the user's query in the first sub-search structure and the second sub-search structure in sequence;

[0124] When retrieving and obtaining the original documents related to the user's query in the first sub-search structure, control is not to allocate computing resources to the second sub-search structure; when retrieving and obtaining the original documents related to the user's query in the second sub-search structure, control is not to allocate computing resources to the first sub-search structure.

[0125] Among them, by retrieving and obtaining the original documents related to the user's query in the first sub-search structure and the second sub-search structure in sequence, documents with high access rates can be quickly obtained, and effectively allocating computing resources can ensure sufficient computing resources for the retrieval.

[0126] In one embodiment, the file fast retrieval method further includes:

[0127] Obtain the average query duration T of the retrieved documents query ;

[0128] Obtain the probability P(c i ) that the user clicks on the document d i ), where rel(d i ) represents the relevance score of the target document d i , and α is a weight coefficient for adjusting the update frequency;

[0129] Obtain the total computing resources R and resource utilization rate R of the execution index structure used ;

[0130] Optimize the index structure by maximizing the system performance metric M. The way to maximize the system performance metric M is where α1 is the query frequency weight parameter and α2 is the call rate weight parameter.

[0131] The function exp(x) represents the natural exponential function, that is, the exponential function with the real number e (e≈2.71828) as the base. α is a weight coefficient for adjusting the update frequency and is a dynamically changing value; α1 and α2 are the values of α in different situations.

[0132] Periodically retrain the model and optimize the index structure. The specific iterative optimization objective is to maximize the system performance metric M through the formula. T queryis the average query time of the retrieved documents, which the system expects to minimize; P(c i ) is the probability that the user clicks on the document d i . The system improves the relevance of the document through user behavior; R used is the resource utilization rate of the system, and resource allocation is optimized to improve computing efficiency.

[0133] The optimization process is dynamic and adjusts the system parameters in real time according to the user query pattern, system load, and introduction of new documents to ensure the stability and continuous improvement of the retrieval effect. When the user feedback (such as click-through rate) is continuously unsatisfactory, the system automatically initiates model retraining and adjusts the document sorting rules; when the system resource occupancy rate is too high, the thresholds for index update and query scheduling are automatically adjusted to give priority to the execution of important tasks.

[0134] In one of the embodiments, the file fast retrieval method further includes:

[0135] Set the update frequency of the index in the index structure to be positively correlated with the dynamic priority of the original document corresponding to the index;

[0136] Obtain the priority threshold for index update by dividing the document weight by the load of the current node;

[0137] Update the index for the original document whose dynamic priority is greater than the priority threshold for index update;

[0138] The update constraint condition for the index in the index structure is where P(d i ) > θ, T update (d i ) is the index update time, T max is the total update time, n is the number of retrieval tasks, and θ is the priority threshold for index update.

[0139] Specifically, obtain the priority threshold θ for index update through , where W i is the document weight and L is the load of the current node. When the system load L increases, only the documents with weight W i higher than θ will be preferentially updated for the index to ensure the retrieval performance of key documents.

[0140] Among them, assuming that the system has a large number of documents to process but limited resources (such as the memory or CPU usage rate is already close to the upper limit), the system will face the situation of being unable to process all documents simultaneously. At this time, the system will assign weights W i to the documents, and the weights can be determined according to the following factors: the access frequency F i of the document, the importance I i (according to user feedback or business rules). The document weight formula is: Wi = λ1F i + λ2I i . λ1 is the weight of document access frequency, and λ2 is the weight of document importance. When resources are scarce, the system will select the document with the highest weight for priority processing. Through weight sorting, those documents frequently accessed by users are preferentially optimized for indexing and classified, so that even when resources are limited, users can still retrieve important documents in a timely manner.

[0141] Among them, the higher the dynamic priority of the document, the higher the frequency of its index update. To limit the index update time T update (d i ), and ensure that the total update time does not exceed T max , the following constraint conditions are designed: Among them, P(d i ) > θ. Only documents with a dynamic priority higher than will be preferentially updated, thus ensuring that the system will not occupy too many resources due to the update of low-priority documents. Of course, when the system load is high, the index update of low-priority documents will be temporarily ignored, and the system will automatically adjust the value to ensure the timely update of the indexes of key documents.

[0142] Specifically, the system may encounter high-load situations when processing queries, especially in the case of a large number of concurrent queries or a large number of document updates. To ensure the retrieval effect of key documents, the system will temporarily ignore the index updates of low-priority documents and automatically adjust parameters to ensure the timely update of the indexes of important documents.

[0143] Suppose the system load is high, resulting in a delay in the index update of low-priority documents. In this case, the system's priority adjustment mechanism will determine which documents should be preferentially updated based on the access frequency and weight of the documents. Let be the index update threshold of the document, and documents below will be temporarily ignored.

[0144] In one of the embodiments, the file quick retrieval method further includes:

[0145] Obtain the click-through rate P(a i ) of the document by dividing the number of times the document is clicked by the number of times the document is displayed;

[0146] Obtain the priority threshold θ of index update. When P(a i ) < θ and / or when , adjust the sorting of the i-th document in the index structure; where N is the total number of documents.

[0147] Among them, the click-through rate P(a ) of the i-th document is obtained through i , where C i is the number of times the i-th document is clicked, and Qi is the number of times the i-th document is displayed.

[0148] Furthermore, when the click-through rate P(a i ) of a certain document is continuously lower than the set threshold θ, the system will automatically retrain the ranking model to further improve the relevance of the recommended documents. The system will optimize the ranking of documents according to user behavior (such as the click-through rate of documents). When the click-through rate P(a i ) of a certain document is continuously lower than the set threshold θ, the system will trigger the retraining of the ranking model to improve the relevance of document recommendations.

[0149] User feedback (such as the click-through rate) is an important indicator to measure the retrieval quality of the system. When the feedback is not ideal, the system will automatically initiate model retraining and adjust the document ranking rules to improve the relevance of retrieval results and user satisfaction. Suppose the user is not satisfied with the results returned by the system (low click-through rate) after multiple queries, the system will automatically adjust the ranking model and relearn the user preferences. The trigger condition for model retraining can be determined by formula.

[0150] In one of the embodiments, the file fast retrieval method further includes:

[0151] Setting multiple nodes in the distributed file retrieval system to jointly execute the index structure to process the document retrieval task;

[0152] Adjusting the load of the target node by minimizing the load variance.

[0153] Specifically, by minimizing the load variance to adjust the load Lj of the j-th node j ; where m is the total number of nodes.

[0154] Among them, by minimizing the load variance ensuring that the loads of each node are evenly distributed to prevent performance bottlenecks caused by some nodes being overloaded. When the load Lj of the j-th node j exceeds the set maximum threshold L max , the system will reallocate tasks to avoid computational bottlenecks.

[0155] Specifically, in the distributed file retrieval system, multiple nodes jointly process the document query task. When the load Lj of the j-th node j exceeds the maximum load threshold L set by the system max , the system will automatically reallocate tasks to avoid a certain node becoming a computational bottleneck.

[0156] Suppose the load Lj of the j-th node j exceeds the threshold L set by the system maxAt this time, the system will transfer some tasks on this node to other nodes.

[0157] In one embodiment, the file fast retrieval method further includes:

[0158] Monitoring the load of nodes in the distributed file retrieval system;

[0159] In response to the load of the target node being greater than the maximum load threshold, reallocate the retrieval tasks and the load of the target node after the task reallocation.

[0160] Specifically, when the target node is the j-th node, monitor the load Lj of the j-th node in the distributed file retrieval system j ; in response to the load Lj of the j-th node j being greater than the maximum load threshold L max perform retrieval task reallocation, and the load of the j-th node after the task reallocation is where Lj j is the load of the j-th node, L max is the maximum load threshold, N k is the number of target nodes for performing retrieval task allocation, and Lj j - L max is the overloaded load part of the j-th node. Through this mechanism, the load of each node can be effectively balanced, preventing a certain node from becoming a performance bottleneck.

[0161] In the above file fast retrieval method, by vectorizing the text content in the original document to generate document vectors, and then clustering to form multiple document clusters, and setting dynamic priorities for the original documents in each document cluster to adjust the document sorting of the index structure, the retrieval efficiency can be effectively improved. It can quickly find documents with high dynamic priorities in the case of high concurrency or high load, reducing system response latency or resource consumption.

[0162] The present invention can automatically adjust the index update strategy when the system load is high, prioritize the processing of documents with high weights (such as high user click-through rate and access frequency), and improve the retrieval speed of key documents. This ability of dynamic adjustment significantly improves the efficiency of the system in a large-scale document retrieval environment. A method for calculating document weights based on user behavior (such as click-through rate and access frequency) is introduced in the solution to ensure that documents with high relevance and high access frequency are preferentially recommended and retrieved. At the same time, when the click-through rate of some documents continues to be lower than the threshold, the system will automatically retrain the ranking model to make the retrieval results more in line with user needs, enhancing the user experience and retrieval accuracy; The solution provides a load balancing and resource self-adaptation module that can monitor the system load in real time. When the load of a certain node exceeds the set threshold, the system can dynamically adjust the task priorities and allocations to avoid computational bottlenecks in some nodes. This mechanism enables the system to operate smoothly under high concurrency and ensures the priority execution of key tasks; The solution also designs a mechanism to automatically prune low-priority tasks when the system resources are tight to ensure that high-priority tasks can be normally executed under limited resources. Compared with traditional systems, it can manage and allocate resources more efficiently, improving the stability and execution efficiency of the system under resource constraints.

[0163] In one embodiment, as Figure 3 shown, a file fast retrieval system 10 is provided, including: a document preprocessing module 1, a document clustering and weight calculation module 2, an index construction and optimization module 3, a task scheduling and load balancing module 4, a feedback and model optimization module 5, and a system monitoring and self-adaptive optimization module 6.

[0164] The document preprocessing module 1 is responsible for vectorizing and feature selecting the original documents, which is the basic part of the entire retrieval system. This module converts the text content into vector representations, generates document vectors using TF-IDF natural language processing technology, and then performs dimensionality reduction operations on the document vectors to reduce redundant information while retaining the most discriminative features;

[0165] The document clustering and weight calculation module 2 is responsible for clustering and weight calculating the documents to ensure that similar documents can be retrieved quickly and important documents can be preferentially displayed. This module uses a clustering algorithm to cluster the document vectors, classifies them based on the similarity of the document content, and dynamically calculates the weights of the documents according to historical behaviors such as the click-through rate and access frequency of the documents. Documents with higher weights are preferentially processed in indexing and retrieval;

[0166] The index construction and optimization module 3 is responsible for building an inverted index, dynamically adjusting the index structure according to the priority and weight of documents, and constructing a retrieval structure based on the inverted index to quickly locate relevant documents. Documents with high weights are indexed first to ensure quick responses for high-priority documents. The system dynamically updates the index according to the load situation. If the system load is high, the index update of low-priority documents will be temporarily ignored to prioritize the index update and retrieval speed of high-priority documents;

[0167] The task scheduling and load balancing module 4 is responsible for scheduling tasks and allocating resources when the system load changes, ensuring that the system can operate smoothly under high load. According to the weight and importance of documents, it prioritizes the index update and retrieval tasks of high-priority documents. When system resources are scarce, low-priority tasks can be postponed for execution, and it monitors the load situation of each node in real time. When the load of a certain node exceeds the threshold, the system will automatically reallocate tasks to avoid computational bottlenecks;

[0168] The feedback and model optimization module 5 continuously optimizes the ranking model based on user behavior data to ensure the relevance and accuracy of document recommendations. Based on indicators such as user click behavior and access frequency, it dynamically adjusts the weights of documents. If the click-through rate of a certain document continuously falls below the set threshold, the system will trigger retraining of the model and optimize the ranking algorithm. The system regularly retrains the document ranking model according to user feedback to optimize the document recommendation order and ensure high relevance and user experience. At the same time, when resources are scarce, low-priority tasks will be trimmed to ensure the execution of high-priority tasks;

[0169] The system monitoring and adaptive optimization module 6 is responsible for the overall health monitoring and adaptive optimization of the system, and monitors the system resource occupancy in real time. When the system load is high, it adjusts the priority and execution order of tasks to ensure that critical tasks are not affected. When some tasks cannot be effectively completed for a long time, the system will automatically adjust the task weights, scheduling strategies or retrain the ranking model to improve the overall efficiency and user experience of the system.

[0170] Through the collaborative work of modules such as document preprocessing, clustering and weight calculation, index optimization, task scheduling, user feedback and model optimization, the entire system realizes an intelligent and efficient file retrieval system. Each module is interrelated and operates in coordination to ensure that in a high-load and resource-constrained environment, the system can still prioritize the smooth completion of important tasks while continuously improving the user experience and the relevance of document recommendations.

[0171] In the above-mentioned file fast retrieval system, the text content in the original document is vectorized to generate document vectors, and then clustered to form multiple document clusters. The dynamic priority is set for the original documents in each document cluster to adjust the document sorting of the index structure, which can effectively improve the retrieval efficiency. In the case of high concurrency or high load, it can quickly find the documents with high dynamic priority, reducing the system response delay or resource consumption.

[0172] For the specific limitations of the file fast retrieval device, reference can be made to the limitations of the file fast retrieval method in the above text, which will not be elaborated here. Each module in the above file fast retrieval device can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor of the computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0173] In one embodiment, a computer device is provided. The computer device can be a server, and its internal structure diagram can be as Figure 4 shown. The computer device includes a processor, a memory, a network interface, and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store file fast retrieval data. The network interface of the computer device is used to communicate with external terminals through a network connection. When the computer program is executed by the processor, it implements a file fast retrieval method.

[0174] Those skilled in the art can understand that Figure 4 the structure shown in

[0175] is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0176] Vectorize the text content in the original document to generate document vectors;

[0177] Cluster and group the original documents according to the similarity between the document vectors to form multiple document clusters;

[0178] Set the dynamic priorities of the original documents in each document cluster according to the update frequency and / or query frequency of the original documents in the document cluster;

[0179] Build an index structure for the document vectors and adjust the sorting of the document vectors in the index structure according to the dynamic priorities of the original documents;

[0180] In response to a user's query, convert the user's query into a query vector, and obtain the original documents relevant to the user's query by calculating the similarity between the query vector and the document vectors in the sorted index structure of the document vectors.

[0181] In one embodiment, the vectorization representation of the text content in the original document to generate a document vector includes:

[0182] Obtain the weighted value of each target word in the original document according to the word frequency of the target word in the target document, the total number of documents in the original document, and the number of documents containing the target word, where the target word is a word in the original document and the target document is a document in the original document;

[0183] Obtain the weighted value of each word in the original document and convert it into a vector representation to form a document vector.

[0184] In one embodiment, setting the dynamic priorities of the original documents in each document cluster according to the update frequency and / or query frequency of the original documents in the document cluster includes:

[0185] Obtain the update frequency and query frequency of the original document, and assign dynamic priorities to the original document by weighted summation of the update frequency and query frequency.

[0186] In one embodiment, in response to a user's query, converting the user's query into a query vector, and obtaining the original documents relevant to the user's query by calculating the similarity between the query vector and the document vectors in the sorted index structure of the document vectors includes:

[0187] Obtain the user's query and convert the user's query into a query vector;

[0188] Obtain the document vectors in the index structure, and calculate the similarity between the query vector and the document vectors in the sorted index structure of the document vectors by the cosine similarity method to measure the relevance between the user's query and the original document.

[0189] In one embodiment, the file fast retrieval method further includes:

[0190] Obtain the ideal sorting after the user's click feedback, and adjust the sorting of the document vectors in the index structure in real time by minimizing the error of the document ranking.

[0191] In one embodiment, the file fast retrieval method further includes:

[0192] Obtain the total computing resources for executing the index structure and the computing resources consumed for executing each retrieval task, and set the sum of the computing resources consumed for executing the retrieval tasks to be less than or equal to the total computing resources of the execution index structure;

[0193] Obtain the remaining computing resources;

[0194] When the remaining computing resources are less than or equal to the first threshold, set the priority weight of the corresponding retrieval task according to the dynamic priority of the original document, and allocate computing resources in the decreasing order of the priority weight of the retrieval task;

[0195] When the remaining computing resources are greater than the first threshold, delete the retrieval tasks with priority weights less than the second threshold, and obtain the remaining computing resources after trimming the low-priority tasks, so that the retrieval tasks with priority weights greater than or equal to the second threshold are allocated the remaining computing resources after trimming the low-priority tasks.

[0196] In one embodiment, deleting the retrieval tasks with priority weights less than the second threshold and obtaining the remaining computing resources after trimming the low-priority tasks includes:

[0197] Obtain the computing resources corresponding to the retrieval tasks with priority weights less than the second threshold;

[0198] Trim the low-priority tasks by subtracting the computing resources corresponding to the retrieval tasks with priority weights less than the second threshold from the total computing resources of the execution index structure, and obtain the remaining computing resources after trimming the low-priority tasks.

[0199] In one embodiment, the file fast retrieval method further includes:

[0200] Obtain the average query duration T of the retrieved documents query ;

[0201] Obtain the probability P(c i ) that the user clicks on the document d i ), where rel(d i ) represents the relevance score of the target document d i ), and α is a weight coefficient for adjusting the update frequency;

[0202] Obtain the total computing resources R and resource utilization rate R of the execution index structure used ;

[0203] Optimize the index structure by maximizing the system performance metric M. The way of maximizing the system performance metric M is where α1 is the query frequency weight parameter and α2 is the call rate weight parameter.

[0204] In one embodiment, the file fast retrieval method further includes:

[0205] Set the update frequency of the index in the index structure to be positively correlated with the dynamic priority of the original document corresponding to the index;

[0206] Obtain the priority threshold for index update by dividing the document weight by the load of the current node;

[0207] Update the index for the original document whose dynamic priority is greater than the priority threshold for index update;

[0208] The update constraint condition for the index in the index structure is where P(d i ) > θ, T update (d i ) is the index update time, T max is the total update time, n is the number of retrieval tasks, and θ is the priority threshold for index update.

[0209] In one embodiment, the file fast retrieval method further includes:

[0210] Obtain the click-through rate P(a i ) of the document by dividing the number of times the document is clicked by the number of times the document is displayed;

[0211] Obtain the priority threshold θ for index update. When P(a i ) < θ and / or when , adjust the sorting of the documents in the index structure; where N is the total number of documents.

[0212] In one embodiment, the file fast retrieval method further includes:

[0213] Set multiple nodes in the distributed file retrieval system to jointly execute the document retrieval task for the index structure;

[0214] Adjust the load of the target node by minimizing the load variance.

[0215] In one embodiment, the file fast retrieval method further includes:

[0216] Monitor the load of the nodes in the distributed file retrieval system;

[0217] In response to the load of the target node being greater than the maximum load threshold, reallocate the retrieval task and the load of the target node after the task reallocation.

[0218] For the specific limitations on the implementation steps when the computer program is executed by the processor, reference can be made to the limitations on the file fast retrieval method in the above text, which will not be elaborated here.

[0219] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented:

[0220] Perform vectorization representation on the text content in the original document to generate a document vector;

[0221] Cluster and group the original documents according to the similarity between the document vectors to form multiple document clusters;

[0222] Set the dynamic priority of the original documents in each document cluster according to the update frequency and / or query frequency of the original documents in the document cluster;

[0223] Build an index structure for the document vectors, and adjust the sorting of the document vectors in the index structure according to the dynamic priority of the original documents;

[0224] In response to a user's query, convert the user's query into a query vector, and obtain the original documents related to the user's query by calculating the similarity between the query vector and the document vectors in the sorted index structure of the document vectors.

[0225] In one of the embodiments, performing vectorization representation on the text content in the original document to generate a document vector includes:

[0226] Obtain the weighted value of each target word in the original document according to the word frequency of the target word in the target document, the total number of documents in the original document, and the number of documents containing the target word, where the target word is a word in the original document and the target document is a document in the original document;

[0227] Obtain the weighted value of each word in the original document and convert it into a vector representation to form a document vector.

[0228] In one of the embodiments, setting the dynamic priority of the original documents in each document cluster according to the update frequency and / or query frequency of the original documents in the document cluster includes:

[0229] Obtain the update frequency and query frequency of the original documents, and assign dynamic priority to the original documents by weighted summation of the update frequency and query frequency.

[0230] In one of the embodiments, in response to a user's query, converting the user's query into a query vector, and obtaining the original documents related to the user's query by calculating the similarity between the query vector and the document vectors in the sorted index structure of the document vectors includes:

[0231] Obtain the user's query, and convert the user's query into a query vector;

[0232] Obtain the document vectors in the index structure, and measure the relevance between the user's query and the original document by calculating the similarity between the query vector and the document vectors in the sorted index structure through the cosine similarity method.

[0233] In one embodiment, the file fast retrieval method further includes:

[0234] Obtain the ideal sorting after the user's click feedback, and adjust the sorting of the document vectors in the index structure in real time by minimizing the error of the document ranking.

[0235] In one embodiment, the file fast retrieval method further includes:

[0236] Obtain the total computing resources for executing the index structure and the computing resources required for each retrieval task, and set the sum of the computing resources consumed by the execution of the retrieval tasks to be less than or equal to the total computing resources for executing the index structure;

[0237] Obtain the remaining computing resources;

[0238] In response to the remaining computing resources being less than or equal to the first threshold, set the priority weight of the corresponding retrieval task according to the dynamic priority of the original document, and allocate computing resources in the decreasing order of the priority weights of the retrieval tasks;

[0239] In response to the remaining computing resources being greater than the first threshold, delete the retrieval tasks with priority weights less than the second threshold, and obtain the remaining computing resources after trimming the low-priority tasks, so that the retrieval tasks with priority weights greater than or equal to the second threshold are allocated the remaining computing resources after trimming the low-priority tasks.

[0240] In one embodiment, deleting the retrieval tasks with priority weights less than the second threshold and obtaining the remaining computing resources after trimming the low-priority tasks includes:

[0241] Obtain the computing resources corresponding to the retrieval tasks with priority weights less than the second threshold;

[0242] Trim the low-priority tasks by subtracting the computing resources corresponding to the retrieval tasks with priority weights less than the second threshold from the total computing resources for executing the index structure, and obtain the remaining computing resources after trimming the low-priority tasks.

[0243] In one embodiment, the file fast retrieval method further includes:

[0244] Obtain the average query duration T of the retrieved documents query ;

[0245] Obtain the probability P(c i ) that the user clicks on the document d i ), where rel(d i ) represents the relevance score of the target document d i , and α is the weight coefficient for adjusting the update frequency;

[0246] Obtain the total computing resources R and resource utilization rate R for executing the index structure used ;

[0247] Optimize the index structure by maximizing the system performance metric M. The way to maximize the system performance metric M is where α1 is the query frequency weight parameter and α2 is the call rate weight parameter.

[0248] In one embodiment, the file fast retrieval method further includes:

[0249] Set the update frequency of the index in the index structure to be positively correlated with the dynamic priority of the original document corresponding to the index;

[0250] Obtain the priority threshold for index update by dividing the document weight by the load of the current node;

[0251] Perform index update on the original document whose dynamic priority is greater than the priority threshold for index update;

[0252] The update constraint condition for the index in the index structure is where P(d i ) > θ, T update (d i ) is the index update time, T max is the total update time, n is the number of retrieval tasks, and θ is the priority threshold for index update.

[0253] In one embodiment, the file fast retrieval method further includes:

[0254] Obtain the click-through rate P(a i ) of the document by dividing the number of times the document is clicked by the number of times the document is displayed;

[0255] Obtain the priority threshold θ for index update. When P(a i ) < θ and / or when , adjust the sorting of the documents in the index structure; where N is the total number of documents.

[0256] In one embodiment, the file fast retrieval method further includes:

[0257] Set multiple nodes in the distributed file retrieval system to jointly execute the document retrieval task for the index structure;

[0258] Adjust the load of the target node by minimizing the load variance.

[0259] In one embodiment, the file fast retrieval method further includes:

[0260] Monitoring the load of nodes in the distributed file retrieval system;

[0261] When the load of the target node is greater than the maximum load threshold, reallocating the retrieval task and the load of the target node after the task reallocation.

[0262] For the specific limitations on the steps implemented when the processor executes the computer program, reference can be made to the limitations on the file fast retrieval method in the above text, which will not be elaborated here.

[0263] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0264] Performing vectorization representation on the text content in the original document to generate a document vector;

[0265] Clustering and grouping the original documents according to the similarity between the document vectors to form multiple document clusters;

[0266] Setting the dynamic priority of the original documents in each document cluster according to the update frequency and / or query frequency of the original documents in the document cluster;

[0267] Establishing an index structure for the document vectors and adjusting the sorting of the document vectors in the index structure according to the dynamic priority of the original documents;

[0268] When a user initiates a query, converting the user's query into a query vector, and obtaining the original documents related to the user's query by calculating the similarity between the query vector and the document vectors in the sorted index structure of the document vectors.

[0269] In one embodiment, performing vectorization representation on the text content in the original document to generate a document vector includes:

[0270] Obtaining the weighted value of each target word in the original document according to the word frequency of the target word in the target document, the total number of documents in the original document, and the number of documents containing the target word, where the target word is a word in the original document and the target document is a document in the original document;

[0271] Obtaining the weighted value of each word in the original document and converting it into a vector representation to form a document vector.

[0272] In one embodiment, setting the dynamic priority of the original documents in each document cluster according to the update frequency and / or query frequency of the original documents in the document cluster includes:

[0273] Obtain the update frequency and query frequency of the original document, and assign dynamic priorities to the original documents by weighted summation of the update frequency and query frequency.

[0274] In one embodiment, when a user initiates a query, convert the user's query into a query vector, and obtain the original documents related to the user's query by calculating the similarity between the query vector and the document vectors in the sorted index structure of the document vectors, including:

[0275] Obtain the user's initiated query and convert the user's query into a query vector;

[0276] Obtain the document vectors in the index structure, and calculate the similarity between the query vector and the document vectors in the sorted index structure of the document vectors by cosine similarity to measure the relevance between the user's query and the original documents.

[0277] In one embodiment, the file fast retrieval method further includes:

[0278] Obtain the ideal sorting after the user's click feedback, and adjust the sorting of the document vectors in the index structure in real time by minimizing the error of the document ranking.

[0279] In one embodiment, the file fast retrieval method further includes:

[0280] Obtain the total computing resources for executing the index structure and the computing resources required for each retrieval task, and set the sum of the computing resources consumed by the execution of the retrieval tasks to be less than or equal to the total computing resources for executing the index structure;

[0281] Obtain the remaining computing resources;

[0282] When the remaining computing resources are less than or equal to the first threshold, set the priority weights of the corresponding retrieval tasks according to the dynamic priorities of the original documents, and allocate the computing resources in the decreasing order of the priority weights of the retrieval tasks;

[0283] When the remaining computing resources are greater than the first threshold, delete the retrieval tasks with priority weights less than the second threshold to obtain the remaining computing resources after pruning the low-priority tasks, so that the retrieval tasks with priority weights greater than or equal to the second threshold are allocated the remaining computing resources after pruning the low-priority tasks.

[0284] In one embodiment, deleting the retrieval tasks with priority weights less than the second threshold to obtain the remaining computing resources after pruning the low-priority tasks includes:

[0285] Obtain the computing resources corresponding to the retrieval tasks with priority weights less than the second threshold;

[0286] Prune low-priority tasks by subtracting the computing resources corresponding to retrieval tasks with a priority weight less than a second threshold from the total computing resources of the index structure, and obtain the remaining computing resources after pruning the low-priority tasks.

[0287] In one embodiment, the file fast retrieval method further includes:

[0288] Obtain the average query duration T of the retrieved documents query ;

[0289] Obtain the probability P(c i ) that the user clicks on the document d i ), where rel(d i ) represents the relevance score of the target document d i ), and α is a weight coefficient for adjusting the update frequency;

[0290] Obtain the total computing resources R of the execution index structure and the resource utilization rate R used ;

[0291] Optimize the index structure by maximizing the system performance metric M. The way to maximize the system performance metric M is where α1 is the query frequency weight parameter and α2 is the call rate weight parameter.

[0292] In one embodiment, the file fast retrieval method further includes:

[0293] Set the update frequency of the index in the index structure to be positively correlated with the dynamic priority of the original document corresponding to the index;

[0294] Obtain the priority threshold for index update by dividing the document weight by the load of the current node;

[0295] Update the index for the original document with a dynamic priority greater than the priority threshold for index update;

[0296] The update constraint condition for the index in the index structure is where P(d i ) > θ, T update (d i ) is the index update time, T max is the total update time, n is the number of retrieval tasks, and θ is the priority threshold for index update.

[0297] In one embodiment, the file fast retrieval method further includes:

[0298] Obtain the click-through rate P(a i ) of the document by dividing the number of times the document is clicked by the number of times the document is displayed;

[0299] Obtain the priority threshold θ for index update. When P(a i ) < θ and / or when , adjust the sorting of documents in the index structure; where N is the total number of documents.

[0300] In one embodiment, the file fast retrieval method further includes:

[0301] Set multiple nodes in the distributed file retrieval system to jointly execute the document retrieval task for the index structure;

[0302] Adjust the load of the target node by minimizing the load variance.

[0303] In one embodiment, the file fast retrieval method further includes:

[0304] Monitor the load of nodes in the distributed file retrieval system;

[0305] In response to the load of the target node being greater than the maximum load threshold, reassign the retrieval task and the load of the target node after the task is reassigned.

[0306] For the specific limitations on the implementation steps when the computer program is executed by the processor, reference can be made to the limitations on the file fast retrieval method in the above text, which will not be elaborated here.

[0307] Those of ordinary skill in the art can understand that all or part of the processes in implementing the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to the memory, storage, database or other media used in the various embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0308] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0309] The above-described embodiments merely represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the appended claims.

Claims

1. A method for quickly retrieving files, characterized in that, Including: Performing vectorization representation on the text content in the original document to generate document vectors; Clustering and grouping the original documents according to the similarity between the document vectors to form multiple document clusters; Setting the dynamic priorities of the original documents in each document cluster according to the update frequency and / or query frequency of the original documents in the document cluster; Establishing an index structure for the document vectors, and adjusting the sorting of the document vectors in the index structure according to the dynamic priorities of the original documents; In response to a user's query, converting the user's query into a query vector, and obtaining the original documents related to the user's query by calculating the similarity between the query vector and the document vectors in the sorted index structure of the document vectors; Wherein, the file fast retrieval method further includes: Obtain the average query duration T of the retrieved documents query ; Obtain the probability P(c i ) that the user clicks on the document; where i is the document serial number; Obtain the total computing resources R and resource utilization rate R for executing the index structure used ; Optimizing the index structure by maximizing the system performance metric M, where the way of maximizing the system performance metric M is where α1 is the query frequency weight parameter and α2 is the call rate weight parameter; Wherein, the file fast retrieval method further includes: Setting the update frequency of the index in the index structure to be positively correlated with the dynamic priority of the original document corresponding to the index; obtaining the priority threshold for index update by dividing the document weight by the load of the current node; Performing index update on the original documents whose dynamic priorities are greater than the priority threshold for index update; The update constraint condition of the index in the index structure is where P(d i ) > θ, T update (d i ) is the index update time, T max is the total update time, n is the number of retrieval tasks, θ is the priority threshold for index update, d i is the document clicked by the user, and P(d i ) is the dynamic priority of the original document; Wherein, the file fast retrieval method further includes: Obtain the click-through rate P(a of the document by dividing the number of times the document is clicked by the number of times the document is displayed i ); Obtain the priority threshold θ for index update. When P(a i ) < θ and / or when , adjust the sorting of documents in the index structure; where N is the total number of documents; Wherein, the file fast retrieval method further includes: Dividing the index structure into a first sub-retrieval structure and a second sub-retrieval structure according to the dynamic priority, and the dynamic priority of the original document corresponding to the document vector in the first sub-retrieval structure is greater than the dynamic priority of the original document corresponding to the document vector in the second sub-retrieval structure; In response to a user's query, sequentially retrieving in the first sub-retrieval structure and the second sub-retrieval structure to obtain the original documents related to the user's query; When retrieving and obtaining the original documents related to the user's query in the first sub-retrieval structure, controlling not to allocate computing resources to the second sub-retrieval structure; when retrieving and obtaining the original documents related to the user's query in the second sub-retrieval structure, controlling not to allocate computing resources to the first sub-retrieval structure.

2. The method for quickly retrieving documents according to claim 1, wherein The performing vectorization representation on the text content in the original document to generate document vectors includes: Obtaining the weighted value of each target word in the original document according to the word frequency of the target word in the target document, the total number of documents in the original document, and the number of documents containing the target word, where the target word is a word in the original document and the target document is a document in the original document; Obtaining the weighted value of each target word in each target document in the original document and converting it into a vector representation to form the document vector of the target document.

3. The file quick retrieval method according to claim 1, characterized in that, The setting the dynamic priorities of the original documents in each document cluster according to the update frequency and / or query frequency of the original documents in the document cluster includes: Obtaining the update frequency and query frequency of the original document, and allocating dynamic priorities to the original document by weighted summing the update frequency and the query frequency.

4. The file quick retrieval method according to claim 1, characterized in that, The in response to a user's query, converting the user's query into a query vector, and obtaining the original documents related to the user's query by calculating the similarity between the query vector and the document vectors in the sorted index structure of the document vectors includes: obtaining the user's query and converting the user's query into a query vector; Obtain the document vectors in the index structure, and measure the relevance between the user's query and the original document by calculating the similarity between the query vector and the document vectors in the sorted index structure through the cosine similarity method.

5. The method for quickly retrieving a document according to claim 1, wherein The file fast retrieval method further includes: Obtain the ideal sorting after the user's click feedback, and adjust the sorting of the document vectors in the index structure in real time by minimizing the error of the document ranking.

6. The method for quickly retrieving documents according to claim 1, wherein The file fast retrieval method further includes: Obtain the total computing resources for executing the index structure and the computing resources required for each retrieval task, and set the sum of the computing resources consumed by the execution of the retrieval tasks to be less than or equal to the total computing resources for executing the index structure; Obtain the remaining computing resources; In response to the remaining computing resources being less than or equal to the first threshold, set the priority weights of the corresponding retrieval tasks according to the dynamic priorities of the original documents, and allocate computing resources in the decreasing order of the priority weights of the retrieval tasks; in response to the remaining computing resources being greater than the first threshold, delete the retrieval tasks with priority weights less than the second threshold to obtain the remaining computing resources after trimming the low-priority tasks, so that the retrieval tasks with priority weights greater than or equal to the second threshold are allocated the remaining computing resources after trimming the low-priority tasks.

7. The method for quickly retrieving documents according to claim 6, characterized in that, The obtaining the remaining computing resources after trimming the low-priority tasks by deleting the retrieval tasks with priority weights less than the second threshold includes: Obtain the computing resources corresponding to the retrieval tasks with priority weights less than the second threshold; Trim the low-priority tasks by subtracting the computing resources corresponding to the retrieval tasks with priority weights less than the second threshold from the total computing resources for executing the index structure, and obtain the remaining computing resources after trimming the low-priority tasks.

8. The method for quickly retrieving documents according to claim 1, wherein The file fast retrieval method further includes: Set that multiple nodes are used to jointly execute the index structure to process the document retrieval tasks in the distributed file retrieval system; adjust the load of the target node by minimizing the load variance.

9. The method for quickly retrieving a document according to claim 8, wherein, The file fast retrieval method further includes: Monitor the load of the nodes in the distributed file retrieval system; In response to the load of the target node being greater than the maximum load threshold, reallocate the retrieval tasks and the load of the target node after the reallocation of the tasks.

10. A computer program product comprising a computer program, characterized in that, 11. A computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9. When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

Images