Fast file retrieval method, product, computer equipment and storage medium

Abstract

The invention relates to a quick file retrieval method, a product, computer equipment and a storage medium. The method comprises the following steps: performing vectorization representation on text content in an original document to generate a document vector; clustering and grouping the original documents according to the similarity between the document vectors to form a plurality of document clusters; setting the dynamic priority of the original documents in each document cluster according to the updating frequency and / or query frequency of the original documents in the document clusters; establishing an index structure for the document vectors, and adjusting the order of the document vectors in the index structure according to the dynamic priority of the original document; when the query initiated by the user is responded, the query of the user is converted into a query vector, and the original document related to the query of the user is obtained by calculating the similarity of the document vectors in the index structure after the query vector and the document vectors are sequenced. Retrieval efficiency can be effectively improved, documents with high dynamic priorities can be quickly searched, and system response delay or resource consumption is reduced.

Description

Technical Field

[0001] The present application relates to the field of server technology, and in particular to a method, product, computer device and storage medium for rapid file retrieval. Background Art

[0002] Existing document retrieval technologies mainly rely on traditional algorithms such as inverted index and keyword matching, combined with natural language processing (NLP) technology, to achieve document and query matching through vectorized representation. At the same time, based on user behavior feedback mechanisms and machine learning technology, existing systems can optimize document sorting through indicators such as click-through rate and access frequency. However, as the scale of data continues to increase, existing technologies often need to rely on distributed architectures to process massive amounts of data, and in high concurrency situations, resource scheduling and load balancing become key challenges. Summary of the invention

[0003] Based on this, a method, apparatus, computer device and storage medium for rapid file retrieval are provided, aiming to solve the technical problem that when a traditional file retrieval system processes a large number of documents, the efficiency of index construction and retrieval gradually decreases, especially in high concurrency or high load conditions, system response delays or resource exhaustion are prone to occur.

[0004] In one aspect, a method for rapid file retrieval is provided, the method comprising: Vectorize the text content in the original document to generate a document vector; Clustering and grouping the original documents according to the similarity between the document vectors to form a plurality of document clusters; Setting the dynamic priority of the original documents in each document cluster according to the update frequency and / or query frequency of the original documents in the document cluster; Establishing an index structure for the document vectors, and adjusting the order of the document vectors in the index structure according to the dynamic priority of the original document; In response to a user initiating a query, the user's query is converted into a query vector, and the original document related to the user's query is obtained by calculating the similarity between the query vector and the document vector in the index structure after the document vector is sorted.

[0005] In one embodiment, vectorizing the text content in the original document to generate a document vector includes: Obtaining a weighted value of each target word in the original document according to the word frequency of the target word in the target document, the total number of documents in the original document, and the number of documents containing the target word, wherein the target word is a word in the original document, and the target document is a document in the original document; The weighted value of each target word in each target document in the original document is obtained and converted into a vector representation to form a document vector of the target document.

[0006] In one embodiment, setting the dynamic priority of the original documents in each document cluster according to the update frequency and / or query frequency of the original documents in the document cluster includes: The update frequency and query frequency of the original document are obtained, and a dynamic priority is assigned to the original document by weighted summing of the update frequency and the query frequency.

[0007] In one embodiment, in response to a user initiating a query, converting the user's query into a query vector, and obtaining original documents related to the user's query by calculating the similarity between the query vector and the document vector in the index structure after the document vector is sorted includes: Get the query initiated by the user and convert the user's query into a query vector; The document vector in the index structure is obtained, and the similarity between the query vector and the document vector in the index structure after the document vector is sorted is calculated by cosine similarity to measure the relevance between the user's query and the original document.

[0008] In one embodiment, the method for rapid file retrieval further includes: The ideal ranking after the user click feedback is obtained, and the document vector ranking in the index structure is adjusted in real time by minimizing the error of the document ranking.

[0009] In one embodiment, the method for rapid file retrieval further includes: Obtain the total computing resources required to execute the index structure and the computing resources required to execute each retrieval task, and set the sum of the computing resources consumed to execute the retrieval task to be less than or equal to the total computing resources required to execute the index structure; Get the remaining computing resources; In response to the remaining computing resources being less than or equal to a first threshold, setting a priority weight of a corresponding retrieval task according to the dynamic priority of the original document, and allocating computing resources in descending order of the priority weights of the retrieval tasks; In response to the remaining computing resources being greater than a first threshold, retrieval tasks whose priority weights are less than a second threshold are deleted to obtain the remaining computing resources after trimming the low priority tasks, so that retrieval tasks whose priority weights are greater than or equal to the second threshold are allocated to the remaining computing resources after trimming the low priority tasks.

[0010] In one embodiment, deleting the search tasks whose priority weight is less than the second threshold to obtain the remaining computing resources after trimming the low priority tasks includes: Acquire computing resources corresponding to retrieval tasks whose priority weights are less than a second threshold; The low priority tasks are trimmed by subtracting the computing resources corresponding to the retrieval tasks whose priority weights are less than the second threshold from the total computing resources of the executing index structure, thereby obtaining the remaining computing resources after trimming the low priority tasks.

[0011] In one embodiment, the method for rapid file retrieval further includes: Get the average query time for retrieving documents ; Get user click document Probability ,in , Represents the target document The relevance score of is the weight coefficient for adjusting the update frequency; Get the total computing resources R and resource utilization of the index structure ; The index structure is optimized by maximizing the system performance index M, where the maximizing system performance index M is: ,in is the query frequency weight parameter, is the call rate weight parameter.

[0012] In one embodiment, the method for rapid file retrieval further includes: Setting the update frequency of the index in the index structure to be positively correlated with the dynamic priority of the original document corresponding to the index; The priority threshold for index updates is obtained by dividing the document weight by the current node load. Performing index update on original documents whose dynamic priority is greater than the priority threshold of index update; The update constraint of the index in the index structure is ,in , The index update time. is the total update time, n is the number of retrieval tasks, The priority threshold for index updates.

[0013] In one embodiment, the method for rapid file retrieval further includes: The click-through rate of a document is obtained by dividing the number of times the document is clicked by the number of times the document is displayed. ; Get the priority threshold for index updates ,when and / or when When , adjusting the order of documents in the index structure; wherein is the total number of documents.

[0014] In one embodiment, the method for rapid file retrieval further includes: In a distributed document retrieval system, multiple nodes are used to jointly execute the document retrieval task of processing the index structure; The load of the target node is adjusted by minimizing the load variance.

[0015] In one embodiment, the method for rapid file retrieval further includes: Monitoring the load of nodes in the distributed file retrieval system; In response to the load of the target node being greater than the maximum load threshold, the retrieval task is redistributed to reduce the load of the target node after the task is redistributed.

[0016] In another aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps: Vectorize the text content in the original document to generate a document vector; Clustering and grouping the original documents according to the similarity between the document vectors to form a plurality of document clusters; Setting the dynamic priority of the original documents in each document cluster according to the update frequency and / or query frequency of the original documents in the document cluster; Establishing an index structure for the document vectors, and adjusting the order of the document vectors in the index structure according to the dynamic priority of the original document; In response to a user initiating a query, the user's query is converted into a query vector, and the original document related to the user's query is obtained by calculating the similarity between the query vector and the document vector in the index structure after the document vector is sorted.

[0017] In another aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented: Vectorize the text content in the original document to generate a document vector; Clustering and grouping the original documents according to the similarity between the document vectors to form a plurality of document clusters; Setting the dynamic priority of the original documents in each document cluster according to the update frequency and / or query frequency of the original documents in the document cluster; Establishing an index structure for the document vectors, and adjusting the order of the document vectors in the index structure according to the dynamic priority of the original document; In response to a user initiating a query, the user's query is converted into a query vector, and the original document related to the user's query is obtained by calculating the similarity between the query vector and the document vector in the index structure after the document vector is sorted.

[0018] In another aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: Vectorize the text content in the original document to generate a document vector; Clustering and grouping the original documents according to the similarity between the document vectors to form a plurality of document clusters; Setting the dynamic priority of the original documents in each document cluster according to the update frequency and / or query frequency of the original documents in the document cluster; Establishing an index structure for the document vectors, and adjusting the order of the document vectors in the index structure according to the dynamic priority of the original document; In response to a user initiating a query, the user's query is converted into a query vector, and the original document related to the user's query is obtained by calculating the similarity between the query vector and the document vector in the index structure after the document vector is sorted.

[0019] The above-mentioned file rapid retrieval method, product, computer device and storage medium can effectively improve the retrieval efficiency by vectorizing the text content in the original document to generate a document vector, and then clustering it to form multiple document clusters, and setting a dynamic priority for the original document in each document cluster to adjust the document sorting of the index structure. It can quickly find documents with high dynamic priority in high concurrency or high load conditions, thereby reducing system response delays or resource consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 This is an application environment diagram of a method for rapid file retrieval in one embodiment of the present application; Figure 2 This is a flowchart of a method for rapid file retrieval in one embodiment of the present application; Figure 3 This is a structural block diagram of a file quick retrieval device in one embodiment of the present application; Figure 4 This is a diagram of the internal structure of a computer device in one embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0023] As described in the background technology, with the acceleration of the digitalization process, the massive data generated by various industries has put forward higher requirements for file retrieval technology. Whether in the fields of e-commerce, medical care, finance, or in applications such as academic research and social media, users have an increasing demand for fast, efficient and accurate file retrieval. File retrieval technology not only needs to process huge amounts of data, but also needs to deal with diverse and unstructured data sources such as text, images, audio, etc. In order to ensure that users can quickly obtain the required information, the retrieval system must have an efficient indexing and sorting mechanism that can adapt to the dynamically changing data environment and continuously optimize the user experience. This makes the construction of a file retrieval technology solution with intelligent optimization and dynamic adjustment capabilities a key research direction in the current information technology field.

[0024] Existing file retrieval technologies mainly rely on traditional algorithms such as inverted indexing and keyword matching, combined with natural language processing (NLP) technology, to achieve document and query matching through vectorized representation. At the same time, based on user behavior feedback mechanisms and machine learning technology, existing systems can optimize document sorting through indicators such as click-through rate and access frequency. However, as the scale of data continues to increase, existing technologies often need to rely on distributed architectures to process massive amounts of data, and in high concurrency situations, resource scheduling and load balancing become key challenges. Some modern systems have introduced deep learning models to enhance the accuracy of file retrieval through semantic understanding and contextual association, providing a more intelligent search experience.

[0025] Although existing technologies have made significant progress in the field of document retrieval, they still have many shortcomings when dealing with dynamic data and coping with resource-constrained or high-load environments. When facing complex queries and large-scale data, existing systems cannot balance the timeliness of index updates and the accuracy of retrieval results. In addition, existing load balancing strategies often fail to handle node overloads in a timely manner, resulting in system performance bottlenecks. The sorting optimization mechanism based on user feedback also lacks sufficient flexibility, making it difficult to dynamically adjust the model to adapt to changing user needs.

[0026] To solve the above problems, a method for rapid file retrieval is creatively proposed in the embodiment of the present invention. The improvement goal is to optimize the document retrieval and sorting process by introducing intelligent weight control, task reallocation and dynamic resource adjustment mechanisms. By combining the logical judgment mechanism with the mathematical model, the system can not only adapt to resource constraints in different scenarios, but also give priority to the completion of important tasks in a high-load environment, thereby improving the overall system response speed and user experience.

[0027] The document rapid retrieval method provided by this application can be applied to Figure 1 In the application environment shown in the figure, the terminal 102 communicates with the server 104 through the network. The terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices, and the server 104 can be implemented as an independent server or a server cluster composed of multiple servers. The server 104 provides a fast file retrieval service, and the terminal 102 initiates a query to find the original document.

[0028] In one embodiment, Figure 2 As shown, a method for rapid file retrieval is provided, which is applied to Figure 1 Taking the server 104 in the example as an example, the following steps are included: Step S1, vectorizing the text content in the original document to generate a document vector; Step S2, clustering and grouping the original documents into multiple document clusters according to the similarity between the document vectors; Step S3, setting the dynamic priority of the original documents in each document cluster according to the update frequency and / or query frequency of the original documents in the document cluster; Step S4, establishing an index structure for the document vectors, and adjusting the order of the document vectors in the index structure according to the dynamic priority of the original document; Step S5, in response to a user initiating a query, the user's query is converted into a query vector, and the original document related to the user's query is obtained by calculating the similarity between the query vector and the document vector in the index structure after the document vector is sorted.

[0029] Among them, the text content in the original document is vectorized to generate a document vector, and then clustered to form multiple document clusters. Dynamic priorities are set for the original documents in each document cluster to adjust the document sorting of the index structure. This can effectively improve the retrieval efficiency and quickly find documents with high dynamic priorities in high concurrency or high load conditions, reducing system response delays or resource consumption.

[0030] In one embodiment, vectorizing the text content in the original document to generate a document vector includes: Obtaining a weighted value of each target word in the original document according to the word frequency of the target word in the target document, the total number of documents in the original document, and the number of documents containing the target word, wherein the target word is a word in the original document, and the target document is a document in the original document; Get the weighted value of each word in the original document and convert it into a vector representation to form a document vector.

[0031] Specifically, the TF-IDF algorithm is used to weight each word in the original document. The weighting method is: ,in Yes word In the documentation The word frequency in is the total number of documents, Contains words The number of documents; by Convert each original document into a vector representation to generate a document vector .

[0032] In one embodiment, clustering and grouping original documents to form multiple document clusters according to the similarity between document vectors includes: The K-means clustering algorithm is used to cluster the original documents according to the similarity between document vectors. The goal of clustering the original documents is to minimize the size of each document vector. The distance to the center of the cluster ; where S represents a document cluster, represents the i-th document cluster, is the center of the cluster.

[0033] The clustered documents are classified according to similarity, which provides a basis for subsequent indexing and query optimization. Of course, when the document vector cannot be well classified or the system resources are tight, the system will adjust the processing according to the document weight and clustering results, giving priority to optimizing those frequently accessed documents or more relevant documents.

[0034] Specifically, in a document retrieval system, documents are first vectorized for clustering and indexing. However, in some cases, document vectors may not be well classified, or system resources are limited and cannot process all documents. Therefore, the system prioritizes documents that are frequently accessed or highly relevant to the query based on the document weights and clustering results.

[0035] In one embodiment, setting the dynamic priority of the original documents in each document cluster according to the update frequency and / or query frequency of the original documents in the document cluster includes: The update frequency and query frequency of the original document are obtained, and a dynamic priority is assigned to the original document by weighted summing of the update frequency and the query frequency.

[0036] Specifically, get the update frequency of the original document and query frequency , assign a dynamic priority to the original document ,in , is the weight coefficient for adjusting the update frequency, It is the weight coefficient that adjusts the impact of query frequency.

[0037] In one embodiment, in response to a user initiating a query, converting the user's query into a query vector, and obtaining original documents related to the user's query by calculating the similarity between the query vector and the document vector in the index structure after the document vector is sorted includes: Get the query initiated by the user and convert the user's query into a query vector; Get the document vector in the index structure, and calculate the similarity between the query vector and the document vector in the index structure after sorting the document vector by cosine similarity to measure the relevance of the user's query and the original document.

[0038] Specifically, through cosine similarity Calculating the query vector and document vectors in the index structure The similarity is used to measure the relevance between the user's query and the original document.

[0039] Among them, through cosine similarity It can accurately measure the similarity between user queries and documents. The documents with higher similarity will be returned first by the retrieval system.

[0040] In one embodiment, the method for rapid file retrieval further includes: The ideal ranking after user click feedback is obtained, and the document vector ranking in the index structure is adjusted in real time by minimizing the error of document ranking.

[0041] Specifically, by minimizing the error of document ranking Adjust the document vector sorting in the index structure in real time; is the current sort, is the ideal ranking after the user clicks on the feedback, and n is the number of retrieval tasks.

[0042] In one embodiment, the method for rapid file retrieval further includes: Obtain the total computing resources for executing the index structure and the computing resources required to execute each retrieval task, and set the sum of the computing resources consumed by executing the retrieval task to be less than or equal to the total computing resources for executing the index structure; Get the remaining computing resources; In response to the remaining computing resources being less than or equal to the first threshold, setting a priority weight of the corresponding retrieval task according to the dynamic priority of the original document, and allocating computing resources in descending order of the priority weights of the retrieval tasks; In response to the remaining computing resources being greater than a first threshold, retrieval tasks whose priority weights are less than a second threshold are deleted to obtain the remaining computing resources after trimming the low priority tasks, so that retrieval tasks whose priority weights are greater than or equal to the second threshold are allocated to the remaining computing resources after trimming the low priority tasks.

[0043] Specifically, the total computing resources for executing the index structure are set to R, and each retrieval task is executed The computing resources required are , set the computing resource allocation to meet , where n is the number of retrieval tasks; Get remaining computing resources ; In response to the remaining computing resources being less than or equal to the first threshold, the original document is sorted according to its dynamic priority. Set the corresponding search task Priority weight , according to the retrieval task Priority weight Allocate computing resources in descending order; In response to the remaining computing resources being greater than a first threshold, deleting the priority weight The retrieval tasks with a priority lower than the second threshold obtain the remaining computing resources after cutting the low-priority tasks, so that the priority weight Retrieval tasks with a priority greater than or equal to the second threshold are allocated to the remaining computing resources after the low-priority tasks are trimmed.

[0044] It can be understood that n is the number of retrieval tasks executed, which corresponds to the total number of resources consumed by executing retrieval tasks. In the case of limited resources, the system will give priority to those retrieval tasks with higher priority. By optimizing resource allocation through linear programming, it can be ensured that tasks with higher priority are allocated more resources: ;in, Indicates the priority weight of the task. The higher the priority of the task, the more computing resources will be allocated to ensure a faster response speed. When resources are exhausted, the system will cut low-priority tasks to ensure that high-priority tasks can be executed smoothly under limited resources. The priority weight of the task Dynamic priority with original document Positively correlated, can be set preferably = .

[0045] In one embodiment, deleting the retrieval tasks whose priority weight is less than the second threshold to obtain the remaining computing resources after trimming the low priority tasks includes: Acquire computing resources corresponding to retrieval tasks whose priority weights are less than a second threshold; The low priority tasks are trimmed by subtracting the computing resources corresponding to the retrieval tasks whose priority weights are less than the second threshold from the total computing resources of the execution index structure, and the remaining computing resources after trimming the low priority tasks are obtained.

[0046] Specifically, get the priority weight The computing resources corresponding to the search tasks less than the second threshold ,in Priority weight The total number of computing resources corresponding to the search tasks that are less than the second threshold, is the weight of each priority The computing resources corresponding to the search tasks that are less than the second threshold; pass Get the remaining computing resources after pruning low-priority tasks.

[0047] Among them, when resources are exhausted, the system will cut low-priority tasks to ensure that high-priority tasks can be executed smoothly under limited resources.

[0048] Specifically, when resources are extremely tight, the system may not be able to complete all tasks. At this time, the system will cut low-priority tasks to ensure that high-priority tasks can be executed smoothly.

[0049] Assume that system resources (such as CPU and memory) are exhausted and cannot process all query tasks at the same time. The system will prune low-priority tasks to ensure that high-priority tasks are processed. The task pruning strategy can be The system can ensure that key tasks can be completed successfully when resources are limited.

[0050] In other embodiments, in order to avoid the problem of low retrieval efficiency and insufficient computing resources caused by too many document vectors in the index structure, the method for fast file retrieval also includes: The index structure is divided into a first sub-search structure and a second sub-search structure according to the dynamic priority, wherein the dynamic priority of the original document corresponding to the document vector in the first sub-search structure is greater than the dynamic priority of the original document corresponding to the document vector in the second sub-search structure; In response to a user initiating a query, searching the first sub-search structure and the second sub-search structure in order to obtain original documents related to the user's query; When retrieving original documents related to the user's query in the first sub-search structure, the control does not allocate computing resources to the second sub-search structure; when retrieving original documents related to the user's query in the second sub-search structure, the control does not allocate computing resources to the first sub-search structure.

[0051] Among them, by searching and obtaining original documents related to the user's query in the first sub-search structure and the second sub-search structure in sequence, documents with high access rates can be quickly obtained, and at the same time, effective allocation of computing resources can ensure that sufficient computing resources are provided for the retrieval.

[0052] In one embodiment, the method for rapid file retrieval further includes: Get the average query time for retrieving documents ; Get user click document Probability ,in , Represents the target document The relevance score of is the weight coefficient for adjusting the update frequency; Get the total computing resources R and resource utilization of the execution index structure ; The index structure is optimized by maximizing the system performance index M. The system performance index M is maximized as follows: ,in is the query frequency weight parameter, is the call rate weight parameter.

[0053] The function exp(x) represents the natural exponential function, that is, the exponential function with the real number e (e≈2.71828) as the base. It is the weight coefficient for adjusting the update frequency, which is a dynamically changing value; and yes Values ​​in different situations.

[0054] Periodically retrain the model and optimize the index structure. The specific iterative optimization goal is to Formula to maximize system performance indicators . is the average query time to retrieve a document, which the system wishes to minimize; The user clicks on the document The probability of the system improving document relevance through user behavior; It is the resource utilization of the system, optimizing resource allocation to improve computing efficiency.

[0055] The optimization process is dynamic, and the system parameters are adjusted in real time according to the user query mode, system load and the introduction of new documents to ensure the stability and continuous improvement of the retrieval effect. When user feedback (such as click-through rate) continues to be unsatisfactory, the system automatically initiates model retraining and adjusts the document sorting rules; when the system resource usage is too high, the thresholds for index updates and query scheduling are automatically adjusted to prioritize the execution of important tasks.

[0056] In one embodiment, the method for rapid file retrieval further includes: The update frequency of the index in the index structure is set to be positively correlated with the dynamic priority of the original document corresponding to the index; The priority threshold for index updates is obtained by dividing the document weight by the current node load. Performing index update on original documents whose dynamic priority is greater than the priority threshold of index update; The update constraints of the index in the index structure are ,in , The index update time. is the total update time, n is the number of retrieval tasks, The priority threshold for index updates.

[0057] Specifically, through Get the priority threshold for index updates ,in is the document weight, and L is the load of the current node. When the system load L increases, only the weight Higher than The documents will be updated with priority in the index to ensure the retrieval performance of key documents.

[0058] Among them, suppose the system has a large number of documents to process, but the resources are limited (such as memory or CPU usage is close to the upper limit), the system will face the situation where it cannot process all documents at the same time. At this time, the system will assign weights to the documents , the weight can be determined based on the following factors: the access frequency of the document , the importance of documentation (Based on user feedback or business rules). The document weight formula is: . is the document access frequency weight, The document importance weight. When resources are tight, the system will prioritize the documents with the highest weight. By weight sorting, the documents that users frequently access are prioritized for index optimization and classification, so that even if resources are limited, users can still retrieve important documents in a timely manner.

[0059] The higher the dynamic priority of a document, the more frequently its index is updated. , and ensure that the total update time does not exceed , the following constraints are designed: ,in Only documents with a higher dynamic priority will be updated first, thus ensuring that the system will not occupy too many resources due to the update of low-priority documents. Of course, when the system load is high, the index update of low-priority documents will be temporarily ignored, and the system will automatically adjust the value to ensure that the index of key documents is updated in time.

[0060] Specifically, the system may encounter high load when processing queries, especially when there are a large number of concurrent queries or a large number of document updates. In order to ensure the retrieval effect of key documents, the system will temporarily ignore the index update of low-priority documents and automatically adjust parameters to ensure that the index of important documents is updated in time.

[0061] Assume that the system load is high, resulting in the delay of index update of low priority documents. In this case, the system's priority adjustment mechanism will determine which documents should be updated first based on the access frequency and weight of the documents. Set as the index update threshold of the document, and documents below it will be temporarily ignored.

[0062] In one embodiment, the method for rapid file retrieval further includes: The click-through rate of a document is obtained by dividing the number of times the document is clicked by the number of times the document is displayed. ; Get the priority threshold for index updates ,when and / or when When , adjust the order of the i-th document in the index structure; is the total number of documents.

[0063] Among them, through Get the click rate of the i-th document ,in is the number of times the i-th document is clicked, is the number of times the i-th document is displayed.

[0064] Furthermore, when the click rate of a document Continued to be below the set threshold , the system will automatically retrain the ranking model to further improve the relevance of recommended documents. The system will optimize the ranking of documents based on user behavior (such as document click-through rate). Continued to be below the set threshold , the system will trigger the retraining of the ranking model to improve the relevance of document recommendations.

[0065] User feedback (such as click-through rate) is an important indicator of the system's retrieval quality. When the feedback is not ideal, the system will automatically initiate model retraining and adjust the document sorting rules to improve the relevance of the retrieval results and user satisfaction. If the user is not satisfied with the results returned by the system after multiple queries (low click-through rate), the system will automatically adjust the sorting model and relearn user preferences. The trigger condition for model retraining can be set by Formula judgment.

[0066] In one embodiment, the method for rapid file retrieval further includes: In a distributed document retrieval system, multiple nodes are used to jointly execute an index structure to process document retrieval tasks; The load of the target node is adjusted by minimizing the load variance.

[0067] Specifically, by minimizing the load variance To adjust the load of the jth node ; Where m is the total number of nodes.

[0068] By minimizing the load variance Ensure that the load of each node is evenly distributed to prevent some nodes from being overloaded and causing performance bottlenecks. Exceeds the set maximum threshold When the system is busy, it will redistribute tasks to avoid computing bottlenecks.

[0069] Specifically, in a distributed document retrieval system, multiple nodes jointly process document query tasks. Exceeds the maximum load threshold set by the system When the system is busy, it will automatically redistribute tasks to prevent a certain node from becoming a computing bottleneck.

[0070] Assume that the load of the jth node is Exceeds the threshold set by the system At this point, the system will transfer some tasks on this node to other nodes.

[0071] In one embodiment, the method for rapid file retrieval further includes: Monitor the load of nodes in a distributed file retrieval system; In response to the load of the target node being greater than the maximum load threshold, the retrieval task is redistributed to reduce the load of the target node after the task is redistributed.

[0072] Specifically, when the target node is the jth node, monitor the load of the jth node in the distributed file retrieval system ; In response to the load of the jth node Greater than the maximum load threshold When , the retrieval task is redistributed, and the load of the jth node after the task is redistributed is ,in is the load of the jth node, is the maximum load threshold, The number of target nodes to be allocated for performing the retrieval task, is the load portion of the jth node that is overloaded. Through this mechanism, the load of each node can be effectively balanced to prevent a node from becoming a performance bottleneck.

[0073] In the above-mentioned fast file retrieval method, the text content in the original document is vectorized to generate a document vector, which is then clustered to form multiple document clusters. Dynamic priorities are set for the original documents in each document cluster to adjust the document sorting of the index structure. This can effectively improve the retrieval efficiency, and can quickly find documents with high dynamic priorities in high concurrency or high load conditions, thereby reducing system response delays or resource consumption.

[0074] The present invention can automatically adjust the index update strategy when the system load is high, give priority to documents with high weight (such as high user click rate and access frequency), and improve the retrieval speed of key documents. This dynamic adjustment capability significantly improves the efficiency of the system in a large-scale document retrieval environment. The solution introduces a document weight calculation method based on user behavior (such as click rate and access frequency) to ensure that documents with high relevance and high access frequency are recommended and retrieved first. At the same time, when the click rate of certain documents is continuously lower than the threshold, the system will automatically retrain the sorting model to make the retrieval results more in line with user needs, enhance user experience and retrieval accuracy; the solution provides a load balancing and resource adaptation module that can monitor the load of the system in real time. When the load of a node exceeds the set threshold, the system can dynamically adjust the priority and allocation of tasks to avoid computing bottlenecks in certain nodes. This mechanism enables the system to run smoothly under high concurrency and ensure the priority execution of key tasks; the solution also designs a mechanism for automatically cutting low-priority tasks when system resources are tight to ensure that high-priority tasks can be executed normally under limited resources. Compared with traditional systems, it can manage and allocate resources more efficiently and improve the stability and execution efficiency of the system under resource-constrained conditions.

[0075] In one embodiment, Figure 3 As shown, a file rapid retrieval system 10 is provided, including: a document preprocessing module 1, a document clustering and weight calculation module 2, an index construction and optimization module 3, a task scheduling and load balancing module 4, a feedback and model optimization module 5 and a system monitoring and adaptive optimization module 6.

[0076] Document preprocessing module 1 is responsible for vectorizing and selecting features of the original document. It is the basic part of the entire retrieval system. This module converts text content into vector representation, generates document vectors using TF-IDF natural language processing technology, and then performs dimensionality reduction on the document vectors to reduce redundant information while retaining the most discriminative features. Document clustering and weight calculation module 2 is responsible for clustering and weight calculation of documents to ensure that similar documents can be quickly retrieved and important documents can be displayed first. This module uses a clustering algorithm to cluster document vectors and classify them based on the similarity of document content. It dynamically calculates the weight of documents based on their historical behaviors such as click-through rate and access frequency. Documents with higher weights are given priority in indexing and retrieval. Index building and optimization module 3 is responsible for building an inverted index, dynamically adjusting the index structure according to the priority and weight of the document, and building a retrieval structure based on the inverted index to quickly locate relevant documents. Documents with high weights are indexed first to ensure fast response of high-priority documents. The system dynamically updates the index according to the load. If the system load is high, the index update of low-priority documents will be temporarily ignored to prioritize the index update and retrieval speed of high-priority documents. The task scheduling and load balancing module 4 is responsible for scheduling tasks and allocating resources when the system load changes, ensuring that the system can run smoothly under high load conditions. According to the weight and importance of the documents, the index update and retrieval tasks of high-priority documents are prioritized. When system resources are tight, low-priority tasks can be postponed, and the load of each node is monitored in real time. When the load of a node exceeds the threshold, the system will automatically reallocate tasks to avoid computing bottlenecks; Feedback and model optimization module 5 continuously optimizes the sorting model based on user behavior data to ensure the relevance and accuracy of document recommendations, and dynamically adjusts the weight of documents based on indicators such as user click behavior and access frequency. If the click rate of a document continues to be lower than the set threshold, the system will trigger model retraining and optimize the sorting algorithm. The system regularly retrains the document sorting model based on user feedback and optimizes the order of document recommendations to ensure high relevance and user experience. At the same time, when resources are tight, low-priority tasks will be cut to ensure the execution of high-priority tasks; The system monitoring and adaptive optimization module 6 is responsible for the overall health monitoring and adaptive optimization of the system, and monitors the system resource usage in real time. When the system load is high, the priority and execution order of tasks are adjusted to ensure that key tasks are not affected. When some tasks cannot be completed effectively for a long time, the system will automatically adjust the task weight, scheduling strategy or retrain the sorting model to improve the overall efficiency of the system and user experience.

[0077] The entire system realizes an intelligent and efficient document retrieval system through the collaborative work of modules such as document preprocessing, clustering and weight calculation, index optimization, task scheduling, user feedback and model optimization. Each module is interrelated and works in coordination to ensure that in a high-load, resource-constrained environment, the system can still give priority to the smooth completion of important tasks, while continuously improving the user experience and the relevance of document recommendations.

[0078] In the above-mentioned rapid file retrieval system, the text content in the original document is vectorized to generate a document vector, which is then clustered to form multiple document clusters. Dynamic priorities are set for the original documents in each document cluster to adjust the document sorting of the index structure. This can effectively improve the retrieval efficiency and quickly find documents with high dynamic priorities in high concurrency or high load conditions, thereby reducing system response delays or resource consumption.

[0079] The specific definition of the file rapid retrieval device can be found in the definition of the file rapid retrieval method above, which will not be repeated here. Each module in the above-mentioned file rapid retrieval device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0080] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store file fast retrieval data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for fast file retrieval is implemented.

[0081] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0082] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps: Vectorize the text content in the original document to generate a document vector; Cluster the original documents according to the similarity between document vectors to form multiple document clusters; Setting the dynamic priority of the original documents in each document cluster according to the update frequency and / or query frequency of the original documents in the document cluster; Establish an index structure for the document vectors, and adjust the order of the document vectors in the index structure according to the dynamic priority of the original documents; In response to a user initiating a query, the user's query is converted into a query vector, and the original document related to the user's query is obtained by calculating the similarity between the query vector and the document vector in the index structure after the document vector is sorted.

[0083] In one embodiment, vectorizing the text content in the original document to generate a document vector includes: Obtaining a weighted value of each target word in the original document according to the word frequency of the target word in the target document, the total number of documents in the original document, and the number of documents containing the target word, wherein the target word is a word in the original document, and the target document is a document in the original document; Get the weighted value of each word in the original document and convert it into a vector representation to form a document vector.

[0084] In one embodiment, setting the dynamic priority of the original documents in each document cluster according to the update frequency and / or query frequency of the original documents in the document cluster includes: The update frequency and query frequency of the original document are obtained, and a dynamic priority is assigned to the original document by weighted summing of the update frequency and the query frequency.

[0085] In one embodiment, in response to a user initiating a query, converting the user's query into a query vector, and obtaining original documents related to the user's query by calculating the similarity between the query vector and the document vector in the index structure after the document vector is sorted includes: Get the query initiated by the user and convert the user's query into a query vector; Get the document vector in the index structure, and calculate the similarity between the query vector and the document vector in the index structure after sorting the document vector by cosine similarity to measure the relevance of the user's query and the original document.

[0086] In one embodiment, the method for rapid file retrieval further includes: The ideal ranking after user click feedback is obtained, and the document vector ranking in the index structure is adjusted in real time by minimizing the error of document ranking.

[0087] In one embodiment, the method for rapid file retrieval further includes: Obtain the total computing resources for executing the index structure and the computing resources required to execute each retrieval task, and set the sum of the computing resources consumed by executing the retrieval task to be less than or equal to the total computing resources for executing the index structure; Get the remaining computing resources; In response to the remaining computing resources being less than or equal to the first threshold, setting a priority weight of the corresponding retrieval task according to the dynamic priority of the original document, and allocating computing resources in descending order of the priority weights of the retrieval tasks; In response to the remaining computing resources being greater than a first threshold, retrieval tasks whose priority weights are less than a second threshold are deleted to obtain the remaining computing resources after trimming the low priority tasks, so that retrieval tasks whose priority weights are greater than or equal to the second threshold are allocated to the remaining computing resources after trimming the low priority tasks.

[0088] In one embodiment, deleting the retrieval tasks whose priority weight is less than the second threshold to obtain the remaining computing resources after trimming the low priority tasks includes: Acquire computing resources corresponding to retrieval tasks whose priority weights are less than a second threshold; The low priority tasks are trimmed by subtracting the computing resources corresponding to the retrieval tasks whose priority weights are less than the second threshold from the total computing resources of the execution index structure, and the remaining computing resources after trimming the low priority tasks are obtained.

[0089] In one embodiment, the method for rapid file retrieval further includes: Get the average query time for retrieving documents ; Get user click document Probability ,in , Represents the target document The relevance score of is the weight coefficient for adjusting the update frequency; Get the total computing resources R and resource utilization of the execution index structure ; The index structure is optimized by maximizing the system performance index M. The system performance index M is maximized as follows: ,in is the query frequency weight parameter, is the call rate weight parameter.

[0090] In one embodiment, the method for rapid file retrieval further includes: The update frequency of the index in the index structure is set to be positively correlated with the dynamic priority of the original document corresponding to the index; The priority threshold for index updates is obtained by dividing the document weight by the current node load. Performing index update on original documents whose dynamic priority is greater than the priority threshold of index update; The update constraints of the index in the index structure are ,in , The index update time. is the total update time, n is the number of retrieval tasks, The priority threshold for index updates.

[0091] In one embodiment, the method for rapid file retrieval further includes: The click-through rate of a document is obtained by dividing the number of times the document is clicked by the number of times the document is displayed. ; Get the priority threshold for index updates ,when and / or when When , adjust the order of documents in the index structure; is the total number of documents.

[0092] In one embodiment, the method for rapid file retrieval further includes: In a distributed document retrieval system, multiple nodes are used to jointly execute an index structure to process document retrieval tasks; The load of the target node is adjusted by minimizing the load variance.

[0093] In one embodiment, the method for rapid file retrieval further includes: Monitor the load of nodes in a distributed file retrieval system; In response to the load of the target node being greater than the maximum load threshold, the retrieval task is redistributed to reduce the load of the target node after the task is redistributed.

[0094] For the specific limitations on the steps implemented when the computer program is executed by the processor, please refer to the limitations on the method for rapid file retrieval above, which will not be repeated here.

[0095] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: Vectorize the text content in the original document to generate a document vector; Cluster the original documents according to the similarity between document vectors to form multiple document clusters; Setting the dynamic priority of the original documents in each document cluster according to the update frequency and / or query frequency of the original documents in the document cluster; Establish an index structure for the document vectors, and adjust the order of the document vectors in the index structure according to the dynamic priority of the original documents; In response to a user initiating a query, the user's query is converted into a query vector, and the original document related to the user's query is obtained by calculating the similarity between the query vector and the document vector in the index structure after the document vector is sorted.

[0096] In one embodiment, vectorizing the text content in the original document to generate a document vector includes: Obtaining a weighted value of each target word in the original document according to the word frequency of the target word in the target document, the total number of documents in the original document, and the number of documents containing the target word, wherein the target word is a word in the original document, and the target document is a document in the original document; Get the weighted value of each word in the original document and convert it into a vector representation to form a document vector.

[0097] In one embodiment, setting the dynamic priority of the original documents in each document cluster according to the update frequency and / or query frequency of the original documents in the document cluster includes: The update frequency and query frequency of the original document are obtained, and a dynamic priority is assigned to the original document by weighted summing of the update frequency and the query frequency.

[0098] In one embodiment, in response to a user initiating a query, converting the user's query into a query vector, and obtaining original documents related to the user's query by calculating the similarity between the query vector and the document vector in the index structure after the document vector is sorted includes: Get the query initiated by the user and convert the user's query into a query vector; Get the document vector in the index structure, and calculate the similarity between the query vector and the document vector in the index structure after sorting the document vector by cosine similarity to measure the relevance of the user's query and the original document.

[0099] In one embodiment, the method for rapid file retrieval further includes: The ideal ranking after user click feedback is obtained, and the document vector ranking in the index structure is adjusted in real time by minimizing the error of document ranking.

[0100] In one embodiment, the method for rapid file retrieval further includes: Obtain the total computing resources for executing the index structure and the computing resources required to execute each retrieval task, and set the sum of the computing resources consumed by executing the retrieval task to be less than or equal to the total computing resources for executing the index structure; Get the remaining computing resources; In response to the remaining computing resources being less than or equal to the first threshold, setting a priority weight of the corresponding retrieval task according to the dynamic priority of the original document, and allocating computing resources in descending order of the priority weights of the retrieval tasks; In response to the remaining computing resources being greater than a first threshold, retrieval tasks whose priority weights are less than a second threshold are deleted to obtain the remaining computing resources after trimming the low priority tasks, so that retrieval tasks whose priority weights are greater than or equal to the second threshold are allocated to the remaining computing resources after trimming the low priority tasks.

[0101] In one embodiment, deleting the retrieval tasks whose priority weight is less than the second threshold to obtain the remaining computing resources after trimming the low priority tasks includes: Acquire computing resources corresponding to retrieval tasks whose priority weights are less than a second threshold; The low priority tasks are trimmed by subtracting the computing resources corresponding to the retrieval tasks whose priority weights are less than the second threshold from the total computing resources of the execution index structure, and the remaining computing resources after trimming the low priority tasks are obtained.

[0102] In one embodiment, the method for rapid file retrieval further includes: Get the average query time for retrieving documents ; Get user click document Probability ,in , Represents the target document The relevance score of is the weight coefficient for adjusting the update frequency; Get the total computing resources R and resource utilization of the execution index structure ; The index structure is optimized by maximizing the system performance index M. The system performance index M is maximized as follows: ,in is the query frequency weight parameter, is the call rate weight parameter.

[0103] In one embodiment, the method for rapid file retrieval further includes: The update frequency of the index in the index structure is set to be positively correlated with the dynamic priority of the original document corresponding to the index; The priority threshold for index updates is obtained by dividing the document weight by the current node load. Performing index update on original documents whose dynamic priority is greater than the priority threshold of index update; The update constraints of the index in the index structure are ,in , The index update time. is the total update time, n is the number of retrieval tasks, The priority threshold for index updates.

[0104] In one embodiment, the method for rapid file retrieval further includes: The click-through rate of a document is obtained by dividing the number of times the document is clicked by the number of times the document is displayed. ; Get the priority threshold for index updates ,when and / or when When , adjust the order of documents in the index structure; is the total number of documents.

[0105] In one embodiment, the method for rapid file retrieval further includes: In a distributed document retrieval system, multiple nodes are used to jointly execute an index structure to process document retrieval tasks; The load of the target node is adjusted by minimizing the load variance.

[0106] In one embodiment, the method for rapid file retrieval further includes: Monitor the load of nodes in a distributed file retrieval system; In response to the load of the target node being greater than the maximum load threshold, the retrieval task is redistributed to reduce the load of the target node after the task is redistributed.

[0107] For the specific limitations on the steps implemented when the processor executes the computer program, please refer to the limitations on the method for rapid file retrieval above, which will not be repeated here.

[0108] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: Vectorize the text content in the original document to generate a document vector; Cluster the original documents according to the similarity between document vectors to form multiple document clusters; Setting the dynamic priority of the original documents in each document cluster according to the update frequency and / or query frequency of the original documents in the document cluster; Establish an index structure for the document vectors, and adjust the order of the document vectors in the index structure according to the dynamic priority of the original documents; In response to a user initiating a query, the user's query is converted into a query vector, and the original document related to the user's query is obtained by calculating the similarity between the query vector and the document vector in the index structure after the document vector is sorted.

[0109] In one embodiment, vectorizing the text content in the original document to generate a document vector includes: Obtaining a weighted value of each target word in the original document according to the word frequency of the target word in the target document, the total number of documents in the original document, and the number of documents containing the target word, wherein the target word is a word in the original document, and the target document is a document in the original document; Get the weighted value of each word in the original document and convert it into a vector representation to form a document vector.

[0110] In one embodiment, setting the dynamic priority of the original documents in each document cluster according to the update frequency and / or query frequency of the original documents in the document cluster includes: The update frequency and query frequency of the original document are obtained, and a dynamic priority is assigned to the original document by weighted summing of the update frequency and the query frequency.

[0111] In one embodiment, in response to a user initiating a query, converting the user's query into a query vector, and obtaining original documents related to the user's query by calculating the similarity between the query vector and the document vector in the index structure after the document vector is sorted includes: Get the query initiated by the user and convert the user's query into a query vector; Get the document vector in the index structure, and calculate the similarity between the query vector and the document vector in the index structure after sorting the document vector by cosine similarity to measure the relevance of the user's query and the original document.

[0112] In one embodiment, the method for rapid file retrieval further includes: The ideal ranking after user click feedback is obtained, and the document vector ranking in the index structure is adjusted in real time by minimizing the error of document ranking.

[0113] In one embodiment, the method for rapid file retrieval further includes: Obtain the total computing resources for executing the index structure and the computing resources required to execute each retrieval task, and set the sum of the computing resources consumed by executing the retrieval task to be less than or equal to the total computing resources for executing the index structure; Get the remaining computing resources; In response to the remaining computing resources being less than or equal to the first threshold, setting a priority weight of the corresponding retrieval task according to the dynamic priority of the original document, and allocating computing resources in descending order of the priority weights of the retrieval tasks; In response to the remaining computing resources being greater than a first threshold, retrieval tasks whose priority weights are less than a second threshold are deleted to obtain the remaining computing resources after trimming the low priority tasks, so that retrieval tasks whose priority weights are greater than or equal to the second threshold are allocated to the remaining computing resources after trimming the low priority tasks.

[0114] In one embodiment, deleting the retrieval tasks whose priority weight is less than the second threshold to obtain the remaining computing resources after trimming the low priority tasks includes: Acquire computing resources corresponding to retrieval tasks whose priority weights are less than a second threshold; The low priority tasks are trimmed by subtracting the computing resources corresponding to the retrieval tasks whose priority weights are less than the second threshold from the total computing resources of the execution index structure, and the remaining computing resources after trimming the low priority tasks are obtained.

[0115] In one embodiment, the method for rapid file retrieval further includes: Get the average query time for retrieving documents ; Get user click document Probability ,in , Represents the target document The relevance score of is the weight coefficient for adjusting the update frequency; Get the total computing resources R and resource utilization of the execution index structure ; The index structure is optimized by maximizing the system performance index M. The system performance index M is maximized as follows: ,in is the query frequency weight parameter, is the call rate weight parameter.

[0116] In one embodiment, the method for rapid file retrieval further includes: The update frequency of the index in the index structure is set to be positively correlated with the dynamic priority of the original document corresponding to the index; The priority threshold for index updates is obtained by dividing the document weight by the current node load. Performing index update on original documents whose dynamic priority is greater than the priority threshold of index update; The update constraints of the index in the index structure are ,in , The index update time. is the total update time, n is the number of retrieval tasks, The priority threshold for index updates.

[0117] In one embodiment, the method for rapid file retrieval further includes: The click-through rate of a document is obtained by dividing the number of times the document is clicked by the number of times the document is displayed. ; Get the priority threshold for index updates ,when and / or when When , adjust the order of documents in the index structure; is the total number of documents.

[0118] In one embodiment, the method for rapid file retrieval further includes: In a distributed document retrieval system, multiple nodes are used to jointly execute an index structure to process document retrieval tasks; The load of the target node is adjusted by minimizing the load variance.

[0119] In one embodiment, the method for rapid file retrieval further includes: Monitor the load of nodes in a distributed file retrieval system; In response to the load of the target node being greater than the maximum load threshold, the retrieval task is redistributed to reduce the load of the target node after the task is redistributed.

[0120] For the specific limitations on the steps implemented when the computer program is executed by the processor, please refer to the limitations on the method for rapid file retrieval above, which will not be repeated here.

[0121] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0122] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0123] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for rapid file retrieval, characterized in that: include: Vectorize the text content in the original document to generate a document vector; Clustering and grouping the original documents according to the similarity between the document vectors to form a plurality of document clusters; Setting a dynamic priority of the original documents in each document cluster according to the update frequency and / or query frequency of the original documents in the document cluster; Establishing an index structure for the document vectors, and adjusting the order of the document vectors in the index structure according to the dynamic priority of the original document; In response to a user initiating a query, the user's query is converted into a query vector, and the original document related to the user's query is obtained by calculating the similarity between the query vector and the document vector in the index structure after the document vector is sorted.

2. The method for rapid file retrieval according to claim 1, characterized in that: The step of vectorizing the text content in the original document to generate a document vector comprises: Obtaining a weighted value of each target word in the original document according to the word frequency of the target word in the target document, the total number of documents in the original document, and the number of documents containing the target word, wherein the target word is a word in the original document, and the target document is a document in the original document; The weighted value of each target word in each target document in the original document is obtained and converted into a vector representation to form a document vector of the target document.

3. The method for rapid file retrieval according to claim 1, characterized in that: The step of setting the dynamic priority of the original documents in each document cluster according to the update frequency and / or query frequency of the original documents in the document cluster includes: The update frequency and query frequency of the original document are obtained, and a dynamic priority is assigned to the original document by weighted summing of the update frequency and the query frequency.

4. The method for rapid file retrieval according to claim 1, characterized in that: In response to a user initiating a query, converting the user's query into a query vector, and obtaining original documents related to the user's query by calculating the similarity between the query vector and the document vector in the index structure after the document vector is sorted includes: Get the query initiated by the user and convert the user's query into a query vector; The document vector in the index structure is obtained, and the similarity between the query vector and the document vector in the index structure after the document vector is sorted is calculated by cosine similarity to measure the relevance between the user's query and the original document.

5. The method for rapid file retrieval according to claim 1, characterized in that: The method for rapid file retrieval also includes: The ideal ranking after the user click feedback is obtained, and the document vector ranking in the index structure is adjusted in real time by minimizing the error of the document ranking.

6. The method for rapid file retrieval according to claim 1, characterized in that: The method for rapid file retrieval also includes: Obtain the total computing resources required to execute the index structure and the computing resources required to execute each retrieval task, and set the sum of the computing resources consumed to execute the retrieval task to be less than or equal to the total computing resources required to execute the index structure; Get the remaining computing resources; In response to the remaining computing resources being less than or equal to a first threshold, setting a priority weight of a corresponding retrieval task according to the dynamic priority of the original document, and allocating computing resources in descending order of the priority weights of the retrieval tasks; In response to the remaining computing resources being greater than a first threshold, retrieval tasks whose priority weights are less than a second threshold are deleted to obtain the remaining computing resources after trimming the low priority tasks, so that retrieval tasks whose priority weights are greater than or equal to the second threshold are allocated to the remaining computing resources after trimming the low priority tasks.

7. The method for rapid file retrieval according to claim 6, characterized in that: The step of deleting the search tasks whose priority weight is less than the second threshold value to obtain the remaining computing resources after cutting the low priority tasks includes: Acquire computing resources corresponding to retrieval tasks whose priority weights are less than a second threshold; The low priority tasks are trimmed by subtracting the computing resources corresponding to the retrieval tasks whose priority weights are less than the second threshold from the total computing resources of the executing index structure, thereby obtaining the remaining computing resources after trimming the low priority tasks.

8. The method for rapid file retrieval according to claim 1, characterized in that: The method for rapid file retrieval also includes: Get the average query time for retrieving documents ; Get the probability of a user clicking on a document ; Get the total computing resources R and resource utilization of the index structure ; The index structure is optimized by maximizing the system performance index M, where the maximizing system performance index M is: ,in is the query frequency weight parameter, is the call rate weight parameter.

9. The method for rapid file retrieval according to claim 1, characterized in that: The method for rapid file retrieval also includes: Setting the update frequency of the index in the index structure to be positively correlated with the dynamic priority of the original document corresponding to the index; The priority threshold for index updates is obtained by dividing the document weight by the current node load. Performing index update on original documents whose dynamic priority is greater than the priority threshold of index update; The update constraint of the index in the index structure is ,in , The index update time. is the total update time, n is the number of retrieval tasks, The priority threshold for index updates.

10. The method for rapid file retrieval according to claim 1, characterized in that: The method for rapid file retrieval also includes: The click-through rate of a document is obtained by dividing the number of times the document is clicked by the number of times the document is displayed. ; Get the priority threshold for index updates ,when and / or when When , adjusting the order of documents in the index structure; wherein is the total number of documents.

11. The method for rapid file retrieval according to claim 1, characterized in that: The method for rapid file retrieval also includes: In a distributed document retrieval system, multiple nodes are used to jointly execute the document retrieval task of processing the index structure; The load of the target node is adjusted by minimizing the load variance.

12. The method for rapid file retrieval according to claim 11, characterized in that: The method for rapid file retrieval also includes: Monitoring the load of nodes in the distributed file retrieval system; In response to the load of the target node being greater than the maximum load threshold, the retrieval task is redistributed to reduce the load of the target node after the task is redistributed.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

14. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 12 are implemented.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

Images