Multi-node data retrieval method, device and computer storage medium

Abstract

The present invention relates to a multi-node data retrieval method, device and storage medium. The method comprises: after normalizing the acquired retrieval tasks, scheduling the retrieval resources and retrieval tasks for each resource node through a preset scheduling buffer area, obtaining the corresponding retrieval engine of the resource node associated with the current business scenario, obtaining the retrieval results of the business scenario by adjusting the scheme and the retrieval engine, deduplicating and sorting the retrieval results, and fusing the retrieval results of each resource node after deduplication and sorting to obtain the return result set of the current business scenario. The present invention meets the needs of big data business analysis, solves the long-standing problems of inaccurate result display and low retrieval efficiency of multi-node massive data retrieval, and improves the efficiency and accuracy of multi-node data retrieval in complex business scenarios.

Description

Technical Field

[0001] The present invention relates to the field of data retrieval, and in particular to a multi-node data retrieval method, device and computer storage medium. Background Art

[0002] With the development of modern information technology and the increasing intelligence of human life, the volume of global data is expanding and increasing without limit. Data is typically stored in a highly reliable distributed manner, with even the same type of data being stored across multiple nodes. When big data applications need to retrieve relevant data, they typically use an "asynchronous search across different storage nodes, returning and displaying the results" approach to simultaneously retrieve massive amounts of data from multiple nodes.

[0003] Existing big data retrieval methods are inefficient, employing a single rule to perform a simultaneous and unified search across all nodes, without optimizing the retrieval scheme based on the characteristics of each storage node. This often results in the efficiency of any one node significantly impacting overall result display and business development, and increases retrieval resource overhead. After obtaining search results, displaying them often requires only a simple display, as the time and quantity of results returned by different storage nodes vary significantly. Furthermore, when viewing the next page of results, sorting errors and duplicate data often occur, severely impacting user business development and limiting the utilization of big data's value. Summary of the Invention

[0004] In order to solve the problems of low efficiency and accuracy of multi-node data retrieval and confusing result display in the above-mentioned prior art, the present invention provides a multi-node data retrieval method, comprising the following steps: S1, obtaining a retrieval task, normalizing the retrieval task, and determining the scene characteristics of the current business scene and the node characteristics of the resource nodes associated with the current business scene based on the normalized retrieval task, wherein the scene characteristics include: scene name, retrieval scheme corresponding to the current scene; the node characteristics include resource node ID, data volume, retrieval engine, and scene retrieval efficiency range; the scene retrieval efficiency range includes the average time it takes for the resource node to return results;

[0005] S2, allocating the first memory space in the preset scheduling buffer area to each resource node associated with the current business scenario in proportion and evenly, and allocating the second memory space in the scheduling buffer area to each associated resource node according to the average time for the associated resource node to return the result, wherein the memory space allocated to the resource node with a long average return result time accounts for a larger proportion than the resource node with a short average return result time, and the memory space allocated to the resource node is used to store the result returned by the corresponding resource node;

[0006] S3, based on the business scenario feature library pre-stored in the scheduling buffer area, determining whether the current business scenario is a new business scenario; when the current business scenario is a new business scenario, adjusting the number of returned results of the resource nodes according to the size of the memory space allocated to each resource node and the data volume of each resource node, increasing the number of returned results of resource nodes with large data volumes, obtaining an adjustment plan for the business scenario, and saving the adjustment plan as a newly created retrieval plan for the current business scenario in the business scenario feature library; wherein the business scenario feature library is pre-established based on the scenario features of historical business scenarios, and the business scenario feature library includes: a scenario name, a retrieval plan corresponding to the business scenario, and an adjustment plan;

[0007] S4, according to the adjustment plan and the resource node ID, obtaining a search engine corresponding to the resource node associated with the current business scenario from the node feature library pre-stored in the scheduling cache area, transferring the adjustment plan to the search engine, and having the search engine search each resource node to obtain a search result for the business scenario, wherein the node feature library is established based on the node features of historical resource nodes, and the node feature library includes: node ID, corresponding search engine, data volume, and scenario retrieval efficiency range;

[0008] S5, deduplication and sorting the search results, and merging the deduplication and sorted search results of each resource node to obtain a return result set for the current business scenario.

[0009] Based on the above solution, the present invention can also make the following improvements.

[0010] Furthermore, the scene name of the current business scenario is compared with the scene name in the business scenario feature library to confirm whether the current business scenario is a new business scenario; when the scene name and retrieval scheme of the current business scenario do not exist in the business scenario feature library, the current business scenario is confirmed to be a new business scenario; otherwise, it is a duplicate business scenario.

[0011] Furthermore, the adjustment plan includes a retrieval node ID, and S4 allocates a corresponding retrieval engine to the current business scenario according to the adjustment plan. In response to the resource node ID of the resource node associated with the current business scenario being the same as the retrieval node ID of the adjustment plan, the resource node is searched through the retrieval engine to obtain a retrieval result of the resource node.

[0012] Furthermore, the scene feature also includes a deduplication field, and S5 also includes: comparing the retrieval result with the returned result set, the scheduling cache area deduplicating the retrieval result and the returned result set according to the deduplication field, and fusing the retrieval result with the returned result set.

[0013] Furthermore, the retrieval scheme also includes a single-page display quantity, a current page number, and a sorting field, and the push status of the returned result set is marked according to the single-page display quantity and the current page number, wherein the push status includes pushed and not pushed; records with a push status of "not pushed" are selected from the returned result set for pushing and their push status is set to "pushed", and the returned result set records with a push status of "not pushed" are + sorted according to the sorting field and marked with a position label.

[0014] Furthermore, the historical record of the return result set of the business scenario, S3 also includes: when the business scenario is a repeated business scenario, the return result set of the business scenario is obtained according to the sorted historical record of the return result set and the position tag.

[0015] Furthermore, when the number of the returned result sets with a push status of "not pushed" is at least twice less than the single-page display quantity, the resource nodes associated with the business scenario are pre-searched through the retrieval scheme to retrieve the data resources of the business scenario in advance.

[0016] Furthermore, the business scenario feature library also includes the last retrieval time, and the method also includes: regularly detecting the last retrieval time. If the interval between the last retrieval time of the business scenario and the current system time exceeds a preset first threshold, the scheduling cache area clears the corresponding cache area according to the scene name in the business scenario feature library.

[0017] Furthermore, the node feature also includes a node status, which includes an available status and an unavailable status. The last retrieval time is regularly checked. If the interval between the last retrieval time of the retrieval scheme and the current system time exceeds a preset second threshold, the node status in the node feature is set to an unavailable status.

[0018] Another object of the present invention is to provide a multi-node data retrieval device, comprising a memory and a processor, wherein the memory stores at least one program, and the at least one program is executed by the processor to implement the above-mentioned multi-node data retrieval method.

[0019] Another object of the present invention is to provide a computer-readable storage medium, comprising a memory, wherein the memory stores a computer program, and when the computer program is executed by a processor, the multi-node data retrieval method described above is implemented.

[0020] The beneficial effects of the technical solution of the present invention are as follows: by normalizing different types of business scenarios and combining the business scenario feature library and the node feature library, efficient calculation and retrieval are automatically performed through computer programs based on the node characteristics; a scheduling cache area is also designed to dynamically adjust and schedule multi-node data retrieval schemes for various types of business scenarios, and the return results output by each node are sorted and integrated, realizing dynamic adjustment and optimization of the retrieval scheme for the business scenario while automatically analyzing the characteristics of each business scenario and each storage node; and obtaining the retrieval results through the automatic matching retrieval engine and the position tags and deduplication schemes preset in the retrieval results to deeply integrate the retrieval results of each storage node, realizing efficient fusion of the retrieval results of each node, meeting the needs of big data business analysis, solving the long-standing problem of "inaccurate result display and low retrieval efficiency" of multi-node massive data retrieval, and improving the efficiency and accuracy of multi-node data retrieval in complex business scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a flowchart of a multi-node data retrieval method according to an embodiment of the present invention;

[0022] Figure 2 A schematic diagram of a multi-node data retrieval process framework according to an embodiment of the present invention;

[0023] Figure 3 Schematic diagram of a specific implementation framework of multi-node data retrieval according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0025] like Figure 1 As shown, the present invention provides a multi-node data retrieval method, comprising the following steps:

[0026] S1, obtain the retrieval task, normalize the retrieval task, and determine the scenario characteristics of the current business scenario and the node characteristics of the resource nodes associated with the current business scenario based on the normalized retrieval task. The scenario characteristics include: scenario name, retrieval scheme corresponding to the current scenario; node characteristics include resource node ID, data volume, retrieval engine, and scenario retrieval efficiency range; scenario retrieval efficiency range includes the average time for the resource node to return results; in one implementation, exemplarily, the above-mentioned retrieval task is obtained from a search application.

[0027] S2, distribute the first memory space in the preset scheduling buffer area evenly and proportionally to each resource node associated with the current business scenario, and distribute the second memory space in the scheduling buffer area to each associated resource node according to the average time it takes for the associated resource node to return results, wherein the memory space allocated to the resource node with a long average time to return results accounts for a larger proportion than the resource node with a short average time to return results, and the memory space allocated to the resource node is used to store the results returned by the corresponding resource node;

[0028] S3, based on the business scenario feature library pre-stored in the scheduling cache, determines whether the current business scenario is a new business scenario; when the current business scenario is a new business scenario, adjusts the number of returned results of the resource nodes according to the size of the memory space allocated to each resource node and the data volume of each resource node, increases the number of returned results of resource nodes with large data volumes, obtains an adjustment plan for the business scenario, and saves the adjustment plan as a newly created retrieval plan for the current business scenario in the business scenario feature library; wherein the business scenario feature library is pre-established based on the scenario features of historical business scenarios, and the business scenario feature library includes: scenario name, retrieval plan corresponding to the business scenario, and adjustment plan;

[0029] S4: Based on the adjustment plan and the resource node ID, the node feature library pre-stored in the scheduling cache area obtains the search engine corresponding to the resource node associated with the current business scenario, and transmits the adjustment plan to the search engine. The search engine searches each resource node and obtains the search results of the business scenario. The node feature library is established based on the node features of the historical resource nodes and includes: node ID, corresponding search engine, data volume, and scenario search efficiency range;

[0030] S5: De-duplicate and sort the search results, merge the search results of each resource node after de-duplicate and sorting, and obtain the return result set of the current business scenario.

[0031] The method of the embodiment of the present invention improves the efficiency and accuracy of retrieval by dynamically scheduling multi-node resources. In one implementation, a scheduling terminal is added to the user end and the search application end, and the above-mentioned method of the embodiment of the present invention is implemented through the scheduling terminal. The scheduling terminal can face each search application, obtain the retrieval tasks issued by each search application end, normalize the obtained retrieval tasks, obtain the business scenario feature information required for scheduling and the node feature information of the resource nodes storing the resources to be retrieved issued by the search application through the retrieval task, and use a pre-set scheduling buffer area. The scheduling terminal can dynamically schedule and adjust the retrieval plan for each business scenario based on a series of information such as the business scenario to be retrieved, the node information of each resource node associated with the business scenario, i.e., the data volume information, and the size of the memory space in the scheduling buffer area used to store the search return results. In this way, a suitable retrieval plan can be provided based on the different business scenarios and the data volume of the resource nodes. In one case, the retrieval task obtained from the search application is determined by the search application based on the retrieval requirements proposed by the end user.

[0032] Specifically, the embodiment of the present invention normalizes different types of business scenarios and combines the business scenario feature library and the node feature library, and automatically performs efficient calculation and retrieval through a computer program based on the node characteristics; it also designs a scheduling cache area to dynamically adjust and schedule multi-node data retrieval schemes for various types of business scenarios, and sorts and integrates the return results output by each node, thereby realizing dynamic adjustment and optimization of the retrieval scheme for the business scenario while automatically analyzing the characteristics of each business scenario and each storage node; and obtains the retrieval results through the automatic matching retrieval engine and the position tags and deduplication schemes preset in the retrieval results to perform deep integration of the retrieval results of each storage node, thereby realizing efficient integration of the retrieval results of each node, meeting the needs of big data business analysis, solving the long-standing problem of "inaccurate result display and low retrieval efficiency" of multi-node massive data retrieval, and improving the efficiency and accuracy of multi-node data retrieval in complex business scenarios.

[0033] Specifically, in this embodiment, the business scenario is determined by a search task that searches for relevant content in a resource node based on a user's search requirements. By normalizing the business scenarios, search tasks in the same field, with similar search requirements, and with the same or partially identical associated resource nodes, as well as those with a certain degree of relevance, are categorized as similar business scenarios. This reduces the number of times the search engine accesses the resource nodes. Furthermore, through the common characteristics of the search tasks, the reasonable allocation of search plans is rapidly judged and analyzed. The characteristics of different types of business scenarios are automatically analyzed, the search plans are dynamically adjusted, and the corresponding data search engines are scheduled, thereby improving the efficiency and accuracy of multi-node data retrieval in complex business scenarios.

[0034] Specifically in this embodiment, Figure 2 As shown, after obtaining the business scenario, the business scenario is normalized, and the retrieval task is parsed to determine the scenario characteristics of the current business scenario and the node characteristics of the resource nodes associated with the current business scenario. Exemplarily, the above processing can use a pre-set corresponding knowledge base to abstract and parse the retrieval task. According to the scenario characteristics in the historical business scenario and the historical resource node characteristics, a business scenario feature library Yn and a node feature library Zn are respectively constructed. The resource nodes and the retrieval tasks are scheduled based on the scheduling buffer to obtain an adjustment plan for the number of result sets returned by the resource node. The retrieval is performed through the retrieval engine to obtain the return result set of the corresponding resource node of the business scenario. After deduplication and sorting of the result set, the return result sets of each resource node are merged, the return results are pushed and the idle and distorted retrieval resources are released.

[0035] Before obtaining the current business scenario to be retrieved, the scenario feature library Yn and the node feature library Zn are constructed using the scenario features of historical business scenarios and the historical resource node features. Because the pre-set scheduling buffer is used for many tasks, the memory space of each block in the scheduling buffer is relatively limited. To better utilize the buffer, it is necessary to adjust and allocate resources based on the scenario retrieval efficiency of the node features of the resource nodes, allocating more resources to tasks with poor retrieval efficiency. This will maximize business response speed.

[0036] A first memory space preset in a preset scheduling buffer is evenly distributed proportionally to each resource node associated with the current business scenario, and a second memory space preset in the scheduling buffer is allocated to each associated resource node based on the average time it takes for the associated resource node to return results. A larger proportion of memory space is allocated to resource nodes with a longer average time to return results than to resource nodes with a shorter average time to return results. The memory space allocated to the resource node is used to store the results returned by the corresponding resource node. For example, first, a "memory space" NCKZ is added to the business scenario feature library Yn in the scheduling buffer to divide the memory space N according to a certain proportion. The memory space N is used to store the result set returned by the associated resource node. A portion of the memory space, such as N*30%, is evenly distributed as the first memory space to each associated resource node. Then, the remaining memory space, i.e., N*70%, is allocated as the second memory space based on the average time consumption ratio of each resource node; i.e., Yn.NCKZ = N*30% / count(Xn.GLZD) + N*70%*(Zn.ZSSL.average time / sum(Zn.ZSSL.average time)).

[0037] As shown in Table 1, the business characteristics of the retrieval task are composed of multiple scenario features, where the scenario features include the scenario name (CZMC), the scenario ID, the data resource (ZSJY), which refers to the main content of the resource node retrieved from the business scenario, the resource node (GLZD), which refers to the resource node associated with the business scenario, and the node ID of the resource node retrieved from the resource node, the duplicate removal field (PZZD), which refers to the array in the retrieval result used to remove duplicates from the returned result set, the retrieval plan (ZSGD), which represents the plan for searching and retrieving the corresponding resource nodes according to the retrieval conditions of the business scenario, the adjustment plan (TZGD), which is the retrieval plan after scheduling and dynamically adjusting the retrieval task through the scheduling buffer, the retrieval result (ZSZG), the scenario status (ZT), which includes the available status and the unavailable status, to indicate the current executable status of the scenario feature of the business scenario, the last retrieval time (LastTime), which indicates the time when the retrieval task was last executed for the business scenario, and the related remarks information (BZ).

[0038]

[0039] Table 1

[0040] The retrieval scheme includes business conditions, single-page display capacity, current page number, and sorting field. The single-page display capacity indicates the number of search results that can be displayed in the user interface. The adjustment scheme includes business conditions, single-page display capacity, current page number, sorting field, and node ID. The node characteristics in the business scenario include node ID, data volume, retrieval engine, scenario retrieval efficiency range, which is used to represent the average time consumption range of resource nodes of different volumes, and node status as shown in Table 2. The scenario retrieval efficiency range is used to count the impact of the data volume of different resource nodes on the retrieval time through each retrieval, and obtain the response efficiency of different nodes, so that the retrieval scheme can be adjusted more accurately.

[0041]

[0042] Table 2

[0043] Specifically in this embodiment, whether the current business scenario is a new business scenario is confirmed based on the retrieval scheme and the scenario name in the scenario characteristics in Table 1, and task scheduling is performed for different types of business scenarios. The specific process of task allocation for new business scenarios is that when the retrieval scheme and the scenario name in the scenario characteristics do not exist in the business scenario feature library, that is, ZSGD.Current page = 1and! (Yn.CZMC = CZMC and Yn.ZSGD = ZSGD), after confirming that the current business scenario is a new business scenario, a memory space (NCKZ) is obtained according to the division in the scheduling buffer to obtain the business scenario feature library Yn and the node feature library Zn, and an adjustment scheme set Tn is obtained according to Yn and Zn, and a retrieval scheme adapted to the data volume of the resource node is scheduled according to the size of the memory space (NCKZ) in the scheduling buffer. For example, when the business scenario is to retrieve the vehicle passing records of the past week and the business scenario is a new business scenario, as shown in FIG. Figure 3 As shown, a retrieval task in a business scenario is obtained, and a retrieval scheme for the business scenario is obtained by scheduling the cache. The retrieval scheme includes business conditions, single-page display volume, current page, and total data volume. Assuming that there are three associated resource nodes corresponding to the business scenario and their data volumes are Node 1 (data volume of 10 billion), Node 2 (data volume of 1 million), and Node 3 (data volume of 100), respectively, and the size of the memory space (NCKZ) named with the "Scenario ID (CZid)" is set to 1M, then the number of result sets that can be returned by all resource nodes and the size of the storage space occupied are 80% of the memory size of the memory space (NCKZ) of 1M. The number and size of the result sets that can be returned by each node are then allocated based on the data volume of each associated resource node. Specifically, the number of result sets returned by nodes that take a long time to retrieve, that is, nodes with a large data volume, is increased. In other words, more storage space is allocated to resource nodes with long retrieval engine resources and large data volumes, thereby reducing the frequency of subsequent retrieval engines. This utilizes cache space to improve retrieval efficiency and reduce the probability of interface paging errors.

[0044] The traditional retrieval method uses a single rule to retrieve data from all nodes, and does not consider the characteristics of each storage node to optimize the retrieval scheme. The efficiency of a certain node often affects the overall result display and business development, and it will also increase a lot of resource overhead. For example, in the above-mentioned business scenario of retrieving vehicle passing records in the past week, the number of result sets returned by all related resource nodes under the traditional retrieval method is the same, which makes the single-page display volume in its retrieval scheme the same (set to 50). However, due to the different data volumes of different resource nodes and the uneven retrieval resources, the method of synchronous retrieval of each resource node is adopted. For nodes with large data volumes, multiple retrievals are required, which not only has an adverse effect on retrieval efficiency but also causes waste of resources. However, through the asynchronous retrieval method of the data retrieval engine that matches complex business scenarios on demand in this application, memory space is allocated on demand according to the data volume of each node by scheduling the cache area. For example, in the above business scenario of retrieving vehicle passing records for the past week, more resources are allocated to node 1 with a larger data volume, so that its single-page display volume is 1000, and the single-page display volume of node 2 is adjusted to 100, and the single-page display volume of node 3 is adjusted to 50. This not only reduces the number of searches for the node with the highest overhead, but also reduces the number of searches for all nodes during paging operations and avoids resource waste, thereby improving the efficiency and accuracy of multi-node data retrieval in complex business scenarios.

[0045] Specifically, in this embodiment, when the search scheme and the scene name in the scene feature exist in the business scene feature library, the current business scene is a historical business scene; the specific process of task allocation for the historical business scene is as follows: when the search scheme and the scene name in the scene feature exist in the business scene feature library, that is, Yn.CZMC=CZMC and Yn.ZSGD=ZSGD, after confirming that the current business scene is a repeated business scene, there is no need to re-search the associated resource nodes. According to the historical records Rn and the location tags of the sorted return result set, the return result set of the business scene is obtained, thereby reducing the search resource overhead of all nodes and quickly responding to business. When the number of result sets returned by any associated resource node minus the number of returned business end records Rn is less than N times the single-page display volume of the node (through comparison, it is found that the value range of N is 2 to 6), the search scheme of the node will be started in advance, and the relevant data will be pre-searched in advance during the idle phase of the search engine, so that the search resources are further effectively utilized.

[0046] Specifically in this embodiment, after determining whether the current business scenario to be retrieved is a new business scenario, a retrieval task is scheduled for the current business scenario. After allocating corresponding retrieval resources, the business scenario feature library is traversed to obtain a retrieval scheme with an available scenario status, i.e., Yn.Zt=1. The associated resource nodes in Zn are retrieved according to the historical retrieval scheme. When the node ID of the adjustment scheme in the business scenario feature library is the same as the node ID of the current retrieval scheme, the corresponding retrieval engine is paired for the business scenario. According to the adjustment scheme and the resource node ID, the retrieval engine corresponding to the resource node associated with the current business scenario is obtained from the node feature library pre-stored in the scheduling cache area, i.e., the condition Yn.TZGD.Node ID=Zn.ZDID, and its corresponding retrieval engine Zn.ZSYC is obtained. The retrieval engine is a retrieval engine pre-paired according to the node characteristics of the resource node corresponding to the business scenario; the retrieval scheme Yn.TZGD in the current business scenario is passed into the retrieval engine to perform the retrieval, and the return result set D [data set n, node n, Yn] of each resource node is obtained.

[0047] Specifically, in this embodiment, due to the significant differences in the time and quantity of results returned by different storage nodes, if only a simple result return display is performed, problems such as sorting errors and data duplication will often occur when viewing the next page of results, seriously affecting user business development and restricting the utilization of the value of big data. The present invention uses a result fusion algorithm, adopts data location tags and duplicate removal design, and deeply integrates the search results of each node to maximize the accuracy of business applications.

[0048] Compare the newly returned result set with the historical search result set, and perform deduplication on the newly returned result set and the historical search result set according to the deduplication field in the scene feature. Traverse the node to return the result set D [dataset n, node n, Yn], and remove the duplicate records with a later return time from the returned result set D according to the deduplication field Yn.PZZD; sort the returned result set D according to the sort field "Yn.TZGD.sort field", and add a position tag to the corresponding position in the historical record of the returned result set, and then return the newly returned result set N [dataset n, node n, Yn]. The new returned result set [data set n, node n, Yn] is compared with the returned result set D, i.e. the historical returned result set [data set n, node n, Yn], and the duplicate records between the new returned result set and the historical returned result set D are removed according to the duplicate elimination field N.Yn.PZZD; the returned result set N is merged with the historical returned result set D, i.e. D=D+N, and after sorting the returned result records with the push status of "not pushed" according to the sorting field "Yn.TZGD.sorting field", the scene retrieval efficiency range of the node feature is updated; the update algorithm is: Vi-rand*(Zn ZSSL.longest time - Zn.ZSSL.longest short time)+Zn.ZSSL.shortest time, and then the average time consumption of each access time point is obtained by the sine algorithm and stored in Zn.ZSSL.average time.

[0049] According to the single-page display quantity and the current page in the search scheme, select the record set Rn that meets the conditions from D, and return Rn according to the single-page display quantity in the search scheme and the push status of the result set marked with the current page number. The condition is: {(current page number - 1) * single-page display quantity + code 1} < D.position label < {(current page * single-page display quantity}, and set the push status of this part of the records in D to "pushed", that is, D.flag = 1.

[0050] Specifically, in this embodiment, the last retrieval time of the resource node characteristics of the business scenario is regularly checked. If the interval between the last retrieval time of the retrieval scheme and the current system time exceeds the preset second threshold, the node status in the node characteristics is set to an unavailable state. Since a resource node has not been retrieved for too long, the resource node retrieval record will be distorted to a certain extent, which will lead to inaccurate scheduling of the retrieval scheme for the resource node, so it is necessary to delete the node characteristic record whose idle time exceeds the set threshold. If the interval between the last retrieval time of the node feature table and the current system time is regularly checked to see if it exceeds the set second threshold (60 days), the node status of the associated resource node is automatically set to an unavailable state, that is, Zn.Zt=0.

[0051] At the same time, the last retrieval time of the retrieval plan in the business scenario is regularly checked. If the interval between the last retrieval time of the retrieval plan and the current system time exceeds the preset time threshold, the corresponding cache area is cleared according to the scenario name in the business scenario feature library, that is, the cache area named Yn.CZId is automatically cleared to release idle resources.

[0052] The present invention also provides a multi-node data retrieval device, including a memory and a processor, wherein the memory stores at least one program, and the at least one program is executed by the processor to implement the above-mentioned multi-node data retrieval method. As an executable solution, the multi-node data retrieval device can be a computing device such as a desktop computer, a notebook, a handheld computer, and a cloud server. The system / electronic device may include, but is not limited to, a processor and a memory. Those skilled in the art will understand that the composition structure of the above-mentioned system / electronic device is only an example of the system / electronic device and does not constitute a limitation on the system / electronic device. It may include more or fewer components than the above-mentioned components, or a combination of certain components, or different components. For example, the system / electronic device may also include input and output devices, network access devices, buses, etc., which are not limited in the embodiments of the present invention.

[0053] Furthermore, as an executable solution, the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the system / electronic device, connecting various parts of the entire system / electronic device using various interfaces and lines.

[0054] The memory can be used to store computer programs and / or modules. The processor implements various functions of the system / electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store an operating system and at least one application required for a function; the data storage area can store data created based on the use of the mobile phone, etc. In addition, the memory can include high-speed random access memory and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0055] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method embodiments of the present invention. If a module or unit integrated into a system or electronic device is implemented as a software functional unit and sold or used as a standalone product, it may be stored in a computer-readable storage medium. Based on this understanding, the present invention can implement all or part of the process steps of the above-described method embodiments by instructing the relevant hardware through a computer program. The computer program may be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of each of the above-described method embodiments. The computer program includes computer program code, which may be in source code form, object code form, executable file, or some intermediate form. Computer-readable media may include any entity or device capable of carrying computer program code, recording media, USB flash drives, removable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), and software distribution media. It should be noted that the content contained in the computer-readable medium may be appropriately expanded or reduced based on the requirements of legislation and patent practice within a jurisdiction.

[0056] The reader should understand that in the description of this specification, reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0057] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method can be implemented in other ways. For example, the method embodiments described above are merely illustrative. For example, the division of steps is merely a logical function division. In actual implementation, other division methods may be used. For example, multiple steps may be combined or integrated into another step, or some features may be ignored or not performed.

[0058] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A multi-node data retrieval method, characterized by: The following steps are included: S1, obtain a retrieval task, normalize the retrieval task, and determine the scenario characteristics of the current business scenario and the node characteristics of the resource nodes associated with the current business scenario based on the normalized retrieval task, wherein the scenario characteristics include: scenario name, retrieval scheme corresponding to the current scenario; the node characteristics include resource node ID, data volume, retrieval engine, and scenario retrieval efficiency range; the scenario retrieval efficiency range includes the average time it takes for the resource node to return results; S2, allocating the first memory space in the preset scheduling buffer area to each resource node associated with the current business scenario in proportion and evenly, and allocating the second memory space in the scheduling buffer area to each associated resource node according to the average time for the associated resource node to return the result, wherein the memory space allocated to the resource node with a long average return result time accounts for a larger proportion than the resource node with a short average return result time, and the memory space allocated to the resource node is used to store the result returned by the corresponding resource node; S3, based on the business scenario feature library pre-stored in the scheduling buffer area, determining whether the current business scenario is a new business scenario; when the current business scenario is a new business scenario, adjusting the number of returned results of the resource nodes according to the size of the memory space allocated to each resource node and the data volume of each resource node, increasing the number of returned results of resource nodes with large data volumes, obtaining an adjustment plan for the business scenario, and saving the adjustment plan as a newly created retrieval plan for the current business scenario in the business scenario feature library; wherein the business scenario feature library is pre-established based on the scenario features of historical business scenarios, and the business scenario feature library includes: a scenario name, a retrieval plan corresponding to the business scenario, and an adjustment plan; S4, according to the adjustment plan and the resource node ID, obtaining a search engine corresponding to the resource node associated with the current business scenario from the node feature library pre-stored in the scheduling cache area, transferring the adjustment plan to the search engine, and having the search engine search each of the resource nodes, and obtaining a search result for each of the resource nodes for the current business scenario, wherein the node feature library is established based on the node features of historical resource nodes, and the node feature library includes: node ID, corresponding search engine, data volume, and scenario retrieval efficiency range; S5, deduplicating and sorting the search results of each resource node, and fusing the deduplicated and sorted search results of each resource node to obtain a return result set for the current business scenario; The scene feature further includes a duplicate elimination field, and S5 includes: Traverse the historical return result set, and remove duplicate records with later return times from the historical return result set according to the deduplication field; sort the historical return result set after deduplication, and add position tags to the corresponding positions of the historical records of the historical return result set after deduplication, then compare the new return result set with the historical return result set after deduplication, remove the duplicate records in the new return result set and the historical return result set after deduplication according to the deduplication field, and then merge the new return result set after deduplication with the historical return result set after deduplication.

2. The multi-node data retrieval method according to claim 1, characterized in that: The S3 also includes: comparing the scene name of the current business scenario with the scene name in the business scenario feature library to confirm whether the current business scenario is a new business scenario; when the scene name of the current business scenario does not exist in the business scenario feature library, confirming that the current business scenario is a new business scenario; otherwise, it is a repeated business scenario.

3. The multi-node data retrieval method according to claim 1, characterized in that: The adjustment plan includes a retrieval node ID, and S4 allocates a corresponding retrieval engine to the current business scenario according to the adjustment plan. In response to the resource node ID of the resource node associated with the current business scenario being the same as the retrieval node ID of the adjustment plan, the resource node is searched through the retrieval engine to obtain a retrieval result of the resource node.

4. The multi-node data retrieval method according to any one of claims 1 to 3, characterized in that: The retrieval scheme also includes a single-page display quantity, a current page number, and a sorting field. The push status of the returned result set is marked according to the single-page display quantity and the current page number, wherein the push status includes pushed and not pushed; records with a push status of "not pushed" are selected from the returned result set for pushing and their push status is set to "pushed", and the returned result set records with a push status of "not pushed" are sorted according to the sorting field and marked with position tags.

5. The multi-node data retrieval method according to claim 4, characterized in that: When the number of the returned result sets with the push status of "not pushed" is at least twice less than the single-page display volume, the resource nodes associated with the business scenario are pre-searched through the retrieval scheme to retrieve the data resources of the business scenario in advance.

6. The multi-node data retrieval method according to claim 1, characterized in that: The business scenario feature library also includes the last retrieval time, and the method also includes: regularly detecting the last retrieval time. If the interval between the last retrieval time of the business scenario and the current system time exceeds a preset first threshold, the scheduling cache area clears the corresponding cache area according to the scene name in the business scenario feature library.

7. The multi-node data retrieval method according to claim 6, characterized in that: The node feature also includes a node status, which includes an available status and an unavailable status. The last retrieval time is regularly checked. If the interval between the last retrieval time of the retrieval scheme and the current system time exceeds a preset second threshold, the node status in the node feature is set to an unavailable status.

8. A multi-node data retrieval device, characterized in that: The method comprises a memory and a processor, wherein the memory stores at least one program, and the at least one program is executed by the processor to implement the multi-node data retrieval method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that The storage medium stores at least one program, and the at least one program is executed by a processor to implement the multi-node data retrieval method according to any one of claims 1 to 7.

Images