A method for solving cache failure of distributed index clusters for massive data

Abstract

The present invention is applicable to the technical field of distributed index clusters, and provides a method for solving the cache failure of distributed index clusters of massive data, comprising the following steps: Step 1: screening the search manager to filter out hot data; Step 2: performing correlation analysis on the filtered hot data, and comprehensively sorting them according to the correlation and the number of searches; Step 3: analyzing the performance of the searcher, and calculating the difference between the preheating time spent on the preheating query and the time gain effect brought by the preheating query; Step 4: setting the preheating query statement, and finding the optimal number of the best preheating query through experiments; Step 5: testing the time gain brought by setting the optimal number of preheating queries under different conditions. The method solves the problem of system cache failure caused by the submission operation, and realizes the hot start of the system at the same time, greatly saves cluster resources, and improves the cluster query speed.

Description

Technical Field

[0001] The present invention belongs to the technical field of distributed index clusters, and in particular, relates to a method for resolving cache failure of a distributed index cluster for massive data. Background Art

[0002] With the continuous increase in the scale of my country's high-speed railways, the use of four electrical equipment such as traction power supply system, railway power system, railway communication signal power supply and contact network disconnector, as well as the use of 6C system, has led to a sharp increase in the data collected by the inspection and monitoring system. The volume of data has exceeded TB level, which makes the effective storage of valuable data a new problem. The traditional TB-level database is difficult to meet the storage requirements, and the speed of data increase has far exceeded the growth of computer performance. How to quickly process the massive data of the distribution network and propose a rapid response technology for the dispatching system has become an urgent problem to be solved.

[0003] The existing methods for processing massive data in distribution networks are mainly divided into two categories. The first category is to use traditional relational data to process massive data. The first method is memory database technology. In-memory computing makes full use of large-capacity memory for data processing, reducing or even avoiding I / O operations. This method has problems such as limited memory capacity, high input and output (I / O) pressure, low cache hit rate, and low overall data processing. The second method is the extended relational database method. Although this method can store a large amount of data, it requires increasing the number of hosts to increase the database capacity, requires expensive computers, and has a high maintenance cost. The second category is NoSQL database cluster technology, including Redis, Couchbase, HBase, etc. Redis is also a memory database with poor persistence function, high overall maintenance cost, and small amount of stored data. Couchbase and HBase both store data through key / value, but Couchbase has a single value type and cannot automatically create a doc id. It is necessary to specify a Document Indentifer for storage for each document, and a large amount of memory is required for key caching. The HBase database is very fast for primary key queries, but the index of non-primary key requires a full table scan, which consumes too much system resources. For this purpose, an inverted index cluster is specially built. The inverted index can quickly index non-primary keys, which solves the limitation of not being able to index non-primary keys. During the data storage process, various sensors will continuously obtain data from the distribution network and transmit it to the data analyzer through the communication network. The data analyzer stores the data in a distributed cluster. The cluster stores the data persistently on the disk, and a commit operation is required. Since each commit operation will open a new searcher, causing the previous cache object to become invalid, this will greatly reduce the cache hit rate of the new searcher, resulting in reduced efficiency in hot data indexing and a waste of cluster resources. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a method for solving the problem of cache failure of a distributed index cluster of massive data, aiming to solve the problem of how to quickly process massive data in a distribution network and propose a rapid response technology for a dispatching system.

[0005] The embodiment of the present invention is implemented as follows: a method for solving cache failure of a distributed index cluster of massive data, comprising the following steps:

[0006] Step 1: Screen the search manager to select hot data;

[0007] Step 2: Analyze the relevance of the selected hotspot data, and sort them comprehensively according to the relevance and search times;

[0008] Step 3: Analyze the performance of the search engine, calculate the difference between the warm-up time spent on the warm-up query and the time gain effect brought by the warm-up query, and analyze the relationship between the number of warm-up query statements set and the reduction of search time;

[0009] Step 4: Set the warm-up query statements and find the optimal number of warm-up queries through experiments;

[0010] Step 5: Test the time gain of setting the optimal number of warm-up queries under different conditions.

[0011] Preferably, step 1 includes the following specific steps: using the search manager to count the data queried by the searcher of the distributed index cluster during the working process, obtaining query words with high query frequency ranking, and preparing data for subsequent cache preheating by defining the query words with high ranking as hot data.

[0012] Preferably, step 2 comprises the following specific steps:

[0013] Step 2.1: Count the number of times the hotspot data appears in the matching documents and calculate the hotspot frequency ( ), calculate the hotspot frequency ( ) is:

[0014]

[0015] In the formula is the number of times the hotspot data appears in the search document. The hotspot frequency is calculated by taking the square root of the number of times the hotspot data appears in the document. );

[0016] Step 2.2: Count the number of times the reverse document of the hot data appears, calculate its inverse based on the document frequency, and thus calculate the frequency of the reverse document ( ), the reverse document frequency is a measure of the rarity of the query term;

[0017] Step 2.3: Calculate the field specification by using the weight of the matching document, the weight of the matching field, and the length normalization factor of the long document penalty. Specifically, the formula for the field specification is calculated by multiplying the weight of the matching document, the weight of the matching field, and the length of the long document penalty ( );

[0018] Step 2.4: Calculate the weighted square sum of the query data and multiply it with the rest of the relevance score to achieve query normalization. Specifically, take the square root of the product and take the inverse to get the query normalization ( );

[0019] Step 2.5: Calculate the proportion of the total matching documents in the hot data query, define this proportion as the coordination factor, and calculate the coordination factor ( );

[0020] Step 2.6: Use the hotspot frequency, reverse document frequency, field specification, query normalization, and coordination factor calculated in steps 2.1 to 2.5 to calculate the relevance score of the document when querying hotspot data. The calculation formula for the relevance score of the document when querying hotspot data is:

[0021] ;

[0022] Step 2.7: Perform comprehensive ranking based on the relevance score and search times of the hot data. Specifically, the method for calculating the comprehensive ranking is: .

[0023] Preferably, step 3 comprises the following specific steps:

[0024] Step 3.1: Calculate the hot data search time when preheat query is set;

[0025] Step 3.2: Calculate the search time when no warm-up query is set;

[0026] Step 3.3: Analyze the relationship between the maximum time benefit generated by the warm-up query and the number of warm-up queries.

[0027] Preferably, the step 3.1 includes the following specific steps:

[0028] Step 3.1.1: Set the number of cache warm-up queries to , when performing a commit operation, the opening and closing times of the new indexer are and , the query time of the system cache miss is , the query time when a cache warm-up query hits is , the probability of hitting the cache warm-up query is ;

[0029] Step 3.1.2: Set the hot data search time for warm-up query ,

[0030] .

[0031] Preferably, the step 3.2 includes the following specific steps:

[0032] Step 3.2.1: When a cache query is not set and a commit operation is performed, the opening and closing times of the new indexer are and , when querying, the system query time is ;

[0033] Step 3.2.2: Search time when warmup query is not set

[0034] .

[0035] Preferably, the step 3.3 includes the following specific steps:

[0036] Step 3.3.1: Calculate the time benefit of setting up warm-up queries , , to generate time benefits , that is, requiring ;

[0037] Step 3.3.2: Regardless of whether or not cache warm-up query design is performed, the opening and closing times of the new cache are the same. , , and in the case of designing cache preheating query, the query time when the cache is not hit is equal to the query time when the cache is not set to preheat the query, that is, ;

[0038] Step 3.3.3: Derivation of the time benefit from the above conditions The range is:

[0039]

[0040] In the inequality, , , as well as All are fixed values, and the query time that does not hit the cache , that is, the right side of the inequality is greater than 0, that is There is an extreme value when When it continues to increase and reaches the extreme point, the time benefit will also reach the extreme point. As you continue to increase it, the time benefits will start to diminish.

[0041] Preferably, step 4 includes the following specific steps:

[0042] Step 4.1: Keep increasing the number of warm-up queries. When the test system is continuously submitting operations, set the search time for hot data of warm-up queries.

[0043] Step 4.2: Find the value of n where query time benefits start to decrease;

[0044] Step 4.3: Define n at this point as the optimal number of warm-up queries.

[0045] Preferably, step 5 includes the following specific steps:

[0046] The query time of hot data for single-condition query and multi-condition query was tested respectively when the optimal number of preheating queries n was set and when no preheating query was set and the query was submitted once. The query time of hot data for single-condition query and multi-condition query was tested respectively when the optimal number of preheating queries n was set and when no preheating query was set and the query was submitted continuously.

[0047] A method for resolving cache failure of a distributed index cluster of massive data provided by an embodiment of the present invention solves the problem of system cache failure caused by a submission operation by analyzing the relationship between the number of preheating queries for hot data and query performance, and by designing an appropriate number of preheating queries for hot data. At the same time, a hot start of the system is achieved, which greatly saves cluster resources and improves cluster query speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 An overall flow chart of the resource optimization method provided by an embodiment of the present invention;

[0049] Figure 2 Flowchart for scoring relevance. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0051] The specific implementation of the present invention is described in detail below in conjunction with specific embodiments.

[0052] like Figure 1-2 As shown, a method for solving cache failure of a massive data distributed index cluster provided by an embodiment of the present invention includes the following steps:

[0053] Step 1: Screen the search manager to select hot data;

[0054] Step 2: Analyze the relevance of the selected hotspot data, and sort them comprehensively according to the relevance and search times;

[0055] Step 3: Analyze the performance of the search engine, calculate the difference between the warm-up time spent on the warm-up query and the time gain effect brought by the warm-up query, and analyze the relationship between the number of warm-up query statements set and the reduction of search time;

[0056] Step 4: Set the warm-up query statements and find the optimal number of warm-up queries through experiments;

[0057] Step 5: Test the time gain of setting the optimal number of warm-up queries under different conditions.

[0058] A further technical solution, step 1 includes the following specific steps: using the search manager to count the data queried by the searcher of the distributed index cluster during operation, obtaining query terms with high query frequency ranking, and preparing data for subsequent cache preheating by defining the query terms with high ranking as hot data.

[0059] According to a further technical solution, step 2 comprises the following specific steps:

[0060] Step 2.1: Count the number of times the hotspot data appears in the matching documents and calculate the hotspot frequency ( ), calculate the hotspot frequency ( ) is:

[0061]

[0062] In the formula is the number of times the hotspot data appears in the search document. The hotspot frequency is calculated by taking the square root of the number of times the hotspot data appears in the document. );

[0063] Step 2.2: Count the number of times the reverse document of the hot data appears, calculate its inverse based on the document frequency, and thus calculate the frequency of the reverse document ( ), the reverse document frequency is a measure of the rarity of the query term;

[0064] Step 2.3: Calculate the field specification by using the weight of the matching document, the weight of the matching field, and the length normalization factor of the long document penalty. Specifically, the formula for the field specification is calculated by multiplying the weight of the matching document, the weight of the matching field, and the length of the long document penalty ( );

[0065] Step 2.4: Calculate the weighted square sum of the query data and multiply it with the rest of the relevance score to achieve query normalization. Specifically, take the square root of the product and take the inverse to get the query normalization ( );

[0066] Step 2.5: Calculate the proportion of the total matching documents in the hot data query, define this proportion as the coordination factor, and calculate the coordination factor ( );

[0067] Step 2.6: Use the hotspot frequency, reverse document frequency, field specification, query normalization, and coordination factor calculated in steps 2.1 to 2.5 to calculate the relevance score of the document when querying hotspot data. The calculation formula for the relevance score of the document when querying hotspot data is:

[0068] ;

[0069] Step 2.7: Perform comprehensive ranking based on the relevance score and search times of the hot data. Specifically, the method for calculating the comprehensive ranking is: .

[0070] Further technical solution, the step 3 comprises the following specific steps:

[0071] Step 3.1: Calculate the hot data search time when preheat query is set;

[0072] Step 3.2: Calculate the search time when no warm-up query is set;

[0073] Step 3.3: Analyze the relationship between the maximum time benefit generated by the warm-up query and the number of warm-up queries.

[0074] Further technical solution, the step 3.1 comprises the following specific steps:

[0075] Step 3.1.1: Set the number of cache warm-up queries to , when performing a commit operation, the opening and closing times of the new indexer are and , the query time of the system cache miss is , the query time when a cache warm-up query hits is , the probability of hitting the cache warm-up query is ;

[0076] Step 3.1.2: Set the hot data search time for warm-up query ,

[0077] .

[0078] Further technical solution, the step 3.2 includes the following specific steps:

[0079] Step 3.2.1: When a cache query is not set and a commit operation is performed, the opening and closing times of the new indexer are and , when querying, the system query time is ;

[0080] Step 3.2.2: Search time when warmup query is not set

[0081] .

[0082] Further technical solution, the step 3.3 includes the following specific steps:

[0083] Step 3.3.1: Calculate the time benefit of setting up warm-up queries , , to generate time benefits , that is, requiring ;

[0084] Step 3.3.2: Regardless of whether or not cache warm-up query design is performed, the opening and closing times of the new cache are the same. , , and in the case of designing cache preheating query, the query time when the cache is not hit is equal to the query time when the cache is not set to preheat the query, that is, ;

[0085] Step 3.3.3: Derivation of the time benefit from the above conditions The range is:

[0086]

[0087] In the inequality, , , as well as All are fixed values, and the query time that does not hit the cache , that is, the right side of the inequality is greater than 0, that is There is an extreme value when When it continues to increase and reaches the extreme point, the time benefit will also reach the extreme point. As you continue to increase it, the time benefits will start to diminish.

[0088] According to a further technical solution, step 4 comprises the following specific steps:

[0089] Step 4.1: Keep increasing the number of warm-up queries. When the test system is continuously submitting operations, set the search time for hot data of warm-up queries.

[0090] Step 4.2: Find the value of n where query time benefits start to decrease;

[0091] Step 4.3: Define n at this point as the optimal number of warm-up queries.

[0092] According to a further technical solution, step 5 comprises the following specific steps:

[0093] The query time of hot data for single-condition query and multi-condition query was tested respectively when the optimal number of preheating queries n was set and when no preheating query was set and the query was submitted once. The query time of hot data for single-condition query and multi-condition query was tested respectively when the optimal number of preheating queries n was set and when no preheating query was set and the query was submitted continuously.

[0094] The present invention analyzes the relationship between the number of preheating queries for hot data and the performance of the query server, and designs an appropriate number of preheating queries for hot data, thereby solving the problem of system cache failure caused by submission operations and realizing hot start of the system, which greatly saves cluster resources and improves cluster query speed.

[0095] The distributed index cluster with preheat query has greatly improved the query speed of hot data. After importing 1.6×108 data, the index cluster was tested under single-thread (query operation only) and multi-thread (data submission and data query at the same time) conditions. The query time of the distributed indexer for single-condition query, double-condition query, and three-condition query was tested under the optimal preheat number n and without preheat query. The experimental results are compared in the following table:

[0096]

[0097] In the table, 1 means that cache preheating query is set, 2 means that cache preheating query is not set, 1_station: station 443 means that when cache preheating query is set, the query condition is to query the data named station 443 in the station attribute. The same applies to other query conditions in the table.

[0098] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for solving cache failure of a massive data distributed index cluster, characterized in that: The following steps are involved: Step 1: Screen the search manager to select hot data; Step 2: Analyze the relevance of the selected hotspot data, and sort them comprehensively according to the relevance and search times; Step 3: Analyze the performance of the search engine, calculate the difference between the warm-up time spent on the warm-up query and the time gain effect brought by the warm-up query, and analyze the relationship between the number of warm-up query statements set and the reduction of search time; Step 4: Set the warm-up query statements and find the optimal number of warm-up queries through experiments; Step 5: Test the time gain of setting the optimal number of warm-up queries under different conditions; The step 3 comprises the following specific steps: Step 3.1: Calculate the search time for hot data when preheating query is set; Step 3.2: Calculate the search time when no warm-up query is set; Step 3.3: Analyze the relationship between the maximum time benefit generated by the warm-up query and the number of warm-up queries; The step 3.1 includes the following specific steps: Step 3.1.1: Set the number of cache warm-up queries to , when performing a commit operation, the opening and closing times of the new indexer are and , the query time of the system cache miss is , the query time when a cache warm-up query hits is , the probability of hitting the cache warm-up query is ; Step 3.1.2: Set the hot data search time for warm-up query , ; The step 3.2 includes the following specific steps: Step 3.2.1: When a cache query is not set and a commit operation is performed, the opening and closing times of the new indexer are and , when querying, the system query time is ; Step 3.2.2: Search time when warmup query is not set , ; The step 3.3 includes the following specific steps: Step 3.3.1: Calculate the time benefit of setting up warm-up queries , , to generate time benefits , that is, requiring ; Step 3.3.2: Regardless of whether or not cache warm-up query design is performed, the opening and closing times of the new cache are the same. , , and in the case of designing cache preheating query, the query time when the cache is not hit is equal to the query time when the cache is not set to preheat the query, that is, ; Step 3.3.3: Derivation of the time benefit from the above conditions The range is: ; In the inequality, , , as well as All are fixed values, and the query time that does not hit the cache , that is, the right side of the inequality is greater than 0, that is There is an extreme value when When it continues to increase and reaches the extreme point, the time benefit will also reach the extreme point. As you continue to increase it, the time benefits will start to diminish.

2. The method for solving cache failure of a massive data distributed index cluster according to claim 1, characterized in that: The step 1 includes the following specific steps: using the search manager to count the data queried by the searcher of the distributed index cluster during the working process, obtaining query words with high query frequency ranking, and preparing data for subsequent cache preheating by defining the query words with high ranking as hot data.

3. The method for solving cache failure of a massive data distributed index cluster according to claim 1, characterized in that: The step 2 comprises the following specific steps: Step 2.1: Count the number of times the hotspot data appears in the matching documents and calculate the hotspot frequency; Step 2.2: Count the number of times the reverse document of the hot data appears, calculate its inverse according to the document frequency, and thus calculate the frequency of the reverse document; Step 2.3: Calculate the field specification by the weight of the matching document, the weight of the matching field, and the length normalization factor that penalizes long documents; Step 2.4: Calculate the sum of squared weights of the query data and multiply it by the rest of the relevance score to normalize the query; Step 2.5: Calculate the proportion of the total matching documents when searching for hot data, define the proportion as the coordination factor, and calculate the coordination factor; Step 2.6: Calculate the relevance score of the document when querying the hot data using the hotspot frequency, reverse document frequency, field specification, query normalization, and coordination factor calculated from steps 2.1 to 2.5; Step 2.7: Perform comprehensive sorting based on the relevance score and search times of the hot data.

4. The method for solving cache failure of a massive data distributed index cluster according to claim 1, characterized in that: The step 4 comprises the following specific steps: Step 4.1: Keep increasing the number of warm-up queries. When the test system is continuously submitting operations, set the search time for hot data of warm-up queries. Step 4.2: Find the value of n where query time benefits start to decrease; Step 4.3: Define n at this point as the optimal number of warm-up queries.

5. The method for solving cache failure of a distributed index cluster of massive data according to claim 1, characterized in that: The step 5 comprises the following specific steps: The query time of hot data for single-condition query and multi-condition query was tested respectively when the optimal number of preheating queries n was set and when no preheating query was set and the query was submitted once. The query time of hot data for single-condition query and multi-condition query was tested respectively when the optimal number of preheating queries n was set and when no preheating query was set and the query was submitted continuously.

Images