Enterprise internal data access retrieval method

Abstract

The invention relates to the technical field of data retrieval, in particular to an enterprise internal data access retrieval method which comprises the following steps: acquiring all data points and all historical retrieval records of each data unit and performance data of various performance indexes of each node in a distributed storage system; calculating the storage load rate of each data unit; calculating a first correlation degree and a second correlation degree of any two data units; determining a retrieval association degree of any two data units; acquiring a node performance sequence and a node load sequence; obtaining each matched node and each unmatched node, calculating the schedulability of any data unit, evaluating an allocation scheme, carrying out iterative adjustment on the allocation scheme, and carrying out node allocation on the data units. According to the method and the device, the data retrieval efficiency is improved while the loads of different nodes are balanced.

Description

Technical Field

[0001] The present application relates to the technical field of data retrieval, and in particular to a method for accessing and retrieving data within an enterprise. Background Art

[0002] With the rapid advancement of information technology, massive amounts of data are constantly being generated within enterprises. How to quickly retrieve from such massive amounts of data is a problem that needs to be solved. By using a distributed hash algorithm to store the data generated within the enterprise and using hash indexing to quickly retrieve the data, the enterprise data can be protected while improving data retrieval efficiency.

[0003] However, when using the distributed hash algorithm to store data generated within the enterprise, the amount of data stored in different storage nodes varies, resulting in different loads on the storage nodes. Some storage nodes with poor performance may be loaded with too much data, which reduces the response time of the nodes during data retrieval, leading to delays and reduced throughput in the distributed system. At the same time, storage nodes with excessive loads may also crash or fail, increasing the risk of data loss and reducing the efficiency of data retrieval. Summary of the Invention

[0004] In order to solve the above technical problems, a method for accessing and retrieving internal enterprise data is provided to solve the existing problems.

[0005] The solution to the technical problem of this application is to provide a method for accessing and retrieving internal enterprise data, including the following steps:

[0006] Divide various types of data within the enterprise into data units, obtain all data points of each data unit, all historical search records, and performance data of various performance indicators of each node in the distributed storage system;

[0007] Calculate the storage load rate of each data unit based on the sampling frequency of the data points in each data unit and the number of all data points;

[0008] Analyze the situation in which any two data units are searched at the same time in the historical search records, and calculate the first correlation degree of the any two data units;

[0009] Determining a second correlation degree between the two data units based on correlation between changes in adjacent data points in the same sampling period in the two data units and differences in changes in data points in the two data units; and determining a retrieval correlation degree between the two data units by fusing the first correlation degree and the second correlation degree.

[0010] The nodes are sorted according to the performance data of various performance indicators of each node to obtain a node performance sequence; the data units are initially allocated based on the distance relationship between the collection location of each data unit and the node, and the nodes are sorted according to the storage load rate of all data units allocated to each node to obtain a node load sequence;

[0011] Analyze the matching status of the elements in the node performance sequence and the node load sequence, obtain each matching node and each unmatched node, and calculate the schedulability of any data unit assigned to each unmatched node based on the retrieval correlation between the data unit and the remaining data units, combined with the storage load rate; evaluate the initial allocation plan based on the number of matching nodes, iteratively adjust the allocation plan in combination with the schedulability, and perform node allocation on the data units.

[0012] Preferably, the calculating the storage load rate of each data unit includes:

[0013] Calculate the average time interval between the corresponding sampling times of any two adjacent data points in each data unit, and record it as the average sampling interval;

[0014] The storage load ratio is the ratio between the number of all data points in each data unit and the average sampling interval.

[0015] Preferably, the first correlation degree R1 between the rth data unit and the hth data unit r,h The calculation formula is: Among them, U r,h is the number of all historical retrieval records for simultaneously retrieving the rth data unit and the hth data unit, U r is the number of all historical retrieval records for retrieving the rth data unit, U h is the number of all historical search records for retrieving the h-th data unit, and ε is a preset value greater than 0.

[0016] Preferably, determining the second degree of association of the arbitrary two data units includes:

[0017] The data unit with the largest average sampling interval among the two data units is selected as the target unit, and the other data unit is recorded as the reference unit;

[0018] Record two adjacent data points as a pair of adjacent data points, and calculate the relative change of each pair of adjacent data points in the target unit;

[0019] Based on a sampling period between each pair of adjacent data points in the target unit, all data points within the sampling period are selected from the reference unit to form a reference data sequence;

[0020] Calculate the cumulative sum of relative changes between all adjacent data points in the reference data sequence, and record it as the reference change;

[0021] The relative changes of all adjacent data points in the target unit are combined into a first change sequence; the reference changes of all reference data sequences in the reference unit are combined into a second change sequence;

[0022] calculating a correlation between the first change sequence and the second change sequence;

[0023] Calculating the difference between each element in the first change sequence and the element at the same position in the second change sequence, and calculating the cumulative sum of the differences of all elements in the first change sequence;

[0024] The second correlation degree is a ratio of an absolute value of the correlation degree to the cumulative sum.

[0025] Preferably, the further acquisition process of the node performance sequence is:

[0026] The performance data of all nodes under each performance indicator are forward-processed, and all nodes are arranged in descending order of performance data. The sum of the rankings of each node in all performance indicators is calculated, and all nodes are arranged in ascending order according to the sum to form a node performance sequence.

[0027] Preferably, the further acquisition process of the node load sequence is:

[0028] Calculate the distance between the collection location of each data unit and the location of each node;

[0029] All data units are pre-allocated to the nearest node as the initial allocation scheme;

[0030] The sum of the storage load rates of all data units allocated to each node in the initial allocation scheme is used as the total load rate of each node in the initial allocation scheme;

[0031] All nodes are arranged in descending order according to the total load rate to form a node load sequence.

[0032] Preferably, obtaining each matching node and each unmatched node includes:

[0033] If the node load sequence is the same as the node at the same position in the node performance sequence, the corresponding node is a matching node; otherwise, it is an unmatched node.

[0034] Preferably, the calculation formula for the schedulability of any data unit is: Among them, T n,ris the schedulable number of the rth data unit assigned to the nth mismatched node, L n,r RJ is the storage load ratio of the rth data unit assigned to the nth mismatched node, r,h H is the retrieval correlation between the rth data unit and the hth data unit assigned to the nth unmatched node, n is the number of all data units allocated to the nth unmatched node, and ∈ is a preset value greater than 0.

[0035] Preferably, the evaluation of the initial allocation plan includes:

[0036] Count the number of all matching nodes as the node matching degree of the initial allocation plan;

[0037] If the node matching degree is less than a preset threshold, the allocation scheme is iteratively adjusted; otherwise, the data unit is stored according to the current allocation scheme.

[0038] Preferably, the iterative adjustment of the allocation scheme includes:

[0039] Allocating the largest schedulable data unit under all unmatched nodes to the second-closest node as the next allocation plan;

[0040] By continuously iteratively adjusting the allocation plan, the optimal allocation plan is obtained, and the data units are allocated to nodes according to the optimal allocation plan.

[0041] This application has at least the following beneficial effects:

[0042] The present application calculates the storage load rate of each data unit by the sampling frequency of the data in each data unit, and its beneficial effect is that it takes into account the speed of the growth of the data volume of the corresponding data unit to reflect the load degree of the data volume of the corresponding data unit; secondly, through the situation in which any two data units are retrieved simultaneously in the historical retrieval record, the first correlation degree of the arbitrary two data units is calculated to reflect the correlation between the corresponding two data units, so as to illustrate the extent to which the corresponding two data units should be allocated to the same node; the change of adjacent data points in the arbitrary two data units is analyzed to calculate the second correlation degree of the arbitrary two data units, and its beneficial effect is that it takes into account the correlation of the change amount of the data points in the two data units to further evaluate the correlation between the two data units; the fusion of the first correlation degree and the second correlation degree to determine the retrieval correlation degree of the arbitrary two data units, and its beneficial effect is that it comprehensively evaluates the correlation between the two data units by comprehensively considering the correlation of the historical retrieval records and the change amount of the data points to reflect the extent to which the corresponding two data units should be allocated to the same node. The method can improve the possibility of accessing multiple nodes during data retrieval, thereby reducing the possibility of accessing multiple nodes during data retrieval and improving data retrieval efficiency; sort the nodes according to the performance data of the nodes, obtain the node performance sequence, pre-allocate the data units according to the distance between the collection position of each data unit and the node, sort the nodes according to the storage load rate of the data units pre-allocated to each node, obtain the node load sequence, and obtain each matching node and each unmatched node by matching the node performance sequence with the nodes at the same position in the node load sequence. The beneficial effect is that it takes into account the performance of the nodes and the load of each node after pre-allocation, and reflects the degree to which data units with higher loads are allocated to nodes with better performance; calculate the schedulability of the data units allocated to each unmatched node, iterate and adjust the allocation scheme continuously through schedulability, obtain the optimal allocation scheme, and allocate the data units to nodes according to the optimal allocation scheme. The beneficial effect is that it balances the loads of different nodes, reduces the risk of data loss due to node crash, reduces the response time of the node, and improves the efficiency of data retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The following is a detailed description of an internal enterprise data access and retrieval method of the present application in conjunction with the accompanying drawings.

[0044] Figure 1 A flowchart of a method for accessing and retrieving internal enterprise data provided by an embodiment of the present application;

[0045] Figure 2 A flowchart of the steps of the method for obtaining the optimal allocation solution provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the following further describes in detail a method for accessing and retrieving enterprise internal data proposed in this application in conjunction with the accompanying drawings and implementation examples. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0047] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0048] See also Figure 1 , which shows a flowchart of a method for accessing and retrieving internal enterprise data provided by an embodiment of the present application, the method comprising the following steps:

[0049] Step 1: Divide various types of data within the enterprise into data units, obtain all data points of each data unit, all historical search records, and performance data of various performance indicators of each node in the distributed storage system.

[0050] In a distributed storage system, there are multiple nodes that work together to store and manage data to achieve high availability, high scalability and load balancing. By distributing data to multiple nodes for storage, overload of a single node is avoided and the overall performance of the distributed storage system is improved. In order to improve the retrieval efficiency of data, when distributed hashing data is stored, in order to solve the problem of excessive load on nodes with poor performance, which causes nodes to crash or fail, the data distribution and storage scheme is optimized so that different data can be stored on appropriate nodes, thereby improving the storage efficiency of nodes in the distributed hash algorithm, thereby improving the load balancing of nodes, reducing the risk of data loss, and ensuring data retrieval efficiency.

[0051] The distributed storage system built within the enterprise continuously monitors the enterprise's production based on different monitoring devices, thereby generating various types of real-time data, dividing the data types, and treating each data type as a data unit to obtain each data unit.

[0052] Therefore, for each data unit, sort according to the sampling time of different data points, obtain all data points of each data unit, and obtain all historical retrieval records through the system log; obtain the performance data of each performance indicator of each node through the system monitoring tool; therefore, it is necessary to select appropriate nodes for storage of different data units.

[0053] At this point, all data points of each data unit are obtained, all historical search records are acquired, and the performance data of various performance indicators of each node are obtained.

[0054] Step 2: Calculate the storage load rate of each data unit based on the sampling frequency of the data points in each data unit and the number of all data points; analyze the situation of searching any two data units at the same time in the historical search records, and calculate the first correlation degree of the said any two data units.

[0055] Since the growth rates of the number of data points in different data units are different, the data volume of the data units varies. For data units with faster data point growth rates, the corresponding data units have higher storage loads. If high-load data units are allocated to nodes with lower performance, the node load will be too large, affecting the node's response speed and thus reducing data retrieval efficiency.

[0056] Therefore, the storage load rate of each data unit is calculated based on the number of data points in each data unit and the sampling interval between adjacent data points to reflect the data load of the corresponding data node. Specifically,

[0057] Calculate the average time interval between the corresponding sampling times of any two adjacent data points in each data unit, and record it as the average sampling interval;

[0058] The ratio between the number of all data points in each data unit and the average sampling interval is used as the storage load rate of each data unit;

[0059] It should be noted that the larger the number, the larger the amount of data contained in the data unit at this time, and the smaller the average sampling interval, the faster the data growth rate, and the larger the resulting storage load rate, which means that the corresponding data unit has a higher load on the node.

[0060] Secondly, since each node in a distributed storage system can store multiple data units, and different data units often have certain correlations when storing, when retrieving data, multiple data units are often targeted and multiple types of data are retrieved. If the data units with strong correlations are allocated to different nodes at this time, multiple nodes will be accessed when retrieving data, which reduces the data retrieval efficiency.

[0061] Therefore, by analyzing the correlation between the retrieval of different data units in all historical retrieval records, the first correlation degree is calculated to reflect the correlation between the retrieval of different data units in the historical retrieval, specifically:

[0062] The calculation formula for the first correlation degree of any two data units is:

[0063]

[0064] Among them, R1 r,his the first correlation degree between the rth data unit and the hth data unit, U r,h is the number of all historical retrieval records for simultaneously retrieving the rth data unit and the hth data unit, U r is the number of all historical retrieval records for retrieving the rth data unit, U h The number of all historical search records for the h-th data unit is retrieved, and ε is a preset value greater than 0 to avoid the denominator being 0. The value range of ε is (0, 1]. In this embodiment, the value range of ε is 0.1. As other implementation methods, the implementer can set it according to the actual situation.

[0065] It should be noted that the greater the first degree of association, the greater the correlation between the data category characteristics of the rth data unit and the hth data unit. When searching for data, the greater the possibility that the data in these two data units will be retrieved together, which reflects that the rth data unit and the hth data unit should be allocated to the same node for storage.

[0066] At this point, the first correlation degree between the arbitrary two data units is obtained.

[0067] Step 3: Determine the second correlation between the two data units based on the correlation between the changes of adjacent data points in the same sampling period in the two data units and the difference in the changes of data points in the two data units; and determine the retrieval correlation between the two data units by fusing the first correlation and the second correlation.

[0068] Furthermore, the first correlation degree reflects the possibility of two data units being retrieved together, thereby indicating the correlation between the data units; but does not involve the correlation of changes in data points between different data units. Since different data units have different average sampling intervals, the amount of data contained in different data units is inconsistent under the same sampling time. Therefore, by analyzing the related changes in data points between different data units under the same sampling time, the second correlation degree is calculated to more accurately reflect the correlation between different data units. Specifically,

[0069] The data unit with the largest average sampling interval among the two data units is selected as the target unit, and the other data unit is recorded as the reference unit;

[0070] In this embodiment, it is assumed that for the rth data unit and the hth data unit, the average sampling interval of the rth data unit is the largest, and therefore, the rth data unit is the target unit and the hth data unit is the reference unit.

[0071] Record two adjacent data points as a pair of adjacent data points, and based on the sampling period between each pair of adjacent data points in the target unit, select all data points within the sampling period from the reference unit to form a reference data sequence;

[0072] It should be noted that, for ease of understanding, assuming that a pair of adjacent data points in the target unit are a1 and a2, the sampling time corresponding to data point a1 is t1, the sampling time corresponding to data point a2 is t2, and the sampling period between data point a1 and data point a2 is [t1, t2]. Data points collected within the time period [t1, t2] are selected from the reference unit. assuming that a total of 5 data points, namely b1, b2, b3, b4, and b5, are collected in the reference unit within the time period [t1, t2], the reference data sequence is {b1, b2, b3, b4, and b5}, then data point a1 and data point a2 correspond to a reference data sequence in the reference unit, and accordingly, data point a2 and data point a3 correspond to a reference data sequence in the reference unit.

[0073] Calculating the relative change of each pair of adjacent data points in the target unit;

[0074] In this embodiment, taking a pair of adjacent data points a1 and a2 in the target unit as an example, the calculation formula for the relative change is: in, is the relative change between a pair of adjacent data points a1 and a2 in the target unit, is the value of data point a1 in the target cell, is the value of data point a2 in the target cell.

[0075] Calculate the cumulative sum of relative changes between all adjacent data points in the reference data sequence, and record it as the reference change;

[0076] In this embodiment, taking a pair of adjacent data points a1 and a2 in a target unit corresponding to a reference data sequence as an example, the calculation formula for the reference variation is: in, is the reference variation of the reference data sequence corresponding to a pair of adjacent data points a1 and a2 in the target unit, is the data point b in the reference data sequence i The value of is the data point b in the reference data sequence i-1 The value of is the number of all data points in the reference data sequence corresponding to a pair of adjacent data points a1 and a2 in the target unit.

[0077] It should be noted that if the value of a data point of a data unit is not numerical data, the non-numerical data is encoded and converted into numerical data for calculation. The method of encoding non-numerical data is a well-known technology and will not be repeated here.

[0078] The relative changes of all adjacent data points in the target unit are combined into a first change sequence;

[0079] Combining the reference changes of all reference data sequences in the reference unit into a second change sequence;

[0080] calculating a correlation between the first change sequence and the second change sequence;

[0081] In this embodiment, the degree of correlation is measured by calculating the Pearson correlation coefficient between the first change sequence and the second change sequence, wherein the calculation method of the Pearson correlation coefficient is a well-known technology. As other implementation methods, implementers can adopt other methods of the existing technology, such as the Spearman correlation coefficient, etc. This embodiment does not impose any special restrictions on this.

[0082] It should be noted that the elements at the same position between the first change sequence and the second change sequence correspond to the same pair of adjacent data points in the target unit. Assuming that the first element in the first change sequence is The first element in the second change sequence is Both elements correspond to the same pair of adjacent data points a1 and a2 in the target unit. Secondly, the larger the absolute value of the correlation degree is, the greater the correlation between the data changes between the target unit and the reference unit is.

[0083] Furthermore, the second correlation degree is analyzed by analyzing the difference in elements between the first change sequence and the second change sequence, specifically:

[0084] Calculating the difference between each element in the first change sequence and the element at the same position in the second change sequence, and calculating the cumulative sum of the differences of all elements in the first change sequence;

[0085] taking the ratio of the absolute value of the correlation degree to the accumulated sum as the second correlation degree of the arbitrary two data units;

[0086] In this embodiment, the calculation formula for the second correlation degree of any two data units is:

[0087]

[0088] Among them, R2 r,h is the second correlation degree between the rth data unit and the hth data unit, Er,h is the absolute value of the correlation between the rth data unit and the hth data unit, M q is the qth element value in the first change sequence, C q is the value of the qth element in the second change sequence, Q is the number of all elements in the first change sequence, τ is a preset value greater than 0 to avoid the denominator being 0, and the value range of τ is (0,1]. In this embodiment, τ is 1. As other implementation methods, the implementer can set it according to actual conditions.

[0089] It should be noted that the smaller the cumulative sum is, the smaller the difference in the change amplitude of the elements between the first change sequence and the second change sequence is, and the larger the obtained second correlation degree is, which means that the correlation between the data changes between data unit A and data unit B is higher, and at this time, data unit A and data unit B should be assigned to the same node.

[0090] Determining a search relevance based on the first relevance and the second relevance is specifically as follows:

[0091] Normalizing a product of the first degree of association and the second degree of association as a retrieval degree of association between the arbitrary two data units;

[0092] In this embodiment, the sigmoid function is used for normalization processing, wherein the sigmoid function is a well-known technology and will not be described in detail here. As other implementation methods, the implementer may adopt other methods, such as the tanh function, etc., and this embodiment does not impose any special restrictions on this.

[0093] It should be noted that, the greater the retrieval correlation is, the higher the correlation between the corresponding two data units is. In this case, the corresponding two data units should be allocated to the same node for storage.

[0094] At this point, the retrieval correlation between any two data units is obtained.

[0095] Step 4: Sort the nodes according to the performance data of various performance indicators of each node to obtain a node performance sequence; based on the distance relationship between the collection location of each data unit and the node, initially allocate the data units, and sort the nodes according to the storage load rate of all data units allocated to each node to obtain a node load sequence.

[0096] The data collection locations of different data units may be different. The farther the distance between the collection location of each data unit and the node, the lower the data transmission efficiency. Therefore, by analyzing the distance relationship between each data unit and the node, all data units are preliminarily allocated to obtain the node performance sequence and node load sequence, which are as follows:

[0097] Calculate the distance between the collection location of each data unit and the location of each node;

[0098] It should be noted that the locations of the nodes in the distributed storage system are obtained through geographic information system technology, wherein the geographic information system technology is a well-known technology and will not be described in detail here.

[0099] It should be noted that, the smaller the distance is, the closer the distance between the corresponding data unit and the node is.

[0100] All data units are pre-allocated to the nearest node as the initial allocation scheme;

[0101] The sum of the storage load rates of all data units allocated to each node in the initial allocation scheme is used as the total load rate of each node in the initial allocation scheme;

[0102] Furthermore, since different performance indicators of each node have different situations where the larger the performance data, the better, and the smaller the performance data, the better, it is necessary to perform positive processing on the performance data of different performance indicators to make the direction of performance data change consistent. Specifically:

[0103] Perform positive processing on the performance data of all nodes under each performance indicator, arrange all nodes in descending order of performance data, calculate the sum of the rankings of each node in all performance indicators, and arrange all nodes in ascending order according to the sum to form a node performance sequence;

[0104] It should be noted that the forward processing method is a well-known technology and will not be described in detail here; secondly, the smaller the sum value, the better the performance of the corresponding node and the higher the data processing capability of the node. Therefore, the closer the node is to the front in the node performance sequence, the better the corresponding performance.

[0105] Arrange all nodes in descending order according to the total load rate to form a node load sequence;

[0106] It should be noted that the nodes that are closer to the front in the node load sequence have higher corresponding loads.

[0107] Step 5: Analyze the matching between the node performance sequence and the elements in the node load sequence, obtain each matching node and each unmatched node, calculate the schedulability of any data unit based on the retrieval correlation between any data unit assigned to each unmatched node and the remaining data units, combined with the storage load rate, evaluate the initial allocation plan based on the number of matching nodes, iteratively adjust the allocation plan in combination with the schedulability, and perform node allocation on the data units.

[0108] Furthermore, the matching between the node performance sequence and the node load sequence is analyzed, the allocation scheme is evaluated, and the allocation scheme is continuously iteratively adjusted to obtain the optimal allocation scheme, specifically:

[0109] If the node performance sequence is the same as the node at the same position in the node load sequence, then the corresponding node is a matching node, otherwise, it is an unmatched node;

[0110] Count the number of all matching nodes as the node matching degree of the initial allocation plan;

[0111] It should be noted that, for ease of understanding, assume that there are 5 nodes in the distributed storage system, where the node performance sequence is {g2, g4, g1, g5, g3}, and the node load sequence is {g3, g4, g2, g5, g1}. Then the second position in the node performance sequence and the node load sequence is both node g4. Therefore, the fourth position in the node performance sequence and the node load sequence is both node g5. Therefore, the node matching degree is 2.

[0112] It should be noted that the smaller the node matching degree is, the more unreasonable the allocation of data units is, and the more necessary it is to reallocate and adjust the data units.

[0113] If the node matching degree is less than a preset threshold, the allocation scheme is iteratively adjusted; otherwise, the data unit is stored according to the current allocation scheme;

[0114] In this embodiment, the preset threshold is Where N is the number of all nodes in the distributed storage system, It is a rounding-up function. In this embodiment, N is set to 11. For other implementations, the implementer can set it according to actual conditions.

[0115] The process of iteratively adjusting the allocation plan is as follows:

[0116] The schedulability of any data unit allocated to each unmatched node is calculated using the following formula:

[0117]

[0118] Among them, T n,r is the schedulable number of the rth data unit assigned to the nth mismatched node, L n,r RJ is the storage load ratio of the rth data unit assigned to the nth mismatched node, r,h H is the retrieval correlation between the rth data unit and the hth data unit assigned to the nth unmatched node, nis the number of all data units allocated to the nth unmatched node, ∈ is a preset value greater than 0 to avoid the denominator being 0, and the value range of ∈ is (0,1]. In this embodiment, ∈ is 0.1. As other implementation methods, the implementer can set it according to the actual situation.

[0119] The largest schedulable data unit under all unmatched nodes is allocated to the node with the second closest distance as the next allocation scheme. Through iterative calculation, until the node matching degree meets the preset threshold, the optimal allocation scheme is obtained. The data units are allocated to nodes according to the optimal allocation scheme, so that the load of different nodes is balanced and the efficiency of data retrieval is improved. The step flow chart of the method for obtaining the optimal allocation scheme provided in the embodiment of the present application is as follows Figure 2 shown.

[0120] The hash value is calculated for the data unit allocated to each node, and the index relationship between the stored hash value and the data storage location is stored to quickly retrieve the data, thereby improving the efficiency of data retrieval.

[0121] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0122] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0123] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the present application. It should be noted that a person skilled in the art can make various modifications and improvements without departing from the spirit of the present application. Therefore, any simple modifications, equivalent variations, and modifications to the above embodiments made in accordance with the technical essence of the present application without departing from the content of the present application's technical solution fall within the scope of protection of the present application's technical solution.

Claims

1. A method for accessing and retrieving data within an enterprise, characterized in that: The method comprises the following steps: Divide various types of data within the enterprise into data units, obtain all data points of each data unit, all historical search records, and performance data of various performance indicators of each node in the distributed storage system; Calculate the storage load rate of each data unit based on the sampling frequency and number of data points in each data unit; Analyze the situation in which any two data units are searched at the same time in the historical search records, and calculate the first correlation degree of the any two data units; Determining a second correlation degree between the two data units based on correlation between changes in adjacent data points in the same sampling period in the two data units and differences in changes in data points in the two data units; and determining a retrieval correlation degree between the two data units by fusing the first correlation degree and the second correlation degree. The nodes are sorted according to the performance data of various performance indicators of each node to obtain a node performance sequence; the data units are initially allocated based on the distance relationship between the collection location of each data unit and the node, and the nodes are sorted according to the storage load rate of all data units allocated to each node to obtain a node load sequence; Analyze the matching status of the elements in the node performance sequence and the node load sequence, obtain each matching node and each unmatched node, and calculate the schedulability of any data unit assigned to each unmatched node based on the retrieval correlation between the data unit and the remaining data units, combined with the storage load rate; evaluate the initial allocation plan based on the number of matching nodes, iteratively adjust the allocation plan in combination with the schedulability, and perform node allocation on the data units.

2. The method for accessing and retrieving internal enterprise data according to claim 1, wherein: Calculating the storage load rate of each data unit includes: Calculate the average time interval between the corresponding sampling times of any two adjacent data points in each data unit, and record it as the average sampling interval; The storage load ratio is the ratio between the number of all data points in each data unit and the average sampling interval.

3. The method for accessing and retrieving internal enterprise data according to claim 1, wherein: The first correlation R1 between the rth data unit and the hth data unit r,h The calculation formula is: Among them, U r,h is the number of all historical retrieval records for simultaneously retrieving the rth data unit and the hth data unit, U r is the number of all historical retrieval records for retrieving the rth data unit, U h is the number of all historical search records for retrieving the h-th data unit, and ε is a preset value greater than 0.

4. The method for accessing and retrieving internal enterprise data according to claim 2, wherein: The determining the second degree of association of the arbitrary two data units includes: The data unit with the largest average sampling interval among the two data units is selected as the target unit, and the other data unit is recorded as the reference unit; Record two adjacent data points as a pair of adjacent data points, and calculate the relative change of each pair of adjacent data points in the target unit; Based on a sampling period between each pair of adjacent data points in the target unit, all data points within the sampling period are selected from the reference unit to form a reference data sequence; Calculate the sum of relative changes between all adjacent data points in the reference data sequence, and record it as a reference change; The relative changes of all adjacent data points in the target unit are combined into a first change sequence; the reference changes of all reference data sequences in the reference unit are combined into a second change sequence; calculating a correlation between the first change sequence and the second change sequence; Calculating the difference between each element in the first change sequence and the element at the same position in the second change sequence, and calculating the cumulative sum of the differences of all elements in the first change sequence; The second correlation degree is a ratio of an absolute value of the correlation degree to the cumulative sum.

5. The method for accessing and retrieving internal enterprise data according to claim 1, wherein: The further acquisition process of the node performance sequence is as follows: The performance data of all nodes under each performance indicator are forward-processed, and all nodes are arranged in descending order of performance data. The sum of the rankings of each node in all performance indicators is calculated, and all nodes are arranged in ascending order according to the sum to form a node performance sequence.

6. The method for accessing and retrieving internal enterprise data according to claim 1, wherein: The further acquisition process of the node load sequence is as follows: Calculate the distance between the collection location of each data unit and the location of each node; All data units are pre-allocated to the nearest node as the initial allocation scheme; The sum of the storage load rates of all data units allocated to each node in the initial allocation scheme is used as the total load rate of each node in the initial allocation scheme; All nodes are arranged in descending order according to the total load rate to form a node load sequence.

7. The method for accessing and retrieving internal enterprise data according to claim 1, wherein: The obtaining of each matching node and each unmatched node includes: If the node load sequence is the same as the node at the same position in the node performance sequence, the corresponding node is a matching node; otherwise, it is an unmatched node.

8. The method for accessing and retrieving internal enterprise data according to claim 1, wherein: The calculation formula for the schedulability of any data unit is: Among them, T n,r is the schedulable number of the rth data unit assigned to the nth mismatched node, L n,r RJ is the storage load ratio of the rth data unit assigned to the nth mismatched node, r,h H is the retrieval correlation between the rth data unit and the hth data unit assigned to the nth unmatched node, n is the number of all data units allocated to the nth unmatched node, and ∈ is a preset value greater than 0.

9. The method for accessing and retrieving internal enterprise data according to claim 1, wherein: The evaluation of the initial allocation plan includes: Count the number of all matching nodes as the node matching degree of the initial allocation plan; If the node matching degree is less than a preset threshold, the allocation scheme is iteratively adjusted; otherwise, the data unit is stored according to the current allocation scheme.

10. The method for accessing and retrieving internal enterprise data according to claim 9, wherein: The iterative adjustment of the allocation scheme includes: Allocating the largest schedulable data unit under all unmatched nodes to the second-closest node as the next allocation plan; By continuously iteratively adjusting the allocation plan, the optimal allocation plan is obtained, and the data units are allocated to nodes according to the optimal allocation plan.

Images