Background data centralized caching method and device, equipment and storage medium

Abstract

The invention discloses a background data centralized caching method, device and equipment and a storage medium, the method is applied to a background data caching system.The method comprises the steps that when a new loading task is added to the downstream, a mulberry-based graph is drawn according to all fields in the new loading task and a current cluster task, and the similarity among all nodes in the mulberry-based graph is determined; updating the background data of the background data cache system according to the similarity among the nodes in the mulberry-based graph; obtaining a data request record and a downstream pre-estimated access amount, and determining the business importance, the access amount and the response time of a cluster task corresponding to the background data; and performing cold and hot data splitting on the background data based on the business importance, the page view and the response time, and storing the background data according to a splitting result. The background data of the background data caching system is updated according to the similarity among the nodes in the mulberry-based graph, and cold and hot data splitting is performed on the background data, so that the waste of hardware resources is reduced, and the storage efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and in particular to a background data centralized caching method, device, equipment and storage medium. Background Art

[0002] In the process of retail customer operations, various background data are needed, such as customer tags, customer asset information, financial product definitions, etc. Each business party needs to load data files into cache, database, distributed database and other storage media for application services, and thus faces the following problems: some high-frequency background data is loaded multiple times, and the full amount of data is stored in expensive cache, wasting a lot of storage resources; for the same background data, there will be differences in storage format and selected fields after loading. If the storage format is not unified before loading, it will be difficult to identify whether the data is duplicated through the cached background data later. Even if duplicate data is identified later, there is a cost and risk of transformation due to inconsistent storage formats, which makes data governance difficult and storage efficiency low.

[0003] Therefore, there is an urgent need for a centralized caching method for background data that can improve storage efficiency and reduce hardware resource waste. Summary of the invention

[0004] The main purpose of the present invention is to provide a background data centralized caching method, device, equipment and storage medium, aiming to solve the technical problems of hardware resource waste and low storage efficiency of background data caching in the prior art.

[0005] To achieve the above object, the present invention provides a background data centralized caching method, which is applied to a background data caching system and comprises:

[0006] When a new loading task is added downstream, a Sankey diagram is drawn according to the fields in the new task and the current cluster task, and the similarity between the nodes in the Sankey diagram is determined;

[0007] Update the background data of the background data cache system according to the similarity between the nodes in the Sankey diagram;

[0008] Obtain data request records and estimated downstream access volume, and determine the business importance, access volume, and response time of the cluster task corresponding to the background data;

[0009] The background data is split into cold data and hot data based on the business importance, access volume and response time, and the background data is stored according to the split result.

[0010] Optionally, after the step of splitting the background data into hot and cold data based on the business importance, access volume and response time, and storing the background data according to the splitting result, the method further includes:

[0011] Determine a sampling ratio of the sampled data amount in the background data, and randomly select data of the sampling ratio from the background data;

[0012] Try writing the data of the sampling ratio to the sampling cluster, and determine the memory usage of the sampling cluster before and after the data is tried written;

[0013] Determine the cluster memory difference before and after the data trial write based on the memory usage;

[0014] Determine the memory occupancy of the background data based on the cluster memory difference and the sampling ratio;

[0015] The remaining memory of each cluster in the background data cache system is traversed, and cluster allocation is performed based on the remaining memory and the memory occupancy.

[0016] Optionally, when a new loading task is added downstream, the step of drawing a Sankey diagram according to the fields in the new task and the current cluster task, and determining the similarity between the nodes in the Sankey diagram includes:

[0017] When a new loading task is added downstream, the data source table and upstream source table corresponding to the new task and the background data corresponding to the current cluster task are obtained;

[0018] Draw a Sankey diagram by using the data source table, the upstream source table and each field in the background data as nodes;

[0019] The similarity between the nodes in the Sankey diagram is calculated by the SimRank graph similarity algorithm.

[0020] Optionally, the formula corresponding to the SimRank graph similarity algorithm is:

[0021]

[0022] In the formula, R k+1 (α, β) represents the similarity between node α and node β in the Sankey diagram. i (α) represents the child node of node α, I j (β) represents the child nodes of node β, i and j represent the number of child nodes of node α and node β respectively, R k (I i (α),I j (β)) represents child node I i (α) and child node I jThe similarity between (β).

[0023] Optionally, the step of splitting the background data into hot and cold data based on the business importance, access volume and response time, and storing the background data according to the split result includes:

[0024] Performing statistical analysis on the access volume and response time of the cluster task to obtain analysis results;

[0025] Determining the importance of the cluster task, and classifying the background data corresponding to the cluster task based on the importance of the cluster task to obtain a classification result;

[0026] According to the analysis result and the classification result, a target splitting strategy is selected from preset cold and hot data splitting strategies;

[0027] The background data corresponding to the cluster task is split into cold data and hot data based on the target splitting strategy, and the background data is stored according to the splitting result.

[0028] Optionally, the step of performing statistical analysis on the access volume and response time of the cluster task to obtain the analysis result includes:

[0029] Performing statistical analysis on the access volume of the cluster task to determine whether there is a business peak period for the cluster task, and obtaining access volume analysis results;

[0030] Determine whether the background data corresponding to the cluster task has a response time requirement according to the response time of the cluster task, and obtain a response time analysis result;

[0031] The access volume analysis result and the response time analysis result are used as analysis results.

[0032] Optionally, the method further comprises:

[0033] Using a timed polling method to detect the current status of each cluster in the background data cache system;

[0034] When the state of the cluster is detected as faulty for a preset number of consecutive times, updating the cluster state of the cluster to faulty;

[0035] The corresponding cluster tasks in the cluster are reallocated, and the query service is adaptively switched based on the current state of the cluster.

[0036] In addition, to achieve the above-mentioned purpose, the present invention also proposes a background data centralized caching device, which is applied to a background data caching system, and comprises:

[0037] The data identification module is used to draw a Sankey diagram based on the fields in the newly added task and the current cluster task when a new loading task is added downstream, and to determine the similarity between the nodes in the Sankey diagram;

[0038] A data updating module, used for updating the background data of the background data cache system according to the similarity between the nodes in the Sankey diagram;

[0039] A data statistics module is used to obtain data request records and estimated downstream access volume, and determine the business importance, access volume and response time of the cluster task corresponding to the background data;

[0040] The splitting storage module is used to split the background data into hot and cold data based on the business importance, access volume and response time, and store the background data according to the splitting result.

[0041] In addition, to achieve the above-mentioned purpose, the present invention also proposes a background data centralized caching device, which includes: a memory, a processor, and a background data centralized caching program stored in the memory and executable on the processor, wherein the background data centralized caching program is configured to implement the steps of the background data centralized caching method described above.

[0042] In addition, to achieve the above objectives, the present invention further proposes a storage medium, on which a background data centralized caching program is stored, and when the background data centralized caching program is executed by a processor, the steps of the background data centralized caching method described above are implemented.

[0043] The present invention discloses that when a new loading task is added downstream, a Sankey diagram is drawn according to each field in the new task and the current cluster task, and the similarity between each node in the Sankey diagram is determined; the background data of the background data cache system is updated according to the similarity between each node in the Sankey diagram; data request records and estimated access volume of the downstream are obtained, and the business importance, access volume and response time of the cluster task corresponding to the background data are determined; the background data is split into hot and cold data based on the business importance, access volume and response time, and the background data is stored according to the split result. Since the present invention draws a Sankey diagram according to each field in the new task and the current cluster task when a new loading task is added downstream, and the background data of the background data cache system is updated according to the similarity between each node in the Sankey diagram, and the background data is split into hot and cold data based on the business importance, access volume and response time, and the background data is stored according to the split result, compared with the prior art, the present invention promotes data reuse in the downstream, reduces hardware resource waste, and improves storage efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1It is a flowchart of the first embodiment of the background data centralized caching method of the present invention;

[0045] Figure 2 A schematic diagram of using the access statistics as a basis for splitting hot and cold data in the background data centralized caching method of the present invention;

[0046] Figure 3 It is a flowchart of a second embodiment of the background data centralized caching method of the present invention;

[0047] Figure 4 A schematic diagram of a process of adaptive resource scheduling in a centralized caching method for background data of the present invention;

[0048] Figure 5 It is a flowchart of a third embodiment of the background data centralized caching method of the present invention;

[0049] Figure 6 It is a schematic diagram of the overall framework of the system in the background data centralized caching method of the present invention;

[0050] Figure 7 It is a structural block diagram of a first embodiment of a background data centralized caching device of the present invention;

[0051] Figure 8 It is a structural diagram of a background data concentration cache device in a hardware operating environment involved in an embodiment of the present invention.

[0052] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0053] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0054] The embodiment of the present invention provides a background data centralized caching method, referring to Figure 1 , Figure 1 FIG. 4 is a flow chart of a first embodiment of a background data centralized caching method according to the present invention.

[0055] In this embodiment, the background data centralized caching method includes steps S10 to S40:

[0056] Step S10: When a new loading task is added downstream, a Sankey diagram is drawn according to the fields in the new task and the current cluster task, and the similarity between the nodes in the Sankey diagram is determined.

[0057] It should be noted that the execution subject of this embodiment can be a computer server device with data processing, network communication and program running functions applied to various background data caching scenarios, such as a personal computer, a laptop computer, a mobile phone, a smart watch, etc., or an electronic device capable of realizing the above functions, a background data centralized caching device, etc. The following takes a system including a background data centralized caching device (hereinafter referred to as the system) as an example to illustrate this embodiment and the following embodiments.

[0058] It should be explained that when a new loading task is added downstream, it is difficult to identify whether the same background data already exists on the platform only by the tag file name and field name. In order to quickly identify data from the same source and avoid repeated loading, the following steps can be used to automatically identify similar data and promote data reuse:

[0059] (1) According to the label file (various fields) configured on the front-end management platform, query the source table information from the interface provided by the system. According to the source table name and field name, query all upstream source tables of each field from the provided interface. Finally, visualize the relationship between nodes through the following Sankey diagram. Platform administrators and task leaders can quickly identify data with exactly the same source.

[0060] (2) When a user adds a new task downstream, during the task approval phase, the platform administrator can perform traceability to immediately calculate and update the Sankey diagram and quickly view the relationship between the new task and the existing tasks.

[0061] (3) Sankey diagrams can quickly identify data from the same source (two different table fields are derived from the same source table field), but it is difficult to determine the similarity of data from non-identical sources (two different table fields are derived from multiple source table fields, one of which is the same and the others are different). The SimRank graph similarity algorithm is used to calculate the similarity between fields. The basic idea of ​​SimRank is that there is a similar relationship between nodes that converge to the same child node. The specific algorithm is as follows:

[0062] The similarity R between two nodes α and β in the Sankey diagram k+1 (α,β) are all the child nodes I of node α 1 (α), I 2 (α)…I i (α), all child nodes I of its node β 1 (β), I 2 (β)…I j (β) Similarity R between two k (I 1 (α),I 1 (β))、R k (I 1 (α),I2 (β))…R k (I i (α),I j (β)) is accumulated and then multiplied by a compensation value, Δ divided by the product of the number of all child nodes of node α ||I(α)|| and the number of all child nodes of node β ||I(β)||:

[0063]

[0064] In order to avoid the problem that the more child nodes two nodes have, the lower their similarity will be even if they are all the same, an additional compensation value is added The more intersections there are between the two, the more compensation is obtained. The final calculation formula is as follows. The similarity between the nodes of the entire Sankey diagram is obtained through multiple iterations:

[0065]

[0066] The source table fields are sorted from high to low to display the top 5 similar nodes. Nodes whose similarity exceeds the threshold are marked in red to help administrators and task leaders quickly identify highly similar fields.

[0067] In a specific implementation, when a new loading task is added downstream, the data source table and upstream source table corresponding to the new task and the background data corresponding to the current cluster task can be obtained; each field in the data source table, the upstream source table and the background data is used as a node to draw a Sankey diagram; and the similarity between the nodes in the Sankey diagram is calculated using the SimRank graph similarity algorithm.

[0068] It should be explained that the formula corresponding to the SimRank graph similarity algorithm is:

[0069]

[0070] In the formula, R k+1 (α, β) represents the similarity between node α and node β in the Sankey diagram. i (α) represents the child node of node α, I j (β) represents the child nodes of node β, i and j represent the number of child nodes of node α and node β respectively, R k (I i (α),I j (β)) represents child node I i (α) and child node I j The similarity between (β).

[0071] Step S20: updating the background data of the background data cache system according to the similarity between the nodes in the Sankey diagram.

[0072] It should be understood that by querying the data source table and all its upstream source tables, visually displaying the Sankey diagram, and using the SimRank graph similarity algorithm to calculate the similarity between nodes and display them in order from high to low, in the new task approval stage, completely identical or highly similar data can be quickly identified, and then the background data of the background data cache system can be updated according to the similarity between the nodes in the Sankey diagram, which can promote data reuse downstream and reduce waste of hardware resources.

[0073] Step S30: obtaining data request records and estimated downstream access volume, and determining the business importance, access volume and response time of the cluster task corresponding to the background data.

[0074] It should be noted that before data storage, the access volume of the cluster task corresponding to the above background data can be based on the estimated access volume of the downstream; after data storage, the access volume of the cluster task corresponding to the above background data can be adjusted according to the actual access volume of the downstream (i.e., data request records).

[0075] Step S40: splitting the background data into hot and cold data based on the business importance, access volume and response time, and storing the background data according to the splitting result.

[0076] It should be noted that the system has built task allocation, resource scheduling, query switching, authentication flow control, data traceability, configuration center, monitoring center, etc. to ensure overall stability and robustness.

[0077] It should be understood that some high-frequency background data is loaded multiple times, and the hot and cold data are not split according to the access frequency and timeliness of the background data. The full amount of data is stored in the expensive cache, wasting a lot of storage resources. Therefore, by splitting the hot and cold data according to the business importance, access volume and response time of the cluster tasks corresponding to the background data, only the hot data can be stored in the Redis cluster, thereby improving storage efficiency and avoiding the waste of storage resources.

[0078] For example, refer to Figure 2 , Figure 2 This is a schematic diagram of using the access statistics as the basis for splitting hot and cold data in the centralized caching method of the background data of the present invention. The system is configured with external services, a configuration center, query switching, a cache, MySQL, a monitoring center, and a management platform; the external service has a built-in request recording module, which reports to the configuration center at regular intervals; the configuration center receives the reporting request, performs group statistics and caches it according to the task dimension, and updates the cache to MySQL at regular intervals, and the management platform is used to visualize the access statistics and query switching results obtained by the monitoring center.

[0079] To further improve storage utilization, this system uses "two dimensions and six elements" to make decisions on hot and cold data, covering all current downstream business scenarios. It should be noted that the two dimensions include: 1) whether it is important or not; 2) whether there is a business peak period. The six elements include: 1) important data; 2) non-important data + response time requirements; 3) non-important data + no response time requirements; 4) business peak period; 5) no business peak period, daily high access volume; 6) no business peak period, daily low access volume, for example, refer to Table 1, Table 1 is a cold and hot data splitting strategy table.

[0080] Table 1 Cold and hot data splitting strategy table

[0081]

[0082]

[0083] It should be noted that during business peak periods, the access volume of some tasks is concentrated at the beginning of the month. This type of background data is only stored in Redis as hot data during the estimated business peak period; the access volume does not have obvious periodicity, and the hot and cold data can be divided according to the average daily access volume of the background data by downstream applications in the past 30 days; some background data has no query response time requirements and can be set as normally cold data and not included in the statistics; important data is stored in multiple Redis, which can further reduce the risk of business impact caused by sudden underlying storage failures.

[0084] It should be explained that multiple Redis storages support the configuration of multiple Redis cluster backups for important tasks and tasks with high traffic volume, which reduces the traffic volume of a single cluster and the risk of sudden underlying storage failures. In addition, query services are deployed in multiple zones to reduce the risk of business impact caused by underlying container failures.

[0085] In a specific implementation, a statistical analysis can be performed on the access volume and response time of the cluster task to obtain an analysis result; the importance of the cluster task is determined, and the background data corresponding to the cluster task is classified based on the importance of the cluster task to obtain a classification result; according to the analysis result and the classification result, a target splitting strategy is selected from the preset cold and hot data splitting strategies; based on the target splitting strategy, the background data corresponding to the cluster task is split into cold and hot data, and the background data is stored according to the splitting result. The specific content of the preset cold and hot data splitting strategy is shown in Table 1.

[0086] It needs to be explained that the above-mentioned steps of performing statistical analysis on the access volume and response time of the cluster task and obtaining the analysis results include: performing statistical analysis on the access volume of the cluster task to determine whether there is a business peak period for the cluster task, and obtaining the access volume analysis results; determining whether there is a response time requirement for the background data corresponding to the cluster task based on the response time of the cluster task, and obtaining the response time analysis results; and using the access volume analysis results and the response time analysis results as the analysis results.

[0087] This embodiment discloses that when a new loading task is added downstream, a Sankey diagram is drawn according to each field in the new task and the current cluster task, and the similarity between each node in the Sankey diagram is determined; the background data of the background data cache system is updated according to the similarity between each node in the Sankey diagram; the data request record and the estimated access volume of the downstream are obtained, and the business importance, access volume and response time of the cluster task corresponding to the background data are determined; the background data is split into hot and cold data based on the business importance, access volume and response time, and the background data is stored according to the split result. Since this embodiment draws a Sankey diagram according to each field in the new task and the current cluster task when a new loading task is added downstream, and the background data of the background data cache system is updated according to the similarity between each node in the Sankey diagram, and the background data is split into hot and cold data based on the business importance, access volume and response time, and the background data is stored according to the split result, compared with the prior art, this embodiment promotes data reuse downstream, reduces hardware resource waste, and improves storage efficiency.

[0088] refer to Figure 3 , Figure 3 FIG. 4 is a flow chart of a second embodiment of a background data centralized caching method according to the present invention.

[0089] Based on the first embodiment, in this embodiment, after step S40, steps S501 to S505 are further included:

[0090] Step S501: determining a sampling ratio of the sampled data volume in the background data, and randomly selecting data of the sampling ratio from the background data.

[0091] Step S502: Try writing the data of the sampling ratio to the sampling cluster, and determine the memory usage of the sampling cluster before and after the data is tried written.

[0092] Step S503: Determine the cluster memory difference before and after the data trial writing based on the memory usage.

[0093] Step S504: determining the memory occupancy of the background data based on the cluster memory difference and the sampling ratio.

[0094] Step S505: traverse the remaining memory of each cluster in the background data cache system, and perform cluster allocation based on the remaining memory and the memory occupancy.

[0095] For example, refer to Figure 4 , Figure 4 The flowchart of adaptive resource scheduling in the background data centralized caching method of the present invention is shown in the figure. In the figure, Task Ready indicates the starting point of the process, indicating that Task 1 is ready to be executed; File Arrival indicates that all files required to execute the task are ready and complete; Trial Write (sampling x rows) indicates that before starting to write data formally, a small-scale trial write operation is performed first, which may be to verify whether the data format or structure of the file meets expectations; Capacity Statistics before Trial Write indicates that before the trial write, the system counts the capacity of the target cluster to estimate whether more resources are needed; Standard files are read from the task loading service through SFTP (a secure file transfer protocol); Actual Write indicates that after the trial write is verified to be correct, the system transfers the data from The task load service actually writes to the estimated Redis; the capacity statistics after the test write means that after the write is completed, the capacity of Redis is counted again to track the usage of resources; when allocating cluster resources, the system will sort the clusters from most to least according to the remaining memory resources of the cluster (minus the allocated unloaded memory). The purpose of this is to use resources more efficiently and assign tasks to clusters with more remaining resources; the return cluster step marks the completion of the resource allocation and processing phase of the task, and may mean that the task will enter the next processing phase or has been successfully executed; the assignment cluster means that the task is assigned to a specific cluster for processing. As can be seen from the picture, cluster 1 and cluster 2 are used for this processing process. This shows that the system has multiple processing clusters to balance the load and ensure high availability.

[0096] It should be noted that in this embodiment, memory usage is estimated by sampling: by sampling calculation, a specified number of rows of data are tried to be written to the sampling cluster, and the memory difference before and after the trial writing is calculated, so as to estimate the memory usage by dividing the memory difference by the row number ratio. The overall situation of the current resources of each cluster is obtained by traversing, and the estimated memory of the allocated but unexecuted tasks is subtracted, and the remaining memory size is sorted in descending order to achieve Redis cluster selection.

[0097] This embodiment discloses determining the sampling ratio of the sampled data volume in the background data, and randomly selecting the data of the sampling ratio from the background data; trial writing the data of the sampling ratio to the sampling cluster, and determining the memory usage of the sampling cluster before and after the data trial writing; determining the cluster memory difference before and after the data trial writing based on the memory usage; determining the memory occupancy of the background data based on the cluster memory difference and the sampling ratio; traversing the memory remaining amount of each cluster in the background data cache system, and performing cluster allocation based on the memory remaining amount and the memory occupancy. Compared with the prior art, this embodiment automatically allocates clusters by sampling and estimating memory occupancy, thereby realizing adaptive resource scheduling of storage resources.

[0098] refer to Figure 5 , Figure 5 FIG. 4 is a flow chart of a third embodiment of a background data centralized caching method according to the present invention.

[0099] Based on the above embodiments, in this embodiment, the background data centralized caching method further includes steps S601 to S603:

[0100] Step S601: Detect the current status of each cluster in the background data cache system by means of periodic polling.

[0101] Step S602: when the state of the cluster is detected as faulty for a preset number of consecutive times, the cluster state of the cluster is updated to faulty.

[0102] Step S603: reallocate the corresponding cluster tasks in the cluster, and adaptively switch the query service based on the current state of the cluster.

[0103] It should be noted that the cluster health status is detected regularly. When a fault is detected multiple times in a row, the cluster status is updated to "fault", and the corresponding cluster tasks are automatically reallocated. Before the allocation is completed, the query switches to read the HBase backup data, and the alarm platform administrator is called. If the task migration is successful, the query automatically switches to the new Redis cluster. The system automatically realizes that no manual intervention is required, which greatly reduces the business impact time of sudden underlying storage failures. The system's monitoring center visualizes indicators such as cluster status and automatically sends alarms when abnormalities occur.

[0104] It should be added that this embodiment implements query truncation processing, and configures the maximum request timeout in Redis and HBase to avoid the problem that the request has not been returned due to the failure of the underlying storage. It supports the task dimension to configure whether the Redis query space penetrates the query HBase to meet the different business needs of downstream users. The system administrator can use the platform to operate the Redis cluster online and offline with one click, the task automatically reallocates the cluster, the query automatically switches, and the administrator communicates with the task leader to confirm and realize the cluster migration of the specified task.

[0105] It should be noted that important tasks and high-access tasks support the configuration of multiple Redis cluster backups to reduce the access volume of a single cluster and the risk of sudden underlying storage failures. In addition, this system supports multiple query methods: HTTP single query interface, HTTP batch query interface, and query SDK.

[0106] For example, refer to Figure 6 , Figure 6 The schematic diagram of the overall framework of the system in the background data centralized caching method of the present invention. In the figure, the overall framework of the system includes a data layer, an execution layer, an interaction layer, a configuration center, and a monitoring center. The data layer includes MySQL for storage configuration, Redis and HBase for storage of background data, and SFTP for background data source. The execution layer includes task allocation Redis, cold and hot data splitting, loading jobs, resource scheduling, query switching, and data traceability. The interaction layer includes: loading job management platform UI: downstream users can easily and quickly query each loading job, and directly control the loading task by configuring a new loading job on the platform. The platform administrator can view the overall operation of the system through monitoring; a variety of external services are provided, which are suitable for downstream users with different access volumes; external services are built-in authentication and flow control modules, and are deployed in multiple zones to ensure the overall stability of the system; single HTTP: provided to downstream with low access volume; batch HTTP: supports querying multiple records of a single task at a time (up to 1000 records), and multi-threaded query is enabled inside the service to improve performance, and is provided to downstream with high access volume; SDK: supports querying data directly from Redis, avoiding network interaction IO overhead, and providing it to downstream with extremely high access volume.

[0107] Figure 6The execution layer in includes the following modules: Task allocation Redis service: Automatically allocate clusters by sampling and estimating memory usage. Hot and cold data splitting service: Count the number of visits as the basis for splitting hot and cold data, and adopt different splitting strategies according to the business meaning of the background data. Loading job service: Supports loading SFTP data into Redis and HBase according to the specified splitting strategy. Cluster management and resource scheduling service: Regularly detect indicators such as cluster health status, and automatically implement cluster migration and query switching when cluster failures are detected, without manual intervention, which greatly reduces the business impact time of sudden underlying storage failures. Important tasks and high-access tasks support the configuration of multiple Redis cluster backups. Query switching service: Adaptively query from Redis and HBase according to hot and cold data splitting strategies, cluster health status, etc. Monitoring center: Monitor and visualize indicators such as Redis cluster resource status, task execution status, external interface requests, etc. Abnormal indicators trigger automatic alarm platform administrators. Data traceability service: Through technical means, query the data source table and all its upstream source tables, visualize the Sankey diagram, and calculate the similarity between nodes through the SimRank graph similarity algorithm and display the TOP5 from high to low. In the approval stage of new tasks, quickly identify data with exactly the same source or highly similar data to promote data reuse downstream.

[0108] This embodiment discloses the use of a timed polling method to detect the current status of each cluster in the background data cache system; when the state of the cluster is detected as a fault for a preset number of consecutive times, the cluster state of the cluster is updated to a fault; the corresponding cluster tasks in the cluster are reallocated, and the query service is adaptively switched based on the current state of the cluster. Since this embodiment detects the health status of the cluster regularly, when a fault is detected for multiple consecutive times, the cluster state is updated to a fault, the corresponding cluster task is automatically reallocated, and the query is switched to read the HBase backup data before the allocation is completed. At the same time, the query service is adaptively switched based on the current state of the cluster. Compared with the prior art, the risk of business impact caused by underlying storage failures is reduced.

[0109] In addition, an embodiment of the present invention further provides a storage medium, on which a background data centralized caching program is stored. When the background data centralized caching program is executed by a processor, the steps of the background data centralized caching method described above are implemented.

[0110] Reference Figure 7 , Figure 7 FIG. 1 is a structural block diagram of a first embodiment of a background data centralized caching device according to the present invention.

[0111] like Figure 7As shown, the background data centralized caching device proposed in the embodiment of the present invention includes: a data identification module 701, a data update module 702, a data statistics module 703 and a split storage module 704.

[0112] The data identification module 701 is used to draw a Sankey diagram according to the fields in the newly added task and the current cluster task when a loading task is added downstream, and to determine the similarity between the nodes in the Sankey diagram.

[0113] The data updating module 702 is used to update the background data of the background data cache system according to the similarity between the nodes in the Sankey diagram.

[0114] The data statistics module 703 is used to obtain data request records and estimated downstream access volume, and determine the business importance, access volume and response time of the cluster task corresponding to the background data.

[0115] The split storage module 704 is used to split the background data into hot and cold data based on the business importance, access volume and response time, and store the background data according to the split result.

[0116] The data identification module 701 is also used to obtain the data source table and upstream source table corresponding to the newly added task and the background data corresponding to the current cluster task when a new loading task is added downstream; draw a Sankey diagram using each field in the data source table, the upstream source table and the background data as nodes; and calculate the similarity between the nodes in the Sankey diagram using the SimRank graph similarity algorithm.

[0117] The split storage module 704 is also used to perform statistical analysis on the access volume and response time of the cluster task to obtain analysis results; determine the importance of the cluster task, and classify the background data corresponding to the cluster task based on the importance of the cluster task to obtain classification results; select a target splitting strategy from preset cold and hot data splitting strategies based on the analysis results and the classification results; perform cold and hot data splitting on the background data corresponding to the cluster task based on the target splitting strategy, and store the background data according to the splitting results.

[0118] The split storage module 704 is also used to perform statistical analysis on the access volume of the cluster task, determine whether the cluster task has a business peak period, and obtain the access volume analysis result; determine whether the background data corresponding to the cluster task has a response time requirement based on the response time of the cluster task, and obtain the response time analysis result; use the access volume analysis result and the response time analysis result as the analysis result.

[0119] The embodiment of the device discloses that when a new loading task is added downstream, a Sankey diagram is drawn according to each field in the new task and the current cluster task, and the similarity between each node in the Sankey diagram is determined; the background data of the background data cache system is updated according to the similarity between each node in the Sankey diagram; the data request record and the estimated access volume of the downstream are obtained, and the business importance, access volume and response time of the cluster task corresponding to the background data are determined; the background data is split into hot and cold data based on the business importance, access volume and response time, and the background data is stored according to the split result. Since the embodiment of the device draws a Sankey diagram according to each field in the new task and the current cluster task when a new loading task is added downstream, and the background data of the background data cache system is updated according to the similarity between each node in the Sankey diagram, and the background data is split into hot and cold data based on the business importance, access volume and response time, and the background data is stored according to the split result, compared with the prior art, the embodiment of the device promotes downstream data reuse, reduces hardware resource waste, and improves storage efficiency.

[0120] Based on the above-mentioned first embodiment of the centralized caching device for background data of the present invention, a second embodiment of the centralized caching device for background data of the present invention is proposed.

[0121] In this embodiment, the split storage module 704 is also used to determine the sampling ratio of the sampled data volume in the background data, and randomly select the data of the sampling ratio from the background data; try to write the data of the sampling ratio to the sampling cluster, and determine the memory usage of the sampling cluster before and after the data try to write; determine the cluster memory difference before and after the data try to write based on the memory usage; determine the memory occupancy of the background data based on the cluster memory difference and the sampling ratio; traverse the memory remaining amount of each cluster in the background data cache system, and perform cluster allocation based on the memory remaining amount and the memory occupancy.

[0122] Other embodiments or specific implementations of the background data centralized caching device of the present invention can refer to the above-mentioned method embodiments, which will not be described in detail here.

[0123] The present application provides a background data centralized caching device, which includes: at least one processor; and a memory that is communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the background data centralized caching method in the above-mentioned embodiment 1.

[0124] Reference below Figure 8, which shows a schematic diagram of the structure of a background data centralized caching device suitable for implementing an embodiment of the present application. The background data centralized caching device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The background data centralized cache device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0125] like Figure 8 As shown, the background data centralized cache device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the background data centralized cache device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the background data centralized cache device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a background data centralized cache device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0126] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0127] The background data centralized caching device provided by the present application adopts the background data centralized caching method in the above embodiment, which can solve the technical problems of hardware resource waste and low storage efficiency of background data caching in the prior art. Compared with the prior art, the beneficial effects of the background data centralized caching device provided by the present application are the same as the beneficial effects of the background data centralized caching method provided by the above embodiment, and other technical features in the background data centralized caching device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0128] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0129] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0130] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0131] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0132] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0133] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A background data centralized caching method, characterized in that: The method is applied to a background data cache system, and the method comprises: When a new loading task is added downstream, a Sankey diagram is drawn according to the fields in the new task and the current cluster task, and the similarity between the nodes in the Sankey diagram is determined; Update the background data of the background data cache system according to the similarity between the nodes in the Sankey diagram; Obtain data request records and estimated downstream access volume, and determine the business importance, access volume, and response time of the cluster task corresponding to the background data; The background data is split into cold data and hot data based on the business importance, access volume and response time, and the background data is stored according to the split result.

2. The method according to claim 1, characterized in that After the step of splitting the background data into hot and cold data based on the business importance, access volume and response time, and storing the background data according to the splitting result, the method further includes: Determine a sampling ratio of the sampled data amount in the background data, and randomly select data of the sampling ratio from the background data; Try writing the data of the sampling ratio to the sampling cluster, and determine the memory usage of the sampling cluster before and after the data is tried written; Determine the cluster memory difference before and after the data trial write based on the memory usage; Determine the memory occupancy of the background data based on the cluster memory difference and the sampling ratio; The remaining memory of each cluster in the background data cache system is traversed, and cluster allocation is performed based on the remaining memory and the memory occupancy.

3. The method according to claim 1, characterized in that When a new loading task is added downstream, a Sankey diagram is drawn according to each field in the new task and the current cluster task, and the similarity between each node in the Sankey diagram is determined, including: When a new loading task is added downstream, the data source table and upstream source table corresponding to the new task and the background data corresponding to the current cluster task are obtained; Draw a Sankey diagram by using the data source table, the upstream source table and each field in the background data as nodes; The similarity between the nodes in the Sankey diagram is calculated by the SimRank graph similarity algorithm.

4. The method according to claim 3, characterized in that The formula corresponding to the SimRank graph similarity algorithm is: In the formula, R k+1 (α, β) represents the similarity between node α and node β in the Sankey diagram. i (α) represents the child node of node α, I j (β) represents the child nodes of node β, i and j represent the number of child nodes of node α and node β respectively, R k (I i (α),I j (β)) represents child node I i (α) and child node I j The similarity between (β).

5. The method according to claim 1, characterized in that The step of splitting the background data into hot and cold data based on the business importance, access volume and response time, and storing the background data according to the splitting result includes: Performing statistical analysis on the access volume and response time of the cluster task to obtain analysis results; Determining the importance of the cluster task, and classifying the background data corresponding to the cluster task based on the importance of the cluster task to obtain a classification result; According to the analysis result and the classification result, a target splitting strategy is selected from preset cold and hot data splitting strategies; The background data corresponding to the cluster task is split into cold data and hot data based on the target splitting strategy, and the background data is stored according to the splitting result.

6. The method according to claim 5, characterized in that The step of performing statistical analysis on the access volume and response time of the cluster task to obtain the analysis result comprises: Performing statistical analysis on the access volume of the cluster task to determine whether there is a business peak period for the cluster task, and obtaining access volume analysis results; Determine whether the background data corresponding to the cluster task has a response time requirement according to the response time of the cluster task, and obtain a response time analysis result; The access volume analysis result and the response time analysis result are used as analysis results.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Using a timed polling method to detect the current status of each cluster in the background data cache system; When the state of the cluster is detected as faulty for a preset number of consecutive times, updating the cluster state of the cluster to faulty; The corresponding cluster tasks in the cluster are reallocated, and the query service is adaptively switched based on the current state of the cluster.

8. A background data centralized caching device, characterized in that: The device is applied to a background data cache system, and comprises: The data identification module is used to draw a Sankey diagram based on the fields in the newly added task and the current cluster task when a new loading task is added downstream, and to determine the similarity between the nodes in the Sankey diagram; A data updating module, used for updating the background data of the background data cache system according to the similarity between the nodes in the Sankey diagram; A data statistics module is used to obtain data request records and estimated downstream access volume, and determine the business importance, access volume and response time of the cluster task corresponding to the background data; The splitting storage module is used to split the background data into hot and cold data based on the business importance, access volume and response time, and store the background data according to the splitting result.

9. A background data centralized caching device, characterized in that: The device comprises: a memory, a processor, and a background data centralized caching program stored in the memory and executable on the processor, wherein the background data centralized caching program is configured to implement the steps of the background data centralized caching method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a background data centralized caching program, and when the background data centralized caching program is executed by the processor, the steps of the background data centralized caching method according to any one of claims 1 to 7 are implemented.

Images