Data caching method, apparatus, device, medium, and product
By calculating and synchronizing early warning cache values in the cluster, the database read/write performance bottleneck is resolved, enabling efficient data reading and simplified system operation and maintenance, and improving user experience and response efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA CONSTRUCTION BANK
- Filing Date
- 2023-02-27
- Publication Date
- 2026-08-04
AI Technical Summary
In existing technologies, databases such as MySQL have read and write performance bottlenecks in the field of big data, especially when reading large amounts of data in batches, which is inefficient and cannot meet the requirements of high-concurrency read and write and data consistency.
The system obtains early warning tasks from the master node in the cluster, calculates the early warning cache value according to the preset calculation rules, and synchronizes it to the slave nodes. Users can directly read the cache value from the slave nodes to avoid frequent data reads. The system adopts a Redis cluster and tiDB database architecture to optimize read performance.
It improves database read performance, reduces frequent read operations on data tables, simplifies the operation and maintenance costs of big data systems, and optimizes the response efficiency of user requests.
Smart Images
Figure CN116166576B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of big data and artificial intelligence technology, and in particular to a data caching method, apparatus, device, medium and product. Background Technology
[0002] In the field of big data engineering, maintaining high-concurrency data read / write operations while ensuring data consistency is a significant technical challenge. A common approach is to use popular databases such as MySQL, PostgreSQL, and MongoDB as storage engines to handle some Online Transaction Processing (OLTP) tasks, while column-oriented databases like ClickHouse and Iceberg are used for real-time reporting and other Online Analytical Processing (OLAP) tasks. OLAP and OLTP databases maintain consistency by monitoring changes in the OLTP database logs and using a message queue framework. However, this solution suffers from the following problems:
[0003] Read / write performance bottlenecks exist: MySQL-like database applications rely on B+ tree (or similar B+ tree family) storage engines. While this algorithm boasts logarithmic time complexity for point-to-point data reads, it becomes a performance bottleneck when handling large batches of data. This is particularly pronounced in the big data domain.
[0004] In other words, currently, when data is frequently read from the database, a large number of read operations will result in low read performance. Summary of the Invention
[0005] This application provides a data caching method, apparatus, device, medium, and product that can avoid reading data from the data table every time a user makes multiple identical read requests, thereby improving the database read performance.
[0006] In a first aspect, embodiments of this application provide a data caching method, the method comprising:
[0007] The early warning task is obtained through the master node in the cluster. The early warning task is used to calculate the data stored in the data table according to the preset calculation rules to obtain the early warning cache value. The cluster includes master nodes and slave nodes.
[0008] Calculations are performed based on the warning task to update the warning cache value in the warning task;
[0009] The updated warning task is synchronized to the slave node, which is used to receive read requests and return the warning cache value obtained from the warning task of the slave node.
[0010] Secondly, this application provides a data caching device, the device comprising:
[0011] The acquisition module is used to acquire early warning tasks through the master node in the cluster. The early warning tasks are used to calculate the data stored in the data table according to the preset calculation rules to obtain the early warning cache value. The cluster includes master nodes and slave nodes.
[0012] An update module is used to perform calculations based on the warning task to update the warning cache value in the warning task;
[0013] The caching module is used to synchronize the updated warning tasks to the slave nodes. The slave nodes are used to receive read requests and return the warning cache value obtained from the warning tasks of the slave nodes based on the read requests.
[0014] Thirdly, embodiments of this application provide an electronic device, which includes: a processor and a memory storing computer program instructions;
[0015] When the processor executes computer program instructions, it implements the data caching method as described in any of the embodiments of the first aspect.
[0016] Fourthly, embodiments of this application provide a computer storage medium storing computer program instructions, which, when executed by a processor, implement the data caching method as described in any of the embodiments of the first aspect.
[0017] Fifthly, embodiments of this application provide a computer program product in which instructions, when executed by a processor of an electronic device, cause the electronic device to perform a data caching method as described in any of the embodiments of the first aspect above.
[0018] The data caching method, apparatus, device, medium, and product in this application embodiment include: obtaining an early warning task through a master node in a cluster. The early warning task is used to calculate data stored in a data table according to preset calculation rules to obtain an early warning cache value. The cluster includes a master node and slave nodes; performing calculations according to the early warning task to update the early warning cache value in the early warning task; and synchronizing the updated early warning task to the slave nodes. The slave nodes are used to receive read requests and return the early warning cache value obtained from the early warning task of the slave nodes. When a user needs to obtain the early warning cache value, they can send a read request to the slave node, which will return the early warning cache value to the user. Since the early warning cache value is obtained through pre-calculation, multiple identical read requests from the user do not need to be calculated each time, avoiding frequent readings of data in the data table and improving read performance. Attached Figure Description
[0019] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0020] Figure 1 This is a flowchart illustrating a data caching method provided in one embodiment of this application;
[0021] Figure 2-1 This is a schematic diagram of the architecture used in the data caching method provided in one embodiment of this application;
[0022] Figure 2-2 This is a schematic diagram of an early warning task provided in one embodiment of this application;
[0023] Figure 2-3 This is a schematic diagram of an early warning update mechanism provided in one embodiment of this application;
[0024] Figure 3 This is a schematic diagram of the structure of a data caching device provided in an embodiment of this application;
[0025] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation
[0026] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.
[0027] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.
[0028] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the term "comprising" or any other variations thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.
[0029] To address the problems of the prior art, embodiments of this application provide a data caching method, apparatus, device, medium, and product. The data caching method provided in this application embodiment will be described first below.
[0030] Figure 1 A flowchart illustrating a data caching method according to an embodiment of this application is shown. Figure 1 As shown, the method may specifically include the following steps:
[0031] Step 101: Obtain the early warning task through the master node in the cluster. The early warning task is used to calculate the data stored in the data table according to the preset calculation rules to obtain the early warning cache value. The cluster includes master nodes and slave nodes.
[0032] Specifically, the cluster can be a Redis cluster, with the master node being a Redis master node and the slave nodes being Redis slave nodes.
[0033] The data caching method in this application is used to perform calculations on data obtained from the database and store the calculation results, i.e., the early warning cache value, in the cluster nodes. In this way, users can obtain the early warning cache value by sending a query request to the cluster.
[0034] like Figure 2-1As shown, the database in this application can utilize a tiDB cluster. tiDB is an HTAP database that supports both transactional and analytical processing. As the storage database, tiDB primarily handles transactional processing, mainly addressing CRUD operations on relational data and point queries on single data triggers for early warning rules in the engineering construction field. tiFlash is primarily used for report data analysis and calculation of statistical early warning rules. tiDB is compatible with MySQL data table formats, resulting in a smoother migration from the original system. It also boasts excellent disaster recovery plans and elastic scaling advantages. As a cloud-native, highly available financial database, tiDB utilizes the Raft protocol at the underlying level to implement a multi-replica storage, multi-site, multi-disaster recovery plan.
[0035] tiDB uses the open-source database RocksDB as its underlying read / write storage engine. This engine uses an LSM-Tree, and the RocksDB storage architecture maintains data through sequentially written logs, enabling more efficient data writing. Simultaneously, tiDB incorporates a Raft mechanism to ensure data consistency and high availability across multiple machines, and can perform cross-rack and cross-datacenter data version control, avoiding the data consistency control challenges associated with simple master-slave architectures.
[0036] The alert task can be generated when a user requests to operate on the data in the database, such as writing data to the database, updating data, or reading data. The conditions for generating the alert task are not limited in this application.
[0037] The early warning task also includes at least one of the following:
[0038] (1) Database identifiers involved in early warning calculations, such as database names;
[0039] (2) Importance of early warning;
[0040] (3) Warning trigger identifier: No matter what changes are made to the data that may trigger the warning, the warning trigger identifier will not be changed. This identifier will only change when the warning task is updated.
[0041] (4) Whether to modify the identifier of the warning: Modifying the data table involved in the warning task will make the following changes to the warning task: the number of warning recalculations will be increased by 1, and whether to modify the identifier of the warning will be changed to True.
[0042] (5) Identification of the data table involved in the early warning calculation, such as the name of the data table;
[0043] (6) Average calculation time for early warning, for example, the average time taken for multiple calculations of an early warning task;
[0044] (7) Number of times the early warning is recalculated, that is, the number of times the early warning task is calculated;
[0045] (8) The time of the most recent single warning calculation, and the time consumed by the most recent calculation of the warning task;
[0046] (9) Warning update time.
[0047] The preset calculation rules can be set according to the actual situation. For example, it can be used to calculate the total amount in data table 1, or to calculate the average value in data table 2, or to retrieve data from a specific column in data table 3, etc. There are no restrictions here. There can be multiple alert tasks, and each alert task can have different preset calculation rules and operate on different data tables. Each alert task can perform calculations on data from one or more data tables in the database, which is also not limited here.
[0048] After the early warning task is completed, it will obtain an early warning cache value, which can be a single value or multiple values.
[0049] Step 102: Calculate based on the warning task to update the warning cache value in the warning task.
[0050] The master node performs calculations on the early warning task, calculates the data stored in the data table according to the preset calculation rules corresponding to the early warning task, obtains the early warning cache value, and uses the early warning cache value to update the early warning cache value in the early warning task.
[0051] There can be multiple early warning tasks, which form a task list. The master node processes each early warning task in the task list in turn to update the early warning cache value of each early warning task.
[0052] Step 103: Synchronize the updated warning task to the slave node. The slave node is used to receive read requests and return the warning cache value obtained from the warning task of the slave node.
[0053] After updating the alert tasks in the task list, the master node synchronizes the task list to the slave nodes, ensuring that the alert task records are identical on both nodes. When a user needs to retrieve the alert cache value, they can send a read request to the slave node, which will then return the cached value. This reduces the workload on the master node in responding to user requests. Furthermore, since the alert cache value is pre-calculated, multiple identical read requests from the same user do not require repeated calculations to obtain the cached value, avoiding frequent reads from the data table and improving read performance.
[0054] Multiple slave nodes can be set up to read cached data from the slave nodes in a unified manner, which can reduce the caching pressure and read / write pressure of a single node.
[0055] In some embodiments of this application, there are multiple early warning tasks; step 102 involves calculating based on the early warning tasks to update the early warning cache value in the early warning tasks, including:
[0056] Step 1021: Sort the multiple warning tasks according to the preset sorting rules to obtain the sorting result.
[0057] The preset sorting rules include at least one of the following: sorting the importance of the multiple early warning tasks from largest to smallest; and sorting the average calculation time of the multiple early warning tasks from smallest to largest.
[0058] For example, the preset sorting rule could be to sort multiple alert tasks from most important to least important, allowing higher-important alert tasks to be calculated earlier, so users can obtain the alert cache value as soon as possible, improving the response efficiency of user requests. Alternatively, it could be to sort multiple alert tasks from least important to most important, processing as many alert tasks as possible to obtain more alert cache values, further improving the response efficiency of user requests. Another option is to first sort multiple alert tasks from most important to least important, and then, for alert tasks of the same importance, sort them by their average alert calculation time from least important to most important.
[0059] Step 1021 specifically includes:
[0060] The first ranking result is obtained by sorting the multiple early warning tasks from highest to lowest importance.
[0061] If the first warning task and the second warning task have the same importance in the first sorting result, and the warning whether the identifier of the first warning task was triggered in the previous calculation cycle, while the warning whether the identifier of the second warning task was not triggered in the previous calculation cycle, then the first warning task is ranked before the second warning task to obtain the second sorting result.
[0062] The second sorting result is obtained by sorting the average calculation time of the multiple early warning tasks from smallest to largest.
[0063] In each calculation cycle, each alert task can be processed at least once to obtain the alert cache value. The previous calculation cycle refers to the calculation cycle preceding the current one. When alert tasks have the same importance, if an alert task's alert modification flag was triggered in the previous calculation cycle, the modified alert task is moved to the top of the list of alert tasks of the same level and sorted in ascending order of average alert calculation time. This measure aims to address situations where personnel information input errors or frequent changes to the same data occur simultaneously in business scenarios.
[0064] Step 1022: The master node is triggered to calculate the multiple warning tasks in the sorting result in the order of arrangement, so as to update the warning cache value in each warning task.
[0065] Specifically, the task list includes multiple alert tasks, such as Figure 2-3 As shown, the task list update mechanism consists of two phases: Phase 1 is to calculate the warning tasks, and Phase 2 is to reorder the task list on the Redis master node.
[0066] In Phase 1, the cursor pointing to the current task prepares to move. If the current alert status indicator is set to True, the current alert is calculated, and the average alert calculation time and the time of the most recent single alert calculation are modified according to the alert task execution time. It's important to note that during Phase 1, there's a possibility that the database involved in the alert might be modified during calculation. Therefore, the alert modification time should be updated before alert calculation. This ensures that even if the alert is modified during calculation, the modified data will only affect the calculation of that alert in the next time period, not the current period's alert calculation. After obtaining the calculation result, the calculated value is compared with the alert trigger threshold, and the alert trigger status indicator is changed accordingly. After the master node completes the calculation, it needs to update the slave nodes. If the alert has never been triggered or has never reached the triggered state, the current alert status is changed through the downstream backend API to notify the client. After all this is completed, the cursor moves to the next task.
[0067] In Phase 2, once all alerts in the currently sorted task list have been calculated (when the cursor is set to the number of alerts), the task list is reordered according to a preset sorting rule, and alert task calculations are performed in the next time period. The calculation time for the most pessimistic alert is recalculated and updated. The next time period refers to the next calculation period.
[0068] After Phase 2 is completed, the task cursor will be reset to 0, and the calculation will start from the beginning.
[0069] In one embodiment of this application, the step of calculating and updating the warning cache value in the warning task based on the warning task includes:
[0070] If the data stored in the data table changes, or if the calculation cycle ends, a new cache value is obtained by calculating according to the warning task.
[0071] The warning cache value in the warning task is updated using the new cache value.
[0072] Specifically, when the data stored in the data table changes, or when the calculation cycle ends, the master node performs calculations based on the early warning task to obtain a new cache value, and uses the new cache value to update the early warning cache value in the early warning task, so as to ensure that when the user requests data, the latest cache value can be returned, thereby improving the accuracy of the response.
[0073] The data caching method provided in this application is illustrated with the following examples.
[0074] Early warning initialization mechanism
[0075] The alert initialization process involves loading alert metadata into the Redis master node cache. This includes loading the database names involved in the alert, recording the alert importance, table names, alert trigger identifier, average alert calculation time, time of the most recent alert calculation, number of alert recalculations, and whether the alert has been modified. The Redis master node then writes and updates the alert task queue, while slave nodes are responsible for reading the specific alert tasks.
[0076] To avoid additional system CPU and I / O load caused by repeated data changes, the following rules are used to constrain the warnings, and an independent thread is used for asynchronous computation so that it does not block the normal system I / O process and affect the user experience.
[0077] We set a certain time as the calculation time for the most pessimistic early warning scenario, denoted as T below. Assume there are n early warning tasks that need to be calculated, and the time for each early warning task is t. i and satisfy The n warnings are strategically distributed over a time period T using time slicing. (tiFlash, as an analytical transaction engine, maintains an internal copy of the data independent of the storage database tiDB. This mechanism ensures that data processing has virtually no performance impact on point-to-point data reads.)
[0078] The Redis master node maintains a list of alert tasks, employing an importance-based sorting algorithm. Each time an alert task is scanned, the task list is re-sorted. Alerts are calculated in ascending order of average computation time to ensure timely processing. Building upon this, when alerts are of equal importance, if an alert's modification flag was triggered in the previous calculation cycle, the modified alert is moved to the top of the list of alerts of the same level and then sorted in ascending order of computation time. This measure aims to address potential issues in business scenarios, such as incorrect input of personnel information or frequent modifications to the same data within the same timeframe.
[0079] Early warning update mechanism
[0080] No matter what modifications are made to data that might trigger an alert, the alert trigger status identifier will not be changed; this identifier only changes when the alert list is updated. However, modifications to the relevant data tables involved in the alert will change the alert data in Redis as follows: the alert recalculation count will be incremented by 1, and the alert modification status identifier will be changed to True. These tasks will be synchronized from the Redis master node to the Redis slave nodes.
[0081] The alert list update mechanism consists of two phases: Phase 1 is to calculate the alerts, and Phase 2 is to reorder the alert task list on the Redis master node.
[0082] In Phase 1, the cursor pointing to the current task prepares to move. If the current alert modification identifier is set to True, the current alert is calculated, and the average alert calculation time and the time of the most recent single alert calculation are modified according to the alert task execution time. It's important to note that during Phase 1, there's a possibility that the database involved in the alert might be modified again during calculation. Therefore, the alert modification time should be updated before alert calculation. This way, even if the alert is modified during calculation, the modified data will only affect the calculation of the alert in the next time period, not the current period's alert calculation. After obtaining the calculation result, the calculated value is compared with the alert trigger threshold, and the alert trigger identifier is changed accordingly. After the master node completes the calculation, it needs to update the slave nodes. If the alert has never been triggered or has never reached the triggered state, the current alert status is changed through the downstream backend API to notify the client. After all this is completed, the cursor moves to the next task.
[0083] In Phase 2, once all alerts in the currently sorted task list have been calculated (when the cursor represents the number of alerts), the current alert list is reordered according to the algorithm in 2.2.3, and alert task calculations are performed in the next time period. The calculation time for the most pessimistic alert is recalculated and updated.
[0084] After Phase 2 is completed, the task cursor will be reset to 0, and the calculation will start from the beginning.
[0085] Early warning reading mechanism
[0086] When reading data related to alerts, the Redis master node is not accessed directly; instead, the Redis slave node is accessed. The change identifier and alert update time are observed. If the difference between the current time and the alert update time is less than twice the most pessimistic alert calculation time, it means the current alert status is valid, and the cached value can be directly read from the corresponding slave node without recalculation.
[0087] In the most optimistic scenario, the alarm update reading time is: for an alarm numbered i, only t_i is needed to complete the reading.
[0088] In the most pessimistic scenario, the alert update reading time is the time T required to recalculate all alerts. However, in most cases, only a few alerts are modified within each time period. The expected update time is the sum of the computation time for common alert tasks, which is within an acceptable efficiency range in the engineering field. This will not significantly impact the real-time performance of alerts while reducing the computational data components and resources that need to be deployed for operations and maintenance.
[0089] The data caching method provided in this application has the following beneficial effects:
[0090] The adoption of the new big data architecture has significantly simplified the operation and maintenance costs of manually maintaining multiple big data middleware, including database migration, backup, and disaster recovery solutions.
[0091] Redis database was used as the early warning caching technology. An additional layer of caching optimization was added to the tiDB and tiFlash architecture, which can avoid performing expensive SQL column calculations as much as possible without changing the data, thus saving costs significantly.
[0092] By prioritizing alert tasks based on their importance, more important alerts are processed first, significantly optimizing the user experience. Even in the most pessimistic scenario (where all alerts need to be run again within the time period), the greatest risks are exposed as quickly as possible. However, in most cases, not every task is triggered within each time period; in reality, important alerts are updated much faster than the pessimistic calculation time. In the most optimistic scenario, this scheme updates alerts at a frequency comparable to continuously updating alerts using column-based calculations.
[0093] Figure 3 A schematic diagram of a data caching device provided in one embodiment of this application is shown. For ease of explanation, only the parts related to the embodiment of this application are shown.
[0094] Reference Figure 3 The data caching device 300 may include:
[0095] The acquisition module 301 is used to acquire an early warning task through the master node in the cluster. The early warning task is used to calculate the data stored in the data table according to the preset calculation rules to obtain the early warning cache value. The cluster includes a master node and slave nodes.
[0096] The update module 302 is used to calculate based on the warning task to update the warning cache value in the warning task;
[0097] The caching module 303 is used to synchronize the updated warning task to the slave node. The slave node is used to receive read requests and return the warning cache value obtained from the warning task of the slave node.
[0098] Optionally, there are multiple early warning tasks, and the update module 302 includes:
[0099] The first acquisition submodule is used to sort the multiple warning tasks according to a preset sorting rule and obtain a sorting result. The preset sorting rule includes at least one of the following: sorting the multiple warning tasks from the importance level to the minimum; sorting the multiple warning tasks from the average calculation time of the warning tasks from the minimum to the maximum.
[0100] The calculation submodule is used to trigger the master node to perform calculations on multiple early warning tasks in the sorting result in the order of arrangement, so as to update the early warning cache value in each early warning task.
[0101] Optionally, the first acquisition submodule includes:
[0102] The first sorting unit is used to sort the multiple early warning tasks from largest to smallest according to their importance to obtain a first sorting result;
[0103] The second sorting unit is used to sort the first early warning task before the second early warning task if the first early warning task and the second early warning task have the same importance in the first sorting result, and the early warning whether-modification identifier of the first early warning task was triggered in the previous calculation cycle, while the early warning whether-modification identifier of the second early warning task was not triggered in the previous calculation cycle, so as to obtain the second sorting result.
[0104] The third sorting unit is used to sort the second sorting result from smallest to largest according to the average calculation time of the warnings of the multiple warning tasks, and obtain the sorting result.
[0105] Optionally, the update module 302 includes:
[0106] The second acquisition submodule is used to calculate and obtain a new cache value according to the early warning task when the data stored in the data table changes or when the calculation cycle ends.
[0107] The update submodule is used to update the early warning cache value in the early warning task using the new cache value.
[0108] Optionally, the early warning task may further include at least one of the following: the database identifier involved in the early warning calculation, the degree of importance of the early warning, the identifier for whether the early warning is triggered, the identifier for whether the early warning is modified, the identifier of the data table involved in the early warning calculation, the average calculation time of the early warning, the number of times the early warning is recalculated, the time of the most recent single early warning calculation, and the early warning update time.
[0109] Optionally, the master node is a write-only Redis master node, and the slave node is a read-only Redis slave node.
[0110] The data caching device 300 provided in this application embodiment can implement the various processes implemented in the foregoing method embodiments, and will not be described again here to avoid repetition.
[0111] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0112] Figure 4 A schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application is shown.
[0113] The device may include a processor 401 and a memory 402 storing program instructions.
[0114] When processor 401 executes the program, it implements the steps in any of the above method embodiments.
[0115] For example, the program can be divided into one or more modules / units, one or more of which are stored in memory 402 and executed by processor 401 to complete this application. The one or more modules / units can be a series of program instruction segments capable of performing a specific function, which describe the execution process of the program in the device.
[0116] Specifically, the processor 401 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.
[0117] Memory 402 may include mass storage for data or instructions. For example, and not limitingly, memory 402 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 402 may include removable or non-removable (or fixed) media. Where appropriate, memory 402 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 402 is non-volatile solid-state memory.
[0118] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the methods according to one aspect of this disclosure.
[0119] The processor 401 implements any of the methods described above by reading and executing program instructions stored in the memory 402.
[0120] In one example, the electronic device may also include a communication interface 403 and a bus 410. The processor 401, memory 402, and communication interface 403 are connected via the bus 410 and communicate with each other.
[0121] The communication interface 403 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.
[0122] Bus 410 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 410 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.
[0123] Furthermore, in conjunction with the methods in the above embodiments, this application embodiment can provide a storage medium for implementation. This storage medium stores program instructions; when these program instructions are executed by a processor, they implement any of the methods in the above embodiments.
[0124] This application also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.
[0125] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.
[0126] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above method embodiments and achieve the same technical effects. To avoid repetition, it will not be described again here.
[0127] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.
[0128] The functional modules shown in the above block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on machine-readable media or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable media" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer grids such as the Internet, intranets, etc.
[0129] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.
[0130] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to create a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.
[0131] The above are merely specific embodiments of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.
Claims
1. A data caching method, characterized in that, The method includes: The early warning task is obtained through the master node in the cluster. The early warning task is used to calculate the data stored in the data table according to the preset calculation rules to obtain the early warning cache value. The cluster includes master nodes and slave nodes. Calculations are performed based on the warning task to update the warning cache value in the warning task; The updated warning task is synchronized to the slave node, which is used to receive read requests and return the warning cache value obtained from the warning task of the slave node; There are multiple early warning tasks; the calculation based on the early warning tasks to update the early warning cache value in the early warning tasks includes: The multiple early warning tasks are sorted according to a preset sorting rule to obtain a sorting result. The preset sorting rule includes at least one of the following: the multiple early warning tasks are sorted from the most important to the least important; the multiple early warning tasks are sorted from the least important to the longest average calculation time. The master node is triggered to calculate the multiple warning tasks in the sorting result in the order of arrangement, so as to update the warning cache value in each warning task; The early warning task also includes at least one of the following: the database identifier involved in the early warning calculation, the degree of importance of the early warning, the identifier for whether the early warning is triggered, the identifier for whether the early warning is modified, the identifier of the data table involved in the early warning calculation, the average calculation time of the early warning, the number of times the early warning is recalculated, the time of the most recent single early warning calculation, and the early warning update time.
2. The method according to claim 1, characterized in that, The step of sorting multiple early warning tasks according to a preset sorting rule to obtain a sorting result includes: The first ranking result is obtained by sorting the multiple early warning tasks from highest to lowest importance. If the first warning task and the second warning task have the same importance in the first sorting result, and the warning whether the identifier of the first warning task was triggered in the previous calculation cycle, while the warning whether the identifier of the second warning task was not triggered in the previous calculation cycle, then the first warning task is ranked before the second warning task to obtain the second sorting result. The second sorting result is obtained by sorting the average calculation time of the multiple early warning tasks from smallest to largest.
3. The method according to claim 1, characterized in that, The step of calculating and updating the early warning cache value in the early warning task based on the early warning task includes: If the data stored in the data table changes, or if the calculation cycle ends, a new cache value is obtained by calculating according to the warning task. The warning cache value in the warning task is updated using the new cache value.
4. The method according to any one of claims 1-3, characterized in that, The master node is a write-only Redis master node, and the slave node is a read-only Redis slave node.
5. A data caching device, characterized in that, The device includes: The acquisition module is used to acquire early warning tasks through the master node in the cluster. The early warning tasks are used to calculate the data stored in the data table according to the preset calculation rules to obtain the early warning cache value. The cluster includes master nodes and slave nodes. An update module is used to perform calculations based on the warning task to update the warning cache value in the warning task; A caching module is used to synchronize the updated warning tasks to the slave nodes. The slave nodes are used to receive read requests and return the warning cache value obtained from the warning tasks of the slave nodes based on the read requests. There are multiple early warning tasks; the update module includes: The first acquisition submodule is used to sort the multiple warning tasks according to a preset sorting rule and obtain a sorting result. The preset sorting rule includes at least one of the following: sorting the multiple warning tasks from the importance level to the minimum; sorting the multiple warning tasks from the average calculation time of the warning tasks from the minimum to the maximum. The calculation submodule is used to trigger the master node to perform calculations on multiple early warning tasks in the sorting result in the order of arrangement, so as to update the early warning cache value in each early warning task; The early warning task also includes at least one of the following: the database identifier involved in the early warning calculation, the degree of importance of the early warning, the identifier for whether the early warning is triggered, the identifier for whether the early warning is modified, the identifier of the data table involved in the early warning calculation, the average calculation time of the early warning, the number of times the early warning is recalculated, the time of the most recent single early warning calculation, and the early warning update time.
6. An electronic device, characterized in that, The device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the data caching method as described in any one of claims 1-4.
7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processor, implement the data caching method as described in any one of claims 1-4.
8. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device performs the data caching method as described in any one of claims 1-4.