Data hierarchical storage and query processing method and device, equipment and medium

Abstract

The invention relates to the technical field of big data and the field of financial science and technology, and discloses a data hierarchical storage and query processing method, which comprises the following steps of: dividing business data into offline data and real-time data, and respectively storing the offline data and the real-time data in an offline database and a real-time database; real-time data processing tasks are generated through a set time interval, and are distributed to a message queue according to priorities; the task processor obtains the task from the message queue, extracts real-time data and stores the real-time data in a distributed cache; the query request obtains data through the distributed cache, and if the data are not found, the data are searched from the real-time database and a result is returned. Through hierarchical storage, priority management and implementation of a cache mechanism on the data, the direct query pressure on the database is reduced, the response speed of the system in a high-concurrency scene is improved, and the real-time performance and stability of data query are ensured.

Description

Technical Field

[0001] The present invention relates to the fields of big data technology and financial technology, and in particular to a data hierarchical storage and query processing method, device, equipment and storage medium. Background Art

[0002] With the accelerated advancement of the digital transformation of the financial industry, the processing and analysis of real-time data has become an important basis for the business decision-making of major financial institutions. Especially in the life insurance industry, the data-driven management model has made managers at all levels put forward higher requirements for the real-time, accuracy and stability of business data. However, in the existing technology, the real-time processing of data faces many bottlenecks and shortcomings.

[0003] Existing technical solutions usually rely on pushing data groups to the database, and then initiating query requests from the front end. This method can meet the needs when the amount of data is small, but with the surge in data volume, especially when financial institutions generate more than millions of real-time data every 30 seconds, the database will face huge query pressure. The direct query method of the database in the existing system is very likely to cause performance bottlenecks when dealing with high concurrent access, affecting the overall stability and response speed of the system. Therefore, the traditional data query mode is difficult to support the second-level update requirements of large-scale financial data.

[0004] The current data management system of financial institutions often adopts a unified push method when pushing data, lacking effective management of data priority. For example, faced with aggregated data at different levels such as the entire system, war zone, and secondary institutions, the existing system is difficult to achieve data priority processing according to the needs of different levels. This lack of hierarchical and priority management cannot ensure the real-time requirements of high-level decision makers for aggregated data, thus affecting the management decision-making efficiency of financial institutions.

[0005] The frequent push of large-scale real-time data has put increasing pressure on financial databases, especially on offline databases that store a large amount of historical data. Existing technical solutions lack effective load balancing and data diversion mechanisms in the real-time data push process, resulting in offline data and real-time data sharing the same database resources, increasing the risk of offline data being over-accessed or even lost. This situation is particularly prominent in the financial field where data security requirements are high. Summary of the invention

[0006] The main purpose of the present invention is to provide a data hierarchical storage and query processing method, device, equipment and storage medium, aiming to solve the technical problem that the existing technology cannot effectively process real-time data in a large-scale high-concurrency environment, resulting in system performance bottlenecks and insufficient data priority management, which affects data query efficiency.

[0007] To achieve the above object, the present invention provides a data hierarchical storage and query processing method, comprising:

[0008] Divide business data into offline data and real-time data;

[0009] The offline data is stored in an offline database, and the real-time data is stored in a real-time database;

[0010] Generate real-time data processing tasks according to a predetermined time interval, and distribute the real-time data processing tasks to a message queue according to the priority order of the real-time data processing tasks;

[0011] Obtaining the real-time data processing task from the message queue according to the priority order through the task processor, extracting corresponding real-time data from the real-time database according to the real-time data processing task, and storing the extracted real-time data in the distributed cache;

[0012] Obtaining a query request, and searching for target real-time data from the distributed cache based on the query request;

[0013] If the target real-time data is found in the distributed cache, the query result is returned;

[0014] If the target real-time data is not found in the distributed cache, the target real-time data is searched from the real-time database based on the query request, and the query result is returned.

[0015] Furthermore, to achieve the above-mentioned purpose, the present invention provides a data hierarchical storage and query processing device, comprising:

[0016] Data classification module, used to divide business data into offline data and real-time data;

[0017] A data storage module, used to store the offline data in an offline database and store the real-time data in a real-time database;

[0018] A task scheduling module is used to generate real-time data processing tasks according to a predetermined time interval, and distribute the real-time data processing tasks to a message queue according to the priority order of the real-time data processing tasks;

[0019] A task processing module, used for obtaining real-time data processing tasks from the message queue according to the priority order through a task processor, extracting corresponding real-time data from the real-time database according to the real-time data processing tasks, and storing the extracted real-time data in a distributed cache;

[0020] A query processing module, used for obtaining a query request and searching for target real-time data from the distributed cache based on the query request;

[0021] A cache query module, used to return the query result if the target real-time data is found in the distributed cache;

[0022] The database query module is used to search for the target real-time data from the real-time database based on the query request if the target real-time data is not found in the distributed cache, and return the query result.

[0023] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and a data hierarchical storage and query processing program stored in the memory and executable on the processor, and the data hierarchical storage and query processing program, when executed by the processor, implements the steps of the data hierarchical storage and query processing method as described above.

[0024] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium, on which a data hierarchical storage and query processing program is stored, and when the data hierarchical storage and query processing program is executed by a processor, the steps of the data hierarchical storage and query processing method described above are implemented.

[0025] Beneficial effects: The present invention relates to the fields of big data technology and financial technology, and discloses a data hierarchical storage and query processing method, which divides business data into offline data and real-time data, and stores them in an offline database and a real-time database respectively; generates real-time data processing tasks by setting time intervals, and distributes them to message queues according to priorities; the task processor obtains tasks from the message queue, extracts real-time data and stores them in a distributed cache; the query request obtains data through the distributed cache, and if not found, searches and returns the results from the real-time database. The present invention reduces the direct query pressure on the database through hierarchical storage of data, priority management, and implementation of a cache mechanism, improves the response speed of the system in high-concurrency scenarios, and ensures the real-time and stability of data queries. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0027] Figure 1 A schematic diagram of an application environment of a data hierarchical storage and query processing method in an embodiment of the present invention;

[0028] Figure 2 A schematic diagram of a process flow of an embodiment of a data hierarchical storage and query processing method of the present invention;

[0029] Figure 3 It is a functional module diagram of a preferred embodiment of the data hierarchical storage and query processing device of the present invention;

[0030] Figure 4 A schematic diagram of the structure of a computer device in one embodiment of the present invention;

[0031] Figure 5 FIG. 4 is another schematic diagram of the structure of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0032] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0033] The data hierarchical storage and query processing method provided by the embodiment of the present invention can be applied in Figure 1 In an application environment, the user end communicates with the server end through the network. The server end can divide the business data into offline data and real-time data through the user end, and store them in the offline database and the real-time database respectively; generate real-time data processing tasks by setting time intervals, and distribute them to the message queue according to priority; the task processor obtains tasks from the message queue, extracts real-time data and stores them in the distributed cache; the query request obtains data through the distributed cache, and if it is not found, it searches from the real-time database and returns the result. The present invention reduces the direct query pressure on the database through hierarchical storage, priority management and implementation of a cache mechanism for data, improves the response speed of the system in high-concurrency scenarios, and ensures the real-time and stability of data queries. Among them, the user end can be but not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server end can be implemented with an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0034] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of a data hierarchical storage and query processing method provided by the present invention. It should be noted that although a logical sequence is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0035] like Figure 2 As shown, the data hierarchical storage and query processing method proposed by the present invention includes the following steps:

[0036] S10, dividing the business data into offline data and real-time data;

[0037] In this embodiment, business data is divided into offline data and real-time data according to business needs and data characteristics. Offline data usually refers to data that does not need to be updated frequently or processed in real time, such as historical records, summary statistics, etc.; real-time data refers to data that needs to be updated frequently and needs to provide responses in a short time, such as real-time transaction data, dynamic inventory information, etc.

[0038] Business data includes all data generated during business operations, decision support, sales or production. The characteristics of this data may include generation frequency, usage frequency and timeliness requirements.

[0039] Offline data is updated less frequently and is usually used in scenarios such as analysis, archiving, and historical query, with low timeliness requirements. Real-time data needs to be updated frequently or respond immediately, and is usually used for monitoring, real-time analysis, and real-time decision support, with high timeliness requirements.

[0040] Design a data classification rule. The rule defines which data needs to be processed in real time and which data can be stored as offline data based on data generation frequency, timeliness requirements, usage scenarios, and other criteria. These rules can be implemented by setting predefined thresholds. For example, data with a generation frequency exceeding a certain threshold (such as updating once a minute) and with high timeliness requirements is classified as real-time data, and the rest is classified as offline data.

[0041] The system automatically processes the collected business data according to the pre-set data classification rules. When the data enters the system, it is divided into real-time data and offline data through the automated data pipeline. The rule-based system logic automatically allocates the data to the appropriate storage and processing path according to the data attributes and business needs.

[0042] As the business changes, the system can continuously monitor data usage and generation, and dynamically adjust classification rules when necessary to ensure that the accuracy of data division matches business needs.

[0043] Example: In the financial field, the division of business data is of great significance. For example, real-time transaction data (such as stock trading and transfer operations) needs to be processed in real time in order to make timely decisions in the ever-changing financial market, while historical transaction records, monthly reports and other data can be archived and stored as offline data. This division can not only improve the efficiency of real-time data processing, but also effectively save system resources and ensure the stable operation of core businesses. In this way, financial institutions can quickly respond to customer needs while maintaining the integrity and security of a large amount of historical data in the background, and realize the efficient management and utilization of data resources.

[0044] By dividing business data into offline data and real-time data, system resources can be reasonably configured according to different data characteristics to improve data processing efficiency. Real-time data processing can respond to business needs in the shortest time, while the classified storage of offline data reduces the system load pressure, reduces unnecessary resource consumption, and improves the overall performance and stability of the system.

[0045] S20, storing the offline data in an offline database, and storing the real-time data in a real-time database;

[0046] In this embodiment, offline data and real-time data are stored in different types of databases according to the data classification results. Offline databases are used to store data that does not need to be frequently accessed and updated, usually including historical records, archived data, etc.; real-time databases are specifically used to process real-time data that requires immediate response, ensuring that the system can quickly access and process it.

[0047] Offline data can be data with a low update frequency that is separated from business data. This data can be processed and archived in batches during off-peak hours. Offline databases are usually databases designed for low-frequency queries and batch processing, and may include traditional relational databases (such as PostgreSQL, MySQL), or data lakes or other large-scale data storage solutions.

[0048] Real-time data can be data that needs to be updated frequently and responded to immediately, which is separated from business data. This data has high timeliness requirements, and the system must ensure fast access. Real-time databases are databases designed for high-frequency data updates and fast reading, such as NoSQL databases (such as Redis, MongoDB) or memory databases designed for real-time data, which can provide low-latency, high-throughput query capabilities.

[0049] Configure independent storage space for offline data. Offline databases can use traditional relational databases or distributed storage systems. In order to save storage space and improve query efficiency, it is usually necessary to design a table structure suitable for batch processing and use data compression technology to reduce storage costs. Import the classified offline data into the offline database in batches through the data pipeline. This process can be performed regularly by scheduling batch tasks, such as batch updates once a day or a week to ensure data integrity and consistency. The index and backup mechanism of the offline database can be configured to ensure that historical data can be quickly retrieved when needed. At the same time, archive or back up data regularly to reduce the storage pressure of the database.

[0050] Real-time databases usually require high-performance storage systems that can support large-scale concurrent access. You can choose a database that supports in-memory computing, such as Redis, Memcached, or other high-availability and high-scalability NoSQL databases. Real-time data needs to be written to the database instantly. Real-time data can be streamed into the database through message queues and data processing tasks to ensure that the system can still maintain efficient writing capabilities in a high-concurrency environment. Real-time databases need to support frequent query and update operations, so it is usually necessary to configure indexes to speed up queries and ensure the effective storage time of data in memory to avoid expiration or invalidation.

[0051] Example: In the financial sector, financial institutions have a huge amount of business data, such as transaction data, customer behavior data, and market data. Storing low-frequency data such as historical transaction records and reports in an offline database can save a lot of storage resources and simplify database query operations. Content that requires immediate response, such as real-time transaction data and market dynamics data, is stored in a high-performance real-time database. Through this hierarchical storage method, financial institutions can effectively manage massive historical data while maintaining efficient real-time processing capabilities, ensuring the continuity of key businesses and data security.

[0052] By storing offline data and real-time data in an offline database and a real-time database respectively, the efficiency of data storage and query response speed can be improved. The offline database is suitable for batch processing and storage of historical data, which reduces the burden of real-time data processing; the real-time database provides higher performance support for high-frequency queries and updates, ensuring the stability of data access in a high-concurrency environment. This hierarchical storage strategy effectively reduces database pressure and improves the efficiency of data processing and response.

[0053] S30, generating a real-time data processing task according to a predetermined time interval, and distributing the real-time data processing task to a message queue according to the priority order of the real-time data processing task;

[0054] In this embodiment, the processing tasks of real-time data are scheduled and managed. The system generates processing tasks regularly at predetermined time intervals. Each processing task contains specific information about processing certain real-time data (such as data extraction, calculation and other operations). At the same time, the system sets a priority order for each task and distributes it to a message queue. A message queue is a first-in, first-out data structure used to store and manage tasks to be processed.

[0055] According to the rules set by the system, real-time data processing tasks are generated regularly. The time interval can be configured according to business needs, such as every minute, every hour, etc. Real-time data processing tasks contain specific real-time data processing instructions, which may include extracting data, performing calculations or storage, etc. Each task is directly related to data processing. Tasks are assigned different priorities based on their importance. High-priority tasks will be processed first to ensure that critical data is processed in a timely manner. Tasks are assigned to message queues in order of priority. The message queue ensures that tasks are processed in order and adjusts the processing order based on priority.

[0056] System administrators or business demand analysts set a suitable time interval based on the frequency of real-time data generation and processing. For example, a processing task is generated every minute to ensure the continuous update of real-time data. The system regularly executes the task generation logic through a timed trigger (such as a Cron Job or scheduler). When the scheduled time arrives, the system automatically generates one or more real-time data processing tasks. Each task defines information such as the type of data to be processed and the processing method.

[0057] The system evaluates the priority of each task using predefined rules. Priority can be set based on the timeliness of the data, the business value of the data, or the urgency of the task. For example, the system sets a high priority for important real-time data tasks involving business decisions, while secondary data analysis tasks are set to medium or low priority. After the tasks are generated, the system classifies them according to the priority order and assigns high-priority tasks to the queue first. This process ensures that the most important tasks are processed quickly.

[0058] The system will distribute tasks to message queues in order of priority. Each message queue can correspond to different priorities to ensure that high-priority tasks are processed first. Message queues can be implemented using message middleware such as Kafka and RabbitMQ to ensure the concurrency and sequentiality of task processing. When tasks are distributed to message queues, the system will monitor the processing status of the queues to ensure that tasks in the message queues are processed in order of priority. This process uses the first-in-first-out logic to ensure that tasks are executed in order.

[0059] Example: In the financial field, priority scheduling of real-time data processing tasks is crucial. Assume that in a financial institution, real-time transaction data and account change data have a higher priority. The system generates corresponding tasks every minute to process these high-priority data in real time and store them in a real-time database. At the same time, low-priority statistical tasks (such as hourly report generation tasks) can be postponed to ensure that core tasks related to transactions are processed first. Through message queue management, financial institutions can ensure the real-time and stability of key business data processing and storage, thereby better supporting decision-making and customer service.

[0060] By generating real-time data processing tasks at predetermined time intervals and distributing tasks to message queues in order of priority, the system can effectively manage the processing flow of real-time data. High-priority tasks can be processed first to ensure the timeliness of key business data. At the same time, task distribution is managed through message queues to ensure orderly execution and parallel processing of tasks. This improves the system's scheduling capabilities in high-concurrency scenarios and ensures the processing efficiency and stability of real-time data.

[0061] S40, obtaining a real-time data processing task from the message queue according to the priority order through a task processor, extracting corresponding real-time data from the real-time database according to the real-time data processing task, and storing the extracted real-time data in a distributed cache;

[0062] In this embodiment, the task processor obtains tasks from the message queue in order of priority, then extracts corresponding real-time data from the real-time database according to the task requirements, and stores the extracted data in the distributed cache to ensure that the system can quickly respond to high-frequency query requirements.

[0063] The task processor is a component used to obtain tasks from the message queue and perform related operations. It can extract tasks from the message queue in order according to the priority order of the tasks. The tasks in the message queue are arranged according to the set priority. The task processor prioritizes important tasks according to the priority order to ensure that high-priority tasks receive timely responses. The real-time database is a database that stores real-time data. The task processor extracts data from the real-time database according to the task requirements, supporting high-frequency access and fast reading.

[0064] Distributed caching refers to storing cached data on multiple servers or nodes to improve the scalability, performance, and fault tolerance of the system. It ensures fast data access during large-scale concurrent requests and maintains high availability when some nodes fail by sharding or replicating data to multiple nodes. Key features of distributed caching:

[0065] Data Sharding: Divide the cached data into different fragments according to certain rules and distribute them to multiple nodes to reduce the load of a single node and improve the concurrent processing capability of the system.

[0066] Data replication: Some distributed caches replicate data to multiple nodes to ensure that when a node fails, other nodes can still provide the same data, thereby improving the reliability and availability of the system.

[0067] Scalability: By adding cache nodes, the system can easily scale to handle more requests and larger data sets without changing the existing system architecture.

[0068] High availability: Even if some nodes fail, the distributed cache can still continue to serve through other available nodes to ensure the stability of the system.

[0069] For example, in a system that uses Redis as a distributed cache, multiple Redis nodes form a cluster, and user requests are automatically routed to the node containing the requested data. If a node goes down, the system automatically obtains data from other nodes to maintain high availability and stability of the service.

[0070] The task processor obtains the tasks arranged in priority order from the queue through the interface with the message queue. The message queue system pushes tasks to the task processor in the order of first-in-first-out, and high-priority tasks are processed first. The implementation of the task processor is usually based on a scheduling algorithm to ensure that high-priority tasks are executed in a short time. The task processor continuously extracts tasks from the queue and determines the execution order based on the task priority.

[0071] The task processor extracts the corresponding data from the real-time database according to the instructions in the task. The task instructions usually contain data identifiers, query conditions, etc. Real-time databases are usually optimized for high concurrency and support fast queries. The task processor can quickly extract the required data from the database through indexes or other optimization mechanisms. When extracting, the task processor will check the integrity and accuracy of the data to ensure that the data extracted from the database meets the task requirements.

[0072] The task processor writes the data extracted from the real-time database into the distributed cache. Redis is an efficient in-memory database that can quickly respond to frequent read and write requests. The task processor stores the data in Redis as key-value pairs so that subsequent query requests can quickly retrieve data from the cache. To prevent expired or invalid data from occupying cache space for a long time, the task processor sets an expiration time for the data stored in Redis. The system configures the expiration policy according to actual business needs to ensure that the data in the cache is updated in a timely manner.

[0073] Example: In the financial sector, efficient execution of real-time data processing tasks is essential for supporting decision-making and responding to customer needs in real time. Assume that a financial institution's trading system needs to process user transaction requests in real time. The system extracts high-priority tasks (such as large transaction processing tasks) from the message queue every few seconds, extracts relevant transaction data from the real-time database, and writes this data into the distributed cache to ensure that the results can be quickly returned in subsequent queries.

[0074] By obtaining tasks from the message queue according to the priority order by the task processor, the system can ensure that high-priority tasks are processed in a timely manner, improving the real-time performance of data processing. At the same time, storing data extracted from the real-time database in the distributed cache can significantly improve the query response speed, reduce direct access to the real-time database, and reduce system pressure.

[0075] S50, obtaining a query request, and searching for target real-time data from the distributed cache based on the query request;

[0076] In this embodiment, the system receives a query request from a user or system, and searches for target data from a distributed cache based on the request. A distributed cache is an efficient in-memory database designed to handle frequent read and write operations. By prioritizing querying data in the cache, the frequency of access to the main database can be significantly reduced, improving query response speed.

[0077] The query request received by the system usually contains the identification information of the target data, such as the data ID, time range or other query parameters related to the data. The system first tries to find the target data from the distributed cache. Redis is an in-memory database that provides fast query and write capabilities and is suitable for high-concurrency data access scenarios. The specific data to be found is defined in the query request. This data has been stored in the distributed cache in advance to quickly respond to query requests.

[0078] When a user or system initiates a query request, the system receives the request through an API or service interface. The query request is usually sent to the system in the form of an HTTP request or message, containing the identifier of the target data or the query conditions (such as timestamp, ID, etc.). The system parses the parameters in the query request, including the identifier of the target data, the query time range, etc., for further processing.

[0079] The system uses the parsed query parameters to initiate a search request to the distributed cache. The system uses the unique identifier of the target data (such as the data ID) as the key through the key-value pair structure of Redis to find the corresponding real-time data. The distributed cache is stored in memory, so the data search process is very fast. The system can quickly return the target data in the cache through a simple key search operation. If the target data that matches the query parameters exists in the distributed cache, the data is returned (cache hit); if no matching data is found, it is called a cache miss, and further search from other data sources is required.

[0080] By first searching for target data from the distributed cache, the system can effectively reduce the query pressure on the main database and improve the response speed of query requests. The fast search capability of the distributed cache enables the system to maintain good performance in a high-concurrency environment, especially in frequently accessed scenarios, significantly reducing query latency.

[0081] S60, if the target real-time data is found in the distributed cache, the query result is returned;

[0082] In this embodiment, after the target data is hit through the distributed cache, the found target real-time data is immediately returned to the requester as the query result. In this way, further queries to the database are avoided, and the response speed of the query request and the overall performance of the system are improved. This step is the processing logic for cache hits, that is, if the system finds the required data in the cache, it is directly returned to the query initiator without further obtaining data from the database.

[0083] A distributed cache hit refers to finding matching target data in the cache. The system preferentially searches for data in the distributed cache based on the identifier in the query request. If matching data is found, it is a cache hit. The data found in the cache that meets the query conditions is real-time data stored in the cache in advance by the system. The found target data is packaged into a response format predefined by the system and returned to the request initiator through the return channel of the query request (such as an API or message queue).

[0084] The system initiates a search request to the distributed cache based on the parameters in the query request. The distributed cache searches for the corresponding data based on the identifier of the target data (such as data ID or timestamp). If the target data matching the query request exists in the distributed cache, it is considered a cache hit. No further database query is required at this time. When the distributed cache hits the target data, the system extracts the data, usually reading the data from the cache in the form of key-value pairs. The extracted target real-time data is packaged into a query result that conforms to the system response format. The response format is usually in JSON or XML format, containing the basic information of the query request and the returned data. The system returns the encapsulated query results to the query initiator through an API or message service. This return process is usually a synchronous operation to ensure a fast response to the query request. The query result contains the target data extracted from the cache, ensuring the integrity and accuracy of the data, and directly meeting the needs of the query request.

[0085] By directly returning the query results after hitting the target real-time data in the distributed cache, the system can significantly reduce the load pressure on the database and improve the response speed of query requests. The efficient data search and fast return mechanism provided by the distributed cache can ensure that the system still has fast response capabilities in high-concurrency scenarios, reduce query waiting time, and improve user experience and data query efficiency.

[0086] S70: If the target real-time data is not found in the distributed cache, the target real-time data is searched from the real-time database based on the query request, and the query result is returned.

[0087] In this embodiment, when the system cannot find the target real-time data in the distributed cache, the system will query the target data from the real-time database instead and return the query result to the request initiator. This is a response mechanism when the cache fails or the data has not been stored in the cache, ensuring that the system can still meet the query request and ensure the continuity and reliability of data query.

[0088] A cache miss means that the system fails to find the target real-time data that meets the query conditions in the distributed cache. In this case, the system must further query the real-time database. The system accesses the real-time database based on the parameters in the query request (such as data ID or timestamp) and queries the target data. The real-time database usually stores the latest data that can be updated in real time. The target data extracted from the real-time database is packaged into query results and returned to the request initiator.

[0089] The system first searches for the target real-time data in the distributed cache. If there is no matching data in the distributed cache, the system determines that the cache misses. When the cache misses, the system triggers the database query mechanism and instead obtains the target data from the real-time database. The system constructs a real-time database query request based on the data identifier or other query conditions (such as data ID, time range, etc.) in the query request. Real-time databases usually store the latest business data and support high-frequency access. The system quickly extracts qualified target data from the database through indexing or query optimization technology. During the data extraction process, the system performs integrity verification on the target data obtained from the database to ensure that the data is not damaged or lost.

[0090] The system encapsulates the target data obtained from the real-time database into query results that conform to the predefined response format. Common response formats include JSON, XML, etc., and the content includes basic information of the query request and the returned data. The system returns the query results to the request initiator through an API or message mechanism to ensure that the user can obtain the required real-time data. In order to improve the processing speed of subsequent identical query requests, the system can choose to update the found data to the distributed cache and set the cache expiration time so that subsequent queries can directly obtain data from the cache.

[0091] In the case of a distributed cache miss, the system can query data from the real-time database to ensure that the query request is effectively responded to, avoiding data loss or query failure. This mechanism ensures the reliability of query requests, and the system can still provide stable services even in the case of cache failure. By updating the query results to the distributed cache, the processing efficiency of subsequent queries can also be improved, improving the overall performance of the system.

[0092] The present invention relates to the fields of big data technology and financial technology, and discloses a data hierarchical storage and query processing method, which divides business data into offline data and real-time data, and stores them in an offline database and a real-time database respectively; generates real-time data processing tasks by setting time intervals, and distributes them to message queues according to priorities; the task processor obtains tasks from the message queue, extracts real-time data and stores them in a distributed cache; the query request obtains data through the distributed cache, and if it is not found, it searches and returns the result from the real-time database. The present invention reduces the direct query pressure on the database through hierarchical storage of data, priority management, and implementation of a cache mechanism, improves the response speed of the system in high-concurrency scenarios, and ensures the real-time and stability of data queries.

[0093] In one embodiment, the above S10 includes:

[0094] S101, analyzing the business data to determine the update frequency and response time requirements of the business data;

[0095] S102, classifying business data whose update frequency is higher than a preset frequency threshold and whose response time requirement is lower than a preset time threshold as real-time data;

[0096] S103: Classify the data in the business data except the real-time data as offline data.

[0097] In this embodiment, the system first needs to collect business data generated in all business activities. These data may include transaction data, customer behavior data, system logs, etc., covering all aspects of business operations. The data collection process needs to ensure the integrity and accuracy of the data. The system needs to identify all sources that may generate business data, including databases, log systems, APIs, etc. By connecting to these data sources, the system can collect data continuously and automatically. Since the data comes from different sources, there may be differences in format. The system needs to standardize the format of the data during collection to ensure the uniformity of subsequent analysis and processing. Real-time data can be continuously collected through a streaming data pipeline, and for low-frequency data, a timed batch processing method can be used to regularly obtain data from the data source.

[0098] The system needs to analyze the collected business data to determine the update frequency and response time requirements of the data. The update frequency indicates how quickly the data needs to be updated, while the response time requirement refers to how quickly the data needs to be returned to the user when queried. The system counts the generation or update frequency of each type of business data. For data that is frequently generated or requires high-frequency updates, the system will identify it as high-frequency data. This can be achieved by regularly checking the update timestamps or system logs of the data. Based on business needs, the system evaluates the response time requirements of the data when it is queried. For example, a user's account balance data may require a response in seconds, while the query response time for historical financial statements can be longer. This process requires setting corresponding thresholds in combination with business scenarios and user needs. The system can use data analysis tools or customized algorithms to automatically analyze the update frequency and response time requirements of business data to avoid manual intervention and improve efficiency.

[0099] Based on the analysis results of the previous step, the system classifies business data with an update frequency higher than the preset threshold and a response time requirement lower than the preset threshold as real-time data. Real-time data needs to be updated frequently and respond quickly to query requests, and is usually suitable for high concurrency and low latency scenarios. System administrators or business analysts set thresholds for update frequency and response time. For example, the update frequency threshold can be set to update once per minute, and the response time threshold can be set to 3 seconds. The system automatically filters business data by comparing with these thresholds. Data that meets the high update frequency and low response time requirements is automatically classified as real-time data. The system marks real-time data for subsequent storage, query, and processing. These data will be stored in high-performance real-time databases first, and cached using fast caching technology such as Redis to ensure low-latency access.

[0100] For data that fails to meet the requirements of high update frequency and low response time, that is, business data with an update frequency below the threshold or a long response time requirement, the system classifies it as offline data. Offline data is usually a type of data that is processed in batches and is not queried urgently. After the system has filtered out the real-time data according to the previous step, it automatically marks the remaining data as offline data. The characteristics of this data are usually low update frequency and do not require a fast response. The system labels the offline data and stores this data in a database with low frequency of access. In order to save storage space, the system can apply data compression technology and archiving strategies to ensure the long-term availability of offline data without occupying a lot of resources.

[0101] This embodiment classifies business data according to update frequency and response time requirements, so that the system can reasonably configure storage and computing resources. The processing priority of real-time data is improved, ensuring that high-frequency access data is processed in a low-latency and high-efficiency manner, while offline data is processed in batches to reduce the consumption of system resources. This hierarchical processing mechanism significantly improves the performance and stability of the system, especially in high-concurrency and big data scenarios, effectively avoiding the emergence of system bottlenecks.

[0102] In one embodiment, the above S30 includes:

[0103] S301, setting a generation time interval of a real-time data processing task according to business requirements and the frequency of real-time data generation;

[0104] S302, generating a real-time data processing task regularly according to the generation time interval through a timing trigger;

[0105] S303, assigning a high priority, a medium priority, and a low priority to the real-time data processing task according to the importance of the real-time data processing task;

[0106] S304, configuring independent message queues for real-time data processing tasks of different priorities;

[0107] S305, distributing the real-time data processing tasks to corresponding message queues according to the priority order of the real-time data processing tasks.

[0108] In this embodiment, the system first sets the time interval for generating real-time data processing tasks based on specific business needs and the frequency of real-time data generation. The time interval defines how often the system generates processing tasks to ensure that real-time data can be processed at the set frequency. The system analyzes the frequency of business data generation and determines the speed of data generation. For high-frequency real-time data, the time interval for generating tasks should be shorter, while the task generation interval can be appropriately extended for low-frequency data. By combining business needs, the administrator or system configuration rules will set appropriate time intervals, such as generating tasks every minute, every 5 seconds, or every hour. The settings of these intervals can be configured through the system's scheduling mechanism and adjusted according to real-time changes.

[0109] The system uses timer triggers to ensure that real-time data processing tasks are generated at predetermined time intervals. Timer triggers are important tools for system scheduling tasks, responsible for executing task generation according to set time rules. Timer triggers in the system can be configured through task schedulers such as Cron Job and Quartz. The timer triggers the task generation logic at set time intervals. The timer triggers the task generation module at regular intervals, monitors the changes of business data in real time, and ensures that the processing tasks can be generated on time and submitted to the subsequent processing flow.

[0110] The system assigns different priorities to real-time data processing tasks based on the importance of different tasks. High-priority tasks need to be processed as quickly as possible, while low-priority tasks can be delayed to ensure that system resources prioritize critical tasks. The system evaluates the importance of each task, and the evaluation criteria can include the timeliness of the data, business impact, and user needs. Tasks with high task importance are marked as high priority. The system divides tasks into three priorities: high, medium, and low according to preset standards, and adds priority identifiers to tasks to ensure that tasks are executed in priority order during subsequent processing.

[0111] In order to prevent high-priority tasks from being blocked by low-priority tasks, the system configures independent message queues for tasks of different priorities. This ensures that the processing of high-priority tasks is not affected by low-priority tasks. The system configures independent message queues for each priority level. High-priority tasks use high-priority queues, and medium-priority and low-priority tasks use their own message queues. This independent queue design allows tasks of different priorities to be processed in parallel without interfering with each other. The system can use message queue systems such as Kafka and RabbitMQ to manage tasks by priority through these queues to ensure the order and independence of task processing.

[0112] The system distributes tasks to corresponding message queues according to their priorities. Tasks with high priorities are distributed to high-priority queues first to ensure that these tasks can be processed as soon as possible. After the task is generated, the system will distribute it to the appropriate message queue according to the priority of the task. The task distribution module will detect the priority label of each task and distribute high-priority tasks to the high-priority queue. Through the parallel processing mechanism of the message queue, the system can process multiple tasks with different priorities at the same time, ensuring that high-priority tasks can be completed first, while low-priority tasks are processed when system resources are sufficient.

[0113] This embodiment generates real-time data processing tasks at time intervals and assigns priorities according to the importance of the tasks, thereby ensuring priority processing of critical tasks and improving the response speed and accuracy of the system when processing high-priority tasks. By using an independent message queue mechanism, the system can effectively prevent the priority of tasks from being reversed, ensuring that high-priority tasks are not delayed by low-priority tasks, thereby improving the overall concurrent processing capability and stability of the system.

[0114] In one embodiment, the above S40 includes:

[0115] S401, the task processor sequentially obtains real-time data processing tasks from message queues of different priorities in order of priority, so as to give priority to processing high-priority real-time data processing tasks;

[0116] S402, extracting corresponding real-time data from the real-time database through a task processor according to extraction parameters in the real-time data processing task, wherein the extraction parameters include a data identifier of the real-time data and a generation or update time of the real-time data;

[0117] S403, performing data cleaning and standardization processing on the extracted real-time data;

[0118] S404, storing the processed real-time data in the distributed cache, and setting a unique key value and cache expiration time for the real-time data.

[0119] In this embodiment, the task processor sequentially obtains real-time data processing tasks from message queues of multiple priorities. The system ensures that high-priority tasks are processed first according to the set priority order, and low-priority tasks are processed when system resources are idle. The system is configured with multiple message queues, each corresponding to a task priority. The task processor extracts tasks from the message queue in order from high to low priority to ensure that high-priority tasks are executed first. The task processor will check message queues of different priorities in real time, and tasks in high-priority queues will enter the processing flow first to ensure that critical tasks will not be delayed due to low-priority tasks.

[0120] The task processor retrieves the corresponding data from the real-time database according to the extraction parameters in the task. These extraction parameters include data identifiers (such as the unique ID of the data) and generation or update time, which are used to locate and extract the required data. Each real-time data processing task will contain specific extraction parameters, which usually include data identifiers (such as transaction IDs) and generation or update time. These parameters will help the system accurately locate the data that needs to be extracted from the real-time database. The task processor constructs query statements based on these parameters and initiates query requests to the real-time database. In order to speed up the query, the system will use indexes or other query optimization techniques to ensure efficient data acquisition.

[0121] Real-time data extracted from the database may contain redundant or non-standard data. Therefore, before the data enters the cache, the system needs to clean and standardize the data to ensure the accuracy and consistency of the data. The task processor will clean the extracted real-time data, delete duplicate data, correct format errors, or filter invalid data to ensure the accuracy of the data. The cleaned data will be standardized according to the system's standard format to maintain data consistency in subsequent processing. This may include standardizing date formats, unifying measurement units for numeric fields, etc.

[0122] The processed real-time data will be stored in the distributed cache to speed up subsequent query responses. Each data will have a unique key value to quickly locate the data. At the same time, the system sets an expiration time for the cached data to ensure the validity and timely update of the data in the cache. The task processor writes the cleaned and standardized data into the distributed cache. Redis is an in-memory database based on key-value pairs, so each data item will be assigned a unique key to ensure that the data can be quickly located in subsequent queries. In order to prevent data from being stale or invalid, the system sets a cache expiration time (TTL, Time to Live) for each data stored in Redis. The expiration time can be configured according to business needs to ensure that the data is automatically deleted after expiration, ensuring the efficiency and freshness of the cache.

[0123] This embodiment obtains tasks in priority order through the task processor, ensuring that high-priority real-time data processing tasks can be completed first, thereby improving the system's response speed to key data. After extracting the data, it is cleaned and standardized to ensure the accuracy and consistency of the data. Storing the processed data in the distributed cache and setting the cache expiration time can significantly improve query efficiency and ensure the timeliness of the data in the cache.

[0124] In one embodiment, the above S50 includes:

[0125] S501, receiving and parsing query parameters in a query request, wherein the query parameters include a data identifier of target real-time data, a query range, and a query time;

[0126] S502, searching for target real-time data from the distributed cache using the data identifier in the query request;

[0127] S503, if the target real-time data is found and the target real-time data is valid, determining that the target real-time data is successfully found;

[0128] S504: If the target real-time data is not found or the target real-time data is invalid, confirm that the target real-time data is not found.

[0129] In this embodiment, when processing a query request, the system first receives and parses the parameters in the query request. Usually, the query parameters include a data identifier (such as a unique ID), a query scope (such as a time range, a geographic area), and a query time (i.e., the timestamp of the request or the time window of the query), which help the system to accurately locate the target real-time data required by the user. The system receives the query request initiated by the user through an interface such as an API, a message queue, or an HTTP request. The system parses the query request and extracts key parameters, including data identifier, query scope, query time, etc. These parameters are used for subsequent data search operations to ensure that the query results meet the request requirements.

[0130] After parsing the data identifier in the query request, the system will use the identifier to search for the corresponding target real-time data in the distributed cache. The distributed cache stores frequently accessed real-time data, and the target data can be quickly obtained by using the data identifier for key-value search. The system uses the key-value pair structure of Redis to quickly locate the target data from the cache based on the data identifier (such as transaction ID, user ID, etc.). As an in-memory database, Redis provides low-latency, high-performance search capabilities, so data retrieval operations can be completed in a short time.

[0131] If the target data matching the query request is found in the distributed cache, the system also needs to verify the validity of the data. Validity verification includes checking the cache expiration time of the data, the integrity of the data, etc. The system determines whether the data is valid by querying the metadata of the data (such as the cache expiration time and the data integrity mark). If the data is not expired in the cache and meets the integrity requirements, the data is considered valid. If the data is valid, the system considers that the required target data has been successfully found and is ready to return the data to the initiator of the query request.

[0132] If no data matching the query parameters is found in the distributed cache, or the data is found but has expired or is invalid, the system confirms that the target data has not been found. This means that it is necessary to further search for data from other data sources (such as a real-time database). If there is no matching data in the cache, the system records the result and prepares for further operations (such as querying a real-time database). If the found data has expired or does not meet the integrity requirements, the system marks it as invalid and treats it as if the data was not found. After confirming that the data has not been found, the system can trigger subsequent operation processes, such as obtaining data from a real-time database or returning a "data not found" response.

[0133] By receiving and parsing query requests, the system can accurately obtain the target data identifier and query conditions required by the user, and preferentially search for data from the distributed cache to significantly improve the response speed of the query request. The validity check ensures the timeliness and accuracy of the data and avoids returning expired or invalid data. In the case of a cache miss, the system can flexibly handle the subsequent search process to ensure that the user obtains data that meets the requirements, thereby improving the reliability and performance of the system.

[0134] In one embodiment, the above S60 includes:

[0135] S601, if the target real-time data is found from the distributed cache, the validity of the target real-time data is checked by comparing the current time with the cache expiration time of the target real-time data;

[0136] S602, if the current time does not exceed the cache expiration time, confirming that the target real-time data is valid;

[0137] S603, extracting valid target real-time data, and processing the target real-time data according to the query request;

[0138] S604, encapsulating the processed target real-time data into a response format that meets the query request requirements, and generating a query result;

[0139] S605: Return the query result to the initiator of the query request.

[0140] In this embodiment, when the target real-time data exists in the distributed cache, the system first determines its validity by checking the cache expiration time of the data. The cache expiration time is a timestamp that the system sets in advance for each data stored in Redis. When the expiration time is reached, the data will be considered invalid. The system obtains the current timestamp and compares it with the expiration time of the cached data. The system extracts the metadata information of the target real-time data from Redis, especially the cache expiration time, to ensure that the current time does not exceed the expiration time.

[0141] If the current time has not exceeded the cache expiration time, the system confirms that the target real-time data is still valid. Valid data means that the data can be used directly without re-extracting from the database. Through time comparison, if the current time is within the cache expiration time, the system considers the data valid. At this time, there is no need to obtain the data from other data sources. After confirming that the data is valid, the system directly uses the data and skips further queries to the database to reduce delays.

[0142] Once the target data is confirmed to be valid, the system will extract the data and further process it according to the specific parameters in the query request. The processing methods may include data format conversion, data filtering, aggregation, etc. The system extracts the confirmed valid data from the distributed cache. The system performs specific operations on the extracted data based on the parameters in the query request (such as query time range, data field, etc.). These operations can be data aggregation, filtering, format conversion, etc., to ensure that the returned data meets the requirements of the query request.

[0143] The processed target data needs to be encapsulated in a format that complies with the query request (such as JSON, XML, etc.) to ensure that the data structure and content returned to the requester meet the specifications. The system encapsulates the processed data in a predetermined response format. Usually the query result format is JSON or XML, and the specific format depends on the protocol or requirements of the query request. The system generates the final query result based on the content of the query request, ensuring that the response contains all necessary fields and data.

[0144] The system returns the processed query results to the initiator of the query request through a predetermined interface. The return process is usually completed through an API interface, a message queue, or an HTTP response. The system returns the encapsulated query results to the request initiator through the API or message channel initiated by the query request. In the process of returning the results, the system will ensure the integrity and security of data transmission and ensure that the query results are accurately delivered to the initiator.

[0145] By checking the expiration time of the target data in the distributed cache, the system can ensure that the data returned to the request initiator is always valid, avoiding expired or invalid data from affecting the accuracy of the query results. After the valid data is confirmed, the system performs necessary processing on it and encapsulates it into a format that meets the request requirements, reducing the query pressure on the main database and significantly improving the response speed of the query request. When returning the results, the appropriate response channel or interface is used to ensure that the data can be accurately and quickly returned to the query initiator.

[0146] In one embodiment, the above S20 includes:

[0147] S201, configuring an independent storage space for offline data and allocating storage capacity of the offline database;

[0148] S202, configuring the data table structure of the offline database to support compression operations and batch processing operations of offline data;

[0149] S203, establishing a data index on the data table of the offline database, and importing the classified offline data into the offline database;

[0150] S204, configuring an independent storage space for the real-time data and allocating storage capacity of the real-time database;

[0151] S205, configuring the data table structure of the real-time database to support high-frequency update operations and fast read operations of real-time data;

[0152] S206, establishing a data index on the data table of the real-time database, and importing the classified real-time data into the real-time database.

[0153] In this embodiment, the system first configures a dedicated storage space for offline data to ensure that offline data is managed separately from real-time data. The storage capacity allocated for offline data needs to be configured according to business needs to ensure that the storage space is large enough to support long-term storage and batch processing operations. The system physically or logically allocates independent storage space for offline data to avoid sharing storage resources with real-time data, thereby ensuring long-term preservation and independent management of offline data. According to the growth rate and storage requirements of offline data, the system pre-allocates sufficient storage capacity for the offline database and expands it according to demand.

[0154] In order to optimize storage space and improve data processing efficiency, the offline database needs to support data compression and batch processing. By configuring the data table structure of the offline database, the system can achieve large-scale storage of historical data while ensuring efficient data processing. According to the characteristics of offline data, the system designs a suitable database table structure, including column types, index strategies, and partition designs to ensure support for data compression and large-scale batch processing. The system enables data compression to reduce storage space consumption, especially for historical data that is not frequently accessed. This mechanism can significantly improve storage efficiency.

[0155] Before data is stored, the system needs to create indexes on the database table to improve the efficiency of subsequent queries. Data indexes can speed up query and filter operations on specific fields. After classification, offline data will be imported into the offline database and organized and managed according to the design of the table structure. The system creates indexes on key fields (such as date, ID, etc.) in the offline database table to ensure that subsequent query operations can be completed efficiently. Especially when processing large amounts of historical data, indexes can significantly speed up queries. The system writes the classified offline data into the offline database through batch import to complete data storage and organization.

[0156] Separated from offline data, real-time data requires independent storage space and storage capacity. Real-time data is characterized by high update frequency, so the system configures high-performance storage space for it to ensure fast access and data updates. The system allocates high-performance storage resources for real-time data to ensure that it can withstand high-frequency read and write operations. Independent storage space can prevent real-time data from interfering with other data. The storage capacity of real-time data needs to be dynamically adjusted according to the data generation speed and business needs. The system can support automatic expansion of storage capacity to ensure that storage resources always meet demand.

[0157] Real-time databases require a special table structure design to ensure that they can support high-frequency data updates and fast reading. Due to the high timeliness requirements of real-time data, the table structure needs to be optimized for high concurrency and large data volumes. The system designs the table structure of the real-time database to support fast insert, update, and query operations. Partition tables, index optimization, and caching mechanisms can help improve the concurrent processing capabilities of the database. Real-time data needs to be updated frequently, and when designing the table structure, it is necessary to consider how to reduce update locks and delays. The system can use optimistic locking or timestamp-based concurrency control strategies to ensure efficient data updates.

[0158] In order to improve the query performance of real-time data, the system creates indexes on the key fields of the real-time database table. Indexes can speed up data retrieval, which is especially important when processing large-scale real-time data. Classified real-time data will be imported into the database on demand for subsequent rapid processing and query. The system creates indexes on key fields based on the characteristics of real-time data. Common index fields include timestamps, user IDs, or other business-critical fields to help speed up the query of real-time data. The system imports classified real-time data into the real-time database in batches regularly or as needed. To ensure data consistency, the system can set up data verification and integrity check mechanisms.

[0159] This embodiment separates offline data from real-time data for storage, so that the system can optimize the processing according to the characteristics of different data, ensuring the flexibility and efficiency of data storage. Offline data uses compression and batch processing technology to effectively reduce storage space, while real-time data can meet business needs with high real-time requirements through high-frequency updates and fast reading support. The introduction of indexes further improves data query performance, enabling the system to maintain a good response speed in the case of large amounts of data.

[0160] In one embodiment, a data hierarchical storage and query processing device is provided, and the data hierarchical storage and query processing device corresponds one-to-one to the data hierarchical storage and query processing method in the above embodiment. Figure 3 , Figure 3 This is a functional module diagram of a preferred embodiment of the data hierarchical storage and query processing device of the present invention. Data classification module 10, data storage module 20, task scheduling module 30, task processing module 40, query processing module 50, cache query module 60 and database query module 70. The functional modules are described in detail as follows:

[0161] A data classification module 10, used to classify business data into offline data and real-time data;

[0162] A data storage module 20, used to store the offline data in an offline database and store the real-time data in a real-time database;

[0163] The task scheduling module 30 is used to generate real-time data processing tasks according to a predetermined time interval, and distribute the real-time data processing tasks to a message queue according to the priority order of the real-time data processing tasks;

[0164] The task processing module 40 is used to obtain the real-time data processing task from the message queue according to the priority order through the task processor, extract the corresponding real-time data from the real-time database according to the real-time data processing task, and store the extracted real-time data in the distributed cache;

[0165] A query processing module 50, configured to obtain a query request and search for target real-time data from the distributed cache based on the query request;

[0166] A cache query module 60, configured to return a query result if the target real-time data is found in the distributed cache;

[0167] The database query module 70 is used to search for the target real-time data from the real-time database based on the query request if the target real-time data is not found in the distributed cache, and return the query result.

[0168] In one embodiment, the data classification module 10 is specifically used for:

[0169] Collect all business data generated during business activities;

[0170] Analyze the business data to determine the update frequency and response time requirements of the business data;

[0171] Classify business data with an update frequency higher than a preset frequency threshold and a response time requirement lower than a preset time threshold as real-time data;

[0172] The data in the business data other than the real-time data is classified as offline data.

[0173] In one embodiment, the task scheduling module 30 is specifically used to:

[0174] Set the generation time interval of real-time data processing tasks according to business needs and the frequency of real-time data generation;

[0175] Generate a real-time data processing task regularly according to the generation time interval through a timing trigger;

[0176] According to the importance of the real-time data processing task, assign a high priority, a medium priority and a low priority to the real-time data processing task;

[0177] Configure independent message queues for real-time data processing tasks of different priorities;

[0178] Distribute the real-time data processing tasks to the corresponding message queues according to their priority order.

[0179] In one embodiment, the task processing module 40 is specifically configured to:

[0180] The task processor obtains the real-time data processing tasks from the message queues of different priorities in order of priority, so as to give priority to the real-time data processing tasks with high priority;

[0181] Extracting corresponding real-time data from the real-time database through a task processor according to extraction parameters in the real-time data processing task, wherein the extraction parameters include a data identifier of the real-time data and a generation or update time of the real-time data;

[0182] Perform data cleaning and standardization on the extracted real-time data;

[0183] Store the processed real-time data in the distributed cache and set the unique key value and cache expiration time of the real-time data.

[0184] In one embodiment, the task processing module 40 is specifically configured to:

[0185] Receiving and parsing query parameters in the query request, wherein the query parameters include a data identifier of the target real-time data, a query range, and a query time;

[0186] Searching for target real-time data from the distributed cache using a data identifier in the query request;

[0187] If the target real-time data is found and the target real-time data is valid, it is determined that the target real-time data is successfully found;

[0188] If the target real-time data is not found or the target real-time data is invalid, it is confirmed that the target real-time data is not found.

[0189] In one embodiment, the cache query module 60 is specifically used to:

[0190] If the target real-time data is found in the distributed cache, the validity of the target real-time data is checked by comparing the current time with the cache expiration time of the target real-time data;

[0191] If the current time does not exceed the cache expiration time, confirming that the target real-time data is valid;

[0192] Extracting effective target real-time data, and processing the target real-time data according to the query request;

[0193] Encapsulating the processed target real-time data into a response format that meets the query request requirements, and generating a query result;

[0194] The query result is returned to the initiator of the query request.

[0195] In one embodiment, the data storage module 20 is specifically used for:

[0196] Configuring independent storage space for offline data and allocating storage capacity of the offline database;

[0197] Configuring the data table structure of the offline database to support compression operations and batch processing operations of offline data;

[0198] Establishing a data index on the data table of the offline database, and importing the classified offline data into the offline database;

[0199] Configuring independent storage space for real-time data and allocating storage capacity of the real-time database;

[0200] Configuring the data table structure of the real-time database to support high-frequency update operations and fast read operations of real-time data;

[0201] A data index is established on the data table of the real-time database, and the classified real-time data is imported into the real-time database.

[0202] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal through a network connection. When the computer program is executed by the processor, it realizes the functions or steps of a data hierarchical storage and query processing method on the server side.

[0203] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the user side of a data hierarchical storage and query processing method

[0204] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0205] Divide business data into offline data and real-time data;

[0206] The offline data is stored in an offline database, and the real-time data is stored in a real-time database;

[0207] Generate real-time data processing tasks according to a predetermined time interval, and distribute the real-time data processing tasks to a message queue according to the priority order of the real-time data processing tasks;

[0208] Obtaining the real-time data processing task from the message queue according to the priority order through the task processor, extracting corresponding real-time data from the real-time database according to the real-time data processing task, and storing the extracted real-time data in the distributed cache;

[0209] Obtaining a query request, and searching for target real-time data from the distributed cache based on the query request;

[0210] If the target real-time data is found in the distributed cache, the query result is returned;

[0211] If the target real-time data is not found in the distributed cache, the target real-time data is searched from the real-time database based on the query request, and the query result is returned.

[0212] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0213] Divide business data into offline data and real-time data;

[0214] The offline data is stored in an offline database, and the real-time data is stored in a real-time database;

[0215] Generate real-time data processing tasks according to a predetermined time interval, and distribute the real-time data processing tasks to a message queue according to the priority order of the real-time data processing tasks;

[0216] Obtaining the real-time data processing task from the message queue according to the priority order through the task processor, extracting corresponding real-time data from the real-time database according to the real-time data processing task, and storing the extracted real-time data in the distributed cache;

[0217] Obtaining a query request, and searching for target real-time data from the distributed cache based on the query request;

[0218] If the target real-time data is found in the distributed cache, the query result is returned;

[0219] If the target real-time data is not found in the distributed cache, the target real-time data is searched from the real-time database based on the query request, and the query result is returned.

[0220] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0221] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0222] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0223] It should be noted that if software tools or components other than those of the Company appear in the embodiments of the present application, they are only used for illustration and do not represent actual use. The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above-mentioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A data hierarchical storage and query processing method, characterized in that: The following steps are involved: Divide business data into offline data and real-time data; The offline data is stored in an offline database, and the real-time data is stored in a real-time database; Generate real-time data processing tasks according to a predetermined time interval, and distribute the real-time data processing tasks to a message queue according to the priority order of the real-time data processing tasks; Obtaining the real-time data processing task from the message queue according to the priority order through the task processor, extracting corresponding real-time data from the real-time database according to the real-time data processing task, and storing the extracted real-time data in the distributed cache; Obtaining a query request, and searching for target real-time data from the distributed cache based on the query request; If the target real-time data is found in the distributed cache, the query result is returned; If the target real-time data is not found in the distributed cache, the target real-time data is searched from the real-time database based on the query request, and the query result is returned.

2. The data hierarchical storage and query processing method according to claim 1, characterized in that: Divide business data into offline data and real-time data, including: Collect all business data generated during business activities; Analyze the business data to determine the update frequency and response time requirements of the business data; Classify business data with an update frequency higher than a preset frequency threshold and a response time requirement lower than a preset time threshold as real-time data; The data in the business data other than the real-time data is classified as offline data.

3. The data hierarchical storage and query processing method according to claim 1, characterized in that: Generating a real-time data processing task according to a predetermined time interval, and distributing the real-time data processing task to a message queue according to a priority order of the real-time data processing task, including: Set the generation time interval of real-time data processing tasks according to business needs and the frequency of real-time data generation; Generate a real-time data processing task regularly according to the generation time interval through a timing trigger; According to the importance of the real-time data processing task, assign a high priority, a medium priority and a low priority to the real-time data processing task; Configure independent message queues for real-time data processing tasks of different priorities; Distribute the real-time data processing tasks to the corresponding message queues according to their priority order.

4. The data hierarchical storage and query processing method according to claim 1, characterized in that: Obtaining the real-time data processing task from the message queue according to the priority order through the task processor, extracting corresponding real-time data from the real-time database according to the real-time data processing task, and storing the extracted real-time data in the distributed cache, including: The task processor obtains the real-time data processing tasks from the message queues of different priorities in order of priority, so as to give priority to the real-time data processing tasks with high priority; Extracting corresponding real-time data from the real-time database through a task processor according to extraction parameters in the real-time data processing task, wherein the extraction parameters include a data identifier of the real-time data and a generation or update time of the real-time data; Perform data cleaning and standardization on the extracted real-time data; Store the processed real-time data in the distributed cache and set the unique key value and cache expiration time of the real-time data.

5. The data hierarchical storage and query processing method as claimed in claim 1, characterized in that: Obtaining a query request, and searching for target real-time data from the distributed cache based on the query request, including: Receiving and parsing query parameters in the query request, wherein the query parameters include a data identifier of the target real-time data, a query range, and a query time; Searching for target real-time data from the distributed cache using a data identifier in the query request; If the target real-time data is found and the target real-time data is valid, it is determined that the target real-time data is successfully found; If the target real-time data is not found or the target real-time data is invalid, it is confirmed that the target real-time data is not found.

6. The data hierarchical storage and query processing method according to claim 1, characterized in that: If the target real-time data is found in the distributed cache, the query result is returned, including: If the target real-time data is found in the distributed cache, the validity of the target real-time data is checked by comparing the current time with the cache expiration time of the target real-time data; If the current time does not exceed the cache expiration time, confirming that the target real-time data is valid; Extracting effective target real-time data, and processing the target real-time data according to the query request; Encapsulating the processed target real-time data into a response format that meets the query request requirements, and generating a query result; The query result is returned to the initiator of the query request.

7. The data hierarchical storage and query processing method according to claim 1, characterized in that: The method further comprises: storing the offline data in an offline database and storing the real-time data in a real-time database, comprising: Configuring independent storage space for offline data and allocating storage capacity of the offline database; Configuring the data table structure of the offline database to support compression operations and batch processing operations of offline data; Establishing a data index on the data table of the offline database, and importing the classified offline data into the offline database; Configuring independent storage space for real-time data and allocating storage capacity of the real-time database; Configuring the data table structure of the real-time database to support high-frequency update operations and fast read operations of real-time data; A data index is established on the data table of the real-time database, and the classified real-time data is imported into the real-time database.

8. A data hierarchical storage and query processing device, characterized in that: The data hierarchical storage and query processing device comprises: Data classification module, used to divide business data into offline data and real-time data; A data storage module, used to store the offline data in an offline database and store the real-time data in a real-time database; A task scheduling module is used to generate real-time data processing tasks according to a predetermined time interval, and distribute the real-time data processing tasks to a message queue according to the priority order of the real-time data processing tasks; A task processing module, used for obtaining real-time data processing tasks from the message queue according to the priority order through a task processor, extracting corresponding real-time data from the real-time database according to the real-time data processing tasks, and storing the extracted real-time data in a distributed cache; A query processing module, used for obtaining a query request and searching for target real-time data from the distributed cache based on the query request; A cache query module, used to return the query result if the target real-time data is found from the distributed cache; The database query module is used to search for the target real-time data from the real-time database based on the query request if the target real-time data is not found in the distributed cache, and return the query result.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a data hierarchical storage and query processing program stored in the memory and executable on the processor. When the data hierarchical storage and query processing program is executed by the processor, the steps of the data hierarchical storage and query processing method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The storage medium stores a data hierarchical storage and query processing program, which, when executed by a processor, implements the steps of the data hierarchical storage and query processing method according to any one of claims 1 to 7.

Images