Consistent query method and device for big data query system

Abstract

The application provides a consistency query method and device of a big data query system, the method comprising: acquiring first data and second data that need to be synchronously stored, the first data being data generated before a batch data date, and the second data being data generated after the batch data date; synchronously storing the first data into batch tables of a first database system and a second database system; synchronously storing the second data into real-time tables of the first database system and the second database system; determining a routing query request based on a data time range in a user data query request and the batch data date; and querying the first data and the second data from the batch tables and / or the real-time tables in the first database system and / or the second database system based on the routing query request.

Description

Technical Field

[0001] This application relates to the field of financial data query technology, and in particular to a consistent query method and apparatus for a big data query system. Background Technology

[0002] In today's financial industry, the querying and storage of massive amounts of data are crucial for business operations. Financial institutions, due to the large volume of business data and the need to retain historical data for extended periods to meet regulatory traceability requirements, are increasingly demanding big data technologies. However, due to the complexity of financial operations, a single database product often cannot meet diverse business needs, necessitating the combination of the advantages of different database products to build an efficient query system.

[0003] In big data query components, HBase is often used for massive detailed queries due to its high concurrency capabilities, but it does not support aggregation operations; while Elasticsearch (ES) excels at aggregation and statistical functions. In financial business query scenarios, it is often necessary to simultaneously involve aggregated statistical data (from ES) and detailed business data (from HBase), which leads to data consistency issues between HBase and ES. This poses a severe challenge for financial businesses that require high consistency and continuity.

[0004] Currently, there are two main existing technical solutions to address the data consistency issue between HBase and Elasticsearch: one is data synchronization based on HBase's Write-Ahead Log (WAL) mechanism. This method involves writing a WAL log parsing service to periodically read HBase's WAL log files, parse the write operations, and synchronize the changed data to Elasticsearch. The other is asynchronous data synchronization based on Kafka. This method introduces Kafka, and after data is successfully written to HBase, it asynchronously pushes the data to a Kafka queue, waiting for Elasticsearch to consume it. Consumption efficiency can be improved by specifying the number of consumption threads.

[0005] However, the first method mentioned above is difficult to maintain and control, has problems with real-time monitoring, is inefficient at consuming WAL change information in Elasticsearch, and suffers from data write latency. The second method, on the other hand, struggles to guarantee atomicity because HBase lacks batch transactions, while write operations are typically performed in large batches. Furthermore, end-to-end monitoring of writes in Kafka and Elasticsearch is complex, and the write-back logic is also quite intricate if a write operation fails at any stage. Summary of the Invention

[0006] In view of this, this application provides a consistent query method and apparatus for a big data query system to solve at least one of the problems mentioned above.

[0007] To achieve the above objectives, this application adopts the following approach: According to a first aspect of this application, a consistent query method for a big data query system is provided, the method comprising: Obtain the first and second data that need to be synchronized and stored. The first data is data generated before the batch data date, and the second data is data generated after the batch data date. The first data is synchronously stored in a batch table of the first database system and the second database system; The second data is synchronously stored in the real-time tables of the first database system and the second database system; The routing query request is determined based on the data time range in the user data query request and the date of the batch data; Based on the routing query request, the first data and the second data are queried from the batch table and / or the real-time table in the first database system and / or the second database system.

[0008] In some embodiments of this application, the method described above synchronously stores the second data in real-time tables of the first database system and the second database system, including: The second data is retrieved from upstream Kafka by a pull thread, and the second data is written to the first database system and the second database system by a save thread; When a write operation fails in the first database system or the second database system, the failed second data is written to the corresponding NAS file system, and the failed second data is retrieved from the NAS file system and then written to the first database system or the second database system.

[0009] In some embodiments of this application, when the above method fails to write to the first database system or the second database system, the failed second data is written to the corresponding NAS file system, including: When a write operation fails in the first database system or the second database system, the second data that failed to be written is written to the corresponding NAS file system after a preset number of retries.

[0010] In some embodiments of this application, the above method further includes: After the second data that failed to be written is successfully written to the first database or the second database, the corresponding second data that failed to be written is deleted from the NAS file system.

[0011] In some embodiments of this application, the above method further includes: Set a batch data date for the first database system and the second database system. The batch data date is used to identify the boundary between batch data and real-time data.

[0012] In some embodiments of this application, the above method further includes: merging the first data obtained from the batch table and the second data obtained from the real-time table.

[0013] In some embodiments of this application, the first database system in the above method is HBase, and the second database system is Elasticsearch.

[0014] According to a second aspect of this application, a consistent query apparatus for a big data query system is provided, comprising: The data acquisition unit is used to acquire first data and second data that need to be stored synchronously. The first data is data generated before the batch data date, and the second data is data generated after the batch data date. The first storage unit is used to synchronously store the first data into a batch table of the first database system and the second database system; The second storage unit is used to synchronously store the second data in the real-time tables of the first database system and the second database system; The query route determination unit is used to determine the route query request based on the data time range in the user data query request and the date of the batch data; The data query unit is used to query the first data and the second data from the batch table and / or the real-time table in the first database system and / or the second database system based on the routing query request.

[0015] According to a third aspect of this application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0016] According to a fourth aspect of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described method.

[0017] According to a fifth aspect of this application, a computer program product is provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0018] As can be seen from the above technical solutions, the consistency query method and apparatus of the big data query system provided in this application effectively manages batch data and real-time data by distinguishing data into batch tables and real-time tables. Furthermore, by simultaneously writing real-time data to both the first and second database systems, and combining normal write processing with abnormal write and retry procedures, a complete data write and fault tolerance mechanism is constructed. Specifically, in the event of a write failure, the data is written to the NAS file system and retried, ensuring that the data is ultimately successfully written to both database systems. This effectively solves the data consistency problem between different big data components such as HBase and Elasticsearch, meeting the high requirements of financial businesses for data consistency, continuity, and real-time performance. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 This is a flowchart illustrating a consistent query method for a big data query system provided in an embodiment of this application; Figure 2 This is a flowchart illustrating the process of synchronously storing second data into a database system, as provided in an embodiment of this application. Figure 3 This is the synchronization process for real-time data between HBase and ES provided in the embodiments of this application; Figure 4 This is a schematic diagram of the structure of a consistency query device for a big data query system provided in an embodiment of this application; Figure 5 This is a schematic block diagram of the system configuration of the electronic device provided in the embodiments of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the embodiments of this application will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments and descriptions of this application are used to explain this application, but are not intended to limit this application.

[0021] The information collected in the technical solution of this application is information and data authorized by the user or fully authorized by all parties. The collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data all comply with the relevant laws, regulations and standards of the relevant countries and regions, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding operation portals are provided for users to choose to authorize or refuse.

[0022] The acquisition, transmission, storage, use, and processing of data in this application all comply with the relevant provisions of national laws and regulations.

[0023] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0024] like Figure 1 The diagram shown is a flowchart illustrating a consistent query method for a big data query system provided in this application. The method includes the following steps: Step S101: Obtain the first data and the second data that need to be synchronized and stored. The first data is data generated before the batch data date, and the second data is data generated after the batch data date.

[0025] During the data synchronization and storage phase, the big data query system collects and identifies the data to be stored from data sources (such as various business systems, sensors, log files, etc.). This data is divided into first data and second data based on the time of its generation. First data specifically refers to all historical data generated before a preset batch data date (e.g., midnight each day). Second data specifically refers to data newly generated on the current day, after the batch data date.

[0026] This step involves the initial classification and differentiation of massive amounts of data, laying the foundation for storing different types of data on different storage media (batch tables and real-time tables in subsequent steps) to optimize storage structure and improve query efficiency.

[0027] Step S102: Synchronously store the first data in the batch tables of the first database system and the second database system.

[0028] The first data, after being classified in step S101, will be synchronously stored by the system in batch tables in two independent database systems. Here, the first and second database systems are two independently running database instances, which may employ different database technologies (in one embodiment of this application, the first database system is HBase and the second database system is Elasticsearch), or simply be two replicas for high availability and disaster recovery. The batch tables here are table structures specifically designed to store large amounts of historical data.

[0029] Synchronous storage in this step means that when the first data is written, it is simultaneously written to batch tables in both database systems in parallel. This mechanism ensures redundant data backup, improves data availability and system fault tolerance, and prevents data loss or service interruption due to single points of failure.

[0030] Step S103: Synchronously store the second data in the real-time tables of the first database system and the second database system.

[0031] Similarly, the second data identified in step S101 will be synchronously stored by the system in real-time tables in the first and second database systems. These real-time tables are table structures designed for storing and processing data that is updated in real-time and requires fast write and read operations.

[0032] Similar to step S102, this step requires ensuring that real-time data can be written to the real-time tables of the two database systems simultaneously and in parallel, which is crucial for real-time data processing that requires high timeliness and high consistency.

[0033] Step S104: Determine the route of the query request based on the data time range in the user data query request and the date of the batch data.

[0034] In this embodiment, after receiving a user's data query request, the system does not directly perform data retrieval, but analyzes the user's query request. The core is to determine the specific route query request based on the data time range specified in the query request and the preset batch data dates in the system.

[0035] Specifically, if the user's requested query time range is entirely before the batch data date, the system will identify that the query primarily targets historical data (first data). If the user's requested query time range only includes data after the batch data date, the system will identify that the query primarily targets real-time data (second data). If the user's requested query time range spans both before and after the batch data date, the system will generate a routed request that needs to query both historical and real-time data simultaneously.

[0036] Step S105: Based on the routing query request, query the first data and the second data from the batch table and / or the real-time table in the first database system and / or the second database system.

[0037] After determining the routing query request in step S104, this step executes the actual data retrieval operation based on this request. Specifically, if the routing query request indicates the need to retrieve historical data (first data), the data query module will initiate a query request to pre-configured batch tables in the first and / or second database systems. Batch tables are specifically used to store historical data prior to the current day; querying these tables allows for efficient retrieval of the historical information required by the user. If the routing query request indicates the need to retrieve real-time data (second data), the data query module will initiate a query request to pre-configured real-time tables in the first and / or second database systems. Real-time tables are specifically used to store newly generated data that requires immediate processing; querying these tables allows for rapid retrieval of the latest data. If the routing query request indicates the need to retrieve both historical and real-time data simultaneously, the data query module will coordinate the two database systems to query their respective batch and real-time tables, and finally integrate the query results and return them to the user.

[0038] As can be seen from the above technical solution, the consistent query method of the big data query system provided in this application achieves effective management of batch data and real-time data by distinguishing data into batch tables and real-time tables. Furthermore, by simultaneously writing real-time data to both the first and second database systems, and combining normal write processing with abnormal write and retry procedures, a complete data writing and fault tolerance mechanism is constructed. Specifically, in the event of a write failure, the data is written to the NAS file system and retried, ensuring that the data is ultimately successfully written to both database systems. This effectively solves the data consistency problem between different big data components such as HBase and Elasticsearch, meeting the high requirements of financial business for data consistency, continuity, and real-time performance.

[0039] In some embodiments of this application, such as Figure 2 As shown, step S103 above, which involves synchronously storing the second data in the real-time tables of the first and second database systems, includes: Step S1031: The second data is pulled from the upstream Kafka by the pull thread, and the second data is written into the first database system and the second database system by the save thread.

[0040] In this embodiment, to ensure stable and orderly data writing, separate normal write queues can be maintained for the first and second database systems. In practice, by setting up a pull thread, the second data to be written can be continuously pulled from the upstream Kafka message queue. Subsequently, a save thread is responsible for writing this pulled second data into the corresponding first and second database systems. This process ensures that, under normal operating conditions, data can be smoothly and efficiently transferred from the data source and stored in the target database.

[0041] Step S1032: When writing to the first database system or the second database system fails, write the failed second data to the corresponding NAS file system, and pull the failed second data from the NAS file system, and continue to write the failed second data to the first database system or the second database system.

[0042] In this embodiment, this step is mainly used to handle possible exceptions during the data writing process and to retry. When a failure occurs while writing second data to the first or second database system, this failed second data can be temporarily written to the corresponding NAS file system. Subsequently, this failed second data is retrieved from the NAS file system and an attempt is made to write it to the first or second database system again. Through this mechanism, this application aims to ensure that even if a failure occurs during the writing process, the data will not be lost and can ultimately be successfully written to the target database system, thereby guaranteeing the reliability and eventual consistency of the data writing.

[0043] In some embodiments of this application, step S1032 above, when the first database system or the second database system fails to write, writes the second data that failed to write to the corresponding NAS file system, includes: when the first database system or the second database system fails to write, after retrying a preset number of times, writing the second data that failed to write to the corresponding NAS file system.

[0044] When writing second data (real-time data) to the first database system (e.g., HBase) or the second database system (e.g., ES) fails, this embodiment will initiate a retry mechanism, and after reaching a preset number of retries, the data that still failed to be written will be transferred to the NAS file system.

[0045] The specific process is as follows: During the normal write operation to HBase or ES, if a write operation failure is detected (e.g., HBase cluster failure, ES cluster failure, or network problems), this embodiment will immediately intervene and first attempt to retry the failed write operation. The number of retries is a preset number. This is to handle temporary network fluctuations or instantaneous system failures, striving to restore write operations in a short time and reduce the overhead of data dumping.

[0046] If, after a preset number of retries, the data still cannot be successfully written to HBase or Elasticsearch, this embodiment will determine that the write operation is a persistent failure. In this case, the failed data will be written to the corresponding NAS file system. The NAS file system, as an independent and persistent storage medium, is used to temporarily store the data that failed to be written to the database, preventing data loss.

[0047] To facilitate management and subsequent retries, data written to the NAS file system follows specific naming rules. For example: Path naming: NAS path / write AP node name / write task number. This helps identify which application node failed in which write task.

[0048] File naming: Task ID_hbase_timestamp.json (for data that failed to be written to HBase) or Task ID_es_timestamp.json (for data that failed to be written to Elasticsearch). The file type is usually JSON, which is convenient for storing structured data, and the timestamp is used to uniquely identify and sort the file.

[0049] In this embodiment, dumping the data to the NAS file system is not the end goal, but rather a preparation for subsequent write retries. This embodiment also includes a process of pulling these failed data from the NAS file system and attempting to write them to HBase and Elasticsearch again. Once the data is successfully written, the corresponding NAS file is deleted, ultimately ensuring that the data in the NAS file has been successfully processed, thereby guaranteeing eventual data consistency.

[0050] Through this mechanism, this application effectively solves various faults that may occur during real-time data writing, and maximizes the eventual consistency of data between HBase and ES, which is crucial for the high requirements of financial business for data accuracy and continuity.

[0051] The retry mechanism in the above embodiments is a simple fixed number of retries. This means that the system will attempt the same number of retries regardless of whether it is a temporary network jitter or a persistent database failure. For example, a momentary network connection interruption and a complete database service crash are two completely different types of errors. A fixed number of retries cannot differentiate between these two types of errors. For the latter, multiple retries may only unnecessarily consume system resources and increase the pressure on NAS file dumping. In the case of database overload or failure, continuous, high-frequency retries may exacerbate the burden and even delay recovery time.

[0052] This embodiment can adjust the retry count strategy based on the error type. Specifically, for transient errors, which are usually temporary, such as network timeouts, connection interruptions, or database connection pool exhaustion, exponential backoff retries are used because the service is likely to recover within a short time. A reasonable retry limit and maximum backoff time can be set.

[0053] For permanent errors, these indicate deeper problems with the service or data, such as incorrect data format, insufficient permissions, missing database tables, or a complete service outage that cannot be recovered in the short term. Excessive retries are meaningless for these types of errors. Therefore, when a specific error code or exception is captured, it can be identified as a permanent error. The number of retries should be reduced, or retries should be abandoned immediately. The data should be dumped to the NAS file system, and an alarm mechanism should be triggered for manual intervention or a more advanced level of automated repair.

[0054] By introducing this intelligent retry strategy, this embodiment can transform passive error handling into proactive and adaptive error handling, further enhancing the data consistency, stability, and resilience of the financial big data query system in complex and ever-changing environments.

[0055] In some embodiments of this application, the method further includes: after the second data that failed to write is successfully written to the first database or the second database, deleting the corresponding second data that failed to write in the NAS file.

[0056] This effectively prevents the NAS file system from accumulating processed data indefinitely, freeing up storage space. In short, the NAS file system acts as a waiting area. Once data is successfully transferred from this waiting area to the target database, it is removed from the NAS, marking the completion of the data processing flow.

[0057] In some embodiments of this application, the above method further includes: setting a batch data date for the first database system and the second database system, wherein the batch data date is used to identify the boundary between batch data and real-time data.

[0058] In this embodiment, in addition to the basic data partitioning function, it also has the function of setting the batch data date (batch_dt). This batch data date is a key timestamp or date identifier, and its main function is to clearly distinguish the boundary between batch data (i.e., the first data) and real-time data (i.e., the second data) in the big data query system.

[0059] Specifically, batch_dt is used to mark the boundary between batch data and real-time data. For example, batch_dt can be set to day T-1. Data before and after batch_dt will be considered batch data and stored in the batch table; while data after batch_dt, up to day T, will be considered real-time data and stored in the real-time table.

[0060] In this embodiment, batch_dt is stored in a persistent medium such as MySQL, and to improve access efficiency, a copy is also kept in a caching system such as Redis. This way, batch_dt can be retrieved quickly when Redis is available, and can be retrieved from MySQL when Redis is unavailable, ensuring the availability and reliability of batch_dt.

[0061] Setting the batch data date is crucial for query operations. When a business performs a data query, the big data query system routes the query request based on batch_dt. Specifically, query requests for data before batch_dt (inclusive) will be routed to the batch table for querying; while query requests for data between batch_dt and day T will be routed to the real-time table for querying.

[0062] As can be seen, by setting batch data dates, the system can manage and query data from different time dimensions more flexibly and efficiently, especially in scenarios where historical and real-time data need to be processed simultaneously, which greatly optimizes data management strategies.

[0063] As explained above regarding batch_dt, in actual operation and maintenance, batch_dt involves updates. These updates are essentially daily routine tasks. The big data query system is responsible for triggering the batch_dt update process at fixed times each day (e.g., during off-peak hours, such as early morning). The triggering condition for batch_dt updates can be set to the successful completion of all batch processing tasks that depend on data from day T-1. For example, the big data query system only issues the batch_dt update command after the batch data in HBase and ES has been successfully loaded and reached a consistent state.

[0064] In this embodiment, to ensure that the read of batch_dt does not result in intermediate states or inconsistencies during the update process, the update operation of batch_dt needs to be atomic. Specifically, when updating batch_dt stored in MySQL, database transactions can be used to ensure the atomicity of the update. That is, either batch_dt is successfully updated, or it is rolled back to the state before the update. For batch_dt in Redis, this embodiment can adopt a strategy of updating the database first and then deleting the cache, or updating the cache asynchronously. For example, after the MySQL update is successful, the batch_dt cache entry in Redis is immediately deleted, forcing subsequent read requests to retrieve the latest value from MySQL and reload it into the cache.

[0065] In other embodiments of this application, for scenarios with extreme requirements of zero downtime or high availability, it is possible to maintain two batch_dts (current_batch_dt and next_batch_dt). During the update process, the query system can first query the current_batch_dt, and then switch the current_batch_dt to the next_batch_dt after the next_batch_dt is fully ready and has passed verification. However, this increases the complexity of the system.

[0066] In other embodiments of this application, to ensure that the update process does not affect business queries, the principle of reading the old and writing the new should be followed at the moment the batch_dt is updated. That is, before the batch_dt update is complete, all query requests still use the old batch_dt to route data. Once the batch_dt update is successful and published (e.g., the Redis cache is updated), new query requests will immediately use the new batch_dt. Since the batch_dt update is atomic, there will be no situation where queries read a partially updated batch_dt.

[0067] In addition, since the batch table stores data prior to batch_dt, updating batch_dt only changes the range of real-time data. The logic for querying historical data that has been archived into the batch table will not be affected.

[0068] In this application's scheme, real-time data writing involves writing the second data of the current day into both the first and second database systems in parallel. This writing process itself does not directly depend on the value of `batch_dt`, but rather on the concept of "current day". Therefore, updates to `batch_dt` primarily affect query routing, with minimal impact on the real-time data writing process. The second data of the current day will still be written to the real-time table normally. Although writing does not directly depend on `batch_dt`, when `batch_dt` updates occur, it is necessary to ensure that all data for the current day (the second data) has been processed and written to the real-time table, and that data from day T-1 has been fully synchronized to the batch table. The big data query system should ensure that `batch_dt` update operations are coordinated in time with the generation of batch data and the archiving of real-time data (if real-time data from day T-1 needs to be archived to the batch table), avoiding updates to `batch_dt` while data is still being written or archived.

[0069] Through the above mechanism, this embodiment can realize the automatic update of batch_dt, and minimize the impact on business queries and data writing during the update process, ensuring the smooth operation of the system and the consistency of data.

[0070] In some embodiments of this application, the above method further includes: merging the first data obtained from the batch table and the second data obtained from the real-time table.

[0071] In this embodiment, a query request spanning a batch_dt is decomposed into two subqueries targeting a batch table and a real-time table. These two subqueries are executed in parallel or sequentially, each returning a portion of the data. The query result merging unit receives historical query results returned from the batch table (e.g., batch data in HBase) and real-time query results returned from the real-time table (e.g., real-time data in HBase or Elasticsearch).

[0072] Upon receiving these two sets of query results, this embodiment integrates the data according to the original logic of the query request and the user's desired display order. For example, if the user requests data sorted by time, the merging unit will uniformly sort the data obtained from the batch table and the real-time table based on the timestamp. If the query involves aggregation operations, it may be necessary to perform secondary aggregation on intermediate aggregation results from different sources.

[0073] In some cases, due to data synchronization mechanisms or defined boundary conditions, there may be a small amount of duplicate data between the batch table and the real-time table, or some form of data deduplication may be required. This embodiment is capable of handling these situations, ensuring that the final query results presented to the user are accurate, non-duplicate, and meet data consistency requirements.

[0074] Finally, the query system can encapsulate the merged data according to a unified format and structure, and then return it to the upper-layer application or user interface. In this way, users do not need to care about the specific storage system from which the data comes or the boundary point of the batch_dt, but directly obtain a complete and consistent query result.

[0075] By introducing a query result merging operation, this embodiment can seamlessly handle complex queries that span historical and real-time data boundaries, providing users with a transparent, efficient, and unified data query experience. This is particularly important in financial business scenarios where it is often necessary to view historical transactions and the latest developments simultaneously, ensuring the integrity of data queries and the timeliness of business decisions.

[0076] The following uses HBase and Elasticsearch as examples, combined with Figure 3 The following will further explain the real-time data synchronization process described in this embodiment. Figure 3 As can be seen, the synchronization process includes the following steps: Step 1: Maintain the normal HBase write process. The HBase write process is divided into pull and save threads. The pull thread pulls real-time data from upstream Kafka, and the save thread writes the pulled data into HBase. Set the number of records to be pulled and written in each batch (batch_num) and the interval between each pull (batch_tm). Step 2: Maintain the normal write process of Elasticsearch. The Elasticsearch write process is divided into pull and save threads. The pull thread pulls real-time data from upstream Kafka, and the save thread writes the pulled data into Elasticsearch. Set the number of data to be pulled and written in each batch (batch_num) and the interval between each pull (batch_tm). Step 3: Maintain the HBase abnormal write process. When the HBase cluster fails or other reasons cause an error in writing to HBase, split a branch in the normal HBase write process. After retrying the write to HBase a few times and failing, write the failed data to the NAS file system. The NAS file path naming rule is: NAS path / write AP node name / write task number. The NAS file name is task number_hbase_timestamp.json. Step 4: Maintain the ES abnormal write process. When the ES cluster fails or other reasons cause an error in writing to ES, split a branch in the normal ES write process. After retrying ES _num times and failing, write the failed data to the NAS file system. The NAS file path naming rule is: NAS path / writing AP node name / writing task number. The NAS file name is task number_es_timestamp.json. Step 5: In the normal HBase and ES retrieval process in Step 1 and Step 2, a separate branch is split off to retrieve the data that failed to be written from the NAS file and continue writing it into HBase and ES. After the write is successful, the NAS file is deleted. Finally, all data in the NAS file is written to HBase and ES, the data synchronization process is completed, and the eventual consistency of the data is guaranteed.

[0077] like Figure 4 The diagram shown is a structural schematic of a consistency query device for a big data query system provided in an embodiment of this application. The device includes: The data acquisition unit 410 is used to acquire first data and second data that need to be stored synchronously. The first data is data generated before the batch data date, and the second data is data generated after the batch data date. The first storage unit 420 is used to synchronously store the first data into a batch table of the first database system and the second database system; The second storage unit 430 is used to synchronously store the second data into the real-time tables of the first database system and the second database system. The query route determination unit 440 is used to determine the route query request based on the data time range in the user data query request and the date of the batch data; The data query unit 450 is used to query the first data and the second data from the batch table and / or the real-time table in the first database system and / or the second database system based on the routing query request.

[0078] For a detailed description of each of the above units, please refer to the corresponding descriptions in the foregoing method embodiments, which will not be repeated here.

[0079] As described above, the consistency query device of the big data query system provided in this application effectively manages batch and real-time data by dividing the data into batch tables and real-time tables. Furthermore, by simultaneously writing real-time data to both the first and second database systems, and combining normal write processing with exception write and retry procedures, a comprehensive data write and fault tolerance mechanism is constructed. Specifically, in the event of a write failure, the data is written to the NAS file system and retried, ensuring that the data is ultimately successfully written to both database systems. This effectively solves the data consistency problem between different big data components such as HBase and Elasticsearch, meeting the high requirements of financial businesses for data consistency, continuity, and real-time performance.

[0080] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described method.

[0081] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described method.

[0082] This application also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the above-described method.

[0083] like Figure 5 As shown, the electronic device 600 may also include: a communication module 110, an input unit 120, an audio processor 130, a display 160, and a power supply 170. It is worth noting that the electronic device 600 does not necessarily need to include these components. Figure 5 All components shown; in addition, the electronic device 600 may also include Figure 5 For components not shown, please refer to existing technology.

[0084] like Figure 5 As shown, the central processing unit 100, sometimes also referred to as a controller or operating control, may include a microprocessor or other processor device and / or logic device. The central processing unit 100 receives inputs and controls the operation of various components of the electronic device 600.

[0085] The memory 140 may be, for example, one or more of a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. It may store the aforementioned failure-related information, and also store a program for executing that information. The central processing unit 100 may execute the program stored in the memory 140 to perform information storage or processing, etc.

[0086] Input unit 120 provides input to central processing unit 100. Input unit 120 may be, for example, a keypad or touch input device. Power supply 170 provides power to electronic device 600. Display 160 displays images and text. Display may be, for example, an LCD display, but is not limited thereto.

[0087] The memory 140 can be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), a SIM card, etc. It can also be a memory that retains information even when power is off, can be selectively erased, and contains more data; examples of this type of memory are sometimes referred to as ERPOM, etc. The memory 140 can also be some other type of device. The memory 140 includes a buffer memory 141 (sometimes referred to as a buffer). The memory 140 may include an application / function storage unit 142 for storing application programs and function programs or processes for executing the operation of the electronic device 600 via the central processing unit 100.

[0088] The memory 140 may also include a data storage unit 143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 144 of the memory 140 may include various drivers for the electronic device for communication functions and / or for performing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0089] The communication module 110 is a transmitter / receiver that transmits and receives signals via the antenna 111. The communication module 110 (transmitter / receiver) is coupled to the central processing unit 100 to provide input signals and receive output signals, which can be the same as in a conventional mobile communication terminal.

[0090] Based on different communication technologies, multiple communication modules 110 can be configured in the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication module 110 (transmitter / receiver) is also coupled to a speaker 131 and a microphone 132 via an audio processor 130 to provide audio output via the speaker 131 and receive audio input from the microphone 132, thereby enabling typical telecommunications functions. The audio processor 130 may include any suitable buffer, decoder, amplifier, etc. Additionally, the audio processor 130 is coupled to a central processing unit 100, enabling on-device recording via the microphone 132 and on-device playback of stored audio via the speaker 131.

[0091] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0092] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0093] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0094] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0095] This application uses specific embodiments to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A consistent query method for a big data query system, characterized in that, The method includes: Obtain the first and second data that need to be synchronized and stored. The first data is data generated before the batch data date, and the second data is data generated after the batch data date. The first data is synchronously stored in a batch table of the first database system and the second database system; The second data is synchronously stored in the real-time tables of the first database system and the second database system; The routing query request is determined based on the data time range in the user data query request and the date of the batch data; Based on the routing query request, the first data and the second data are queried from the batch table and / or the real-time table in the first database system and / or the second database system.

2. The consistent query method for a big data query system as described in claim 1, characterized in that, The step of synchronously storing the second data into the real-time tables of the first database system and the second database system includes: The second data is retrieved from upstream Kafka by a pull thread, and the second data is written to the first database system and the second database system by a save thread; When a write operation fails in the first database system or the second database system, the failed second data is written to the corresponding NAS file system, and the failed second data is retrieved from the NAS file system and then written to the first database system or the second database system.

3. The consistent query method for a big data query system as described in claim 2, characterized in that, When a write operation fails in either the first or second database system, the failed second data is written to the corresponding NAS file system, including: When a write operation fails in the first database system or the second database system, the second data that failed to be written is written to the corresponding NAS file system after a preset number of retries.

4. The consistent query method for a big data query system as described in claim 3, characterized in that, The method further includes: After the second data that failed to be written is successfully written to the first database or the second database, the corresponding second data that failed to be written is deleted from the NAS file system.

5. The consistent query method for a big data query system as described in claim 1, characterized in that, The method further includes: Set a batch data date for the first database system and the second database system. The batch data date is used to identify the boundary between batch data and real-time data.

6. The consistent query method for a big data query system as described in claim 1, characterized in that, The method further includes merging the first data obtained from the batch table and the second data obtained from the real-time table.

7. The consistent query method for a big data query system as described in claim 1, characterized in that, The first database system is HBase, and the second database system is Elasticsearch.

8. A consistency query device for a big data query system, characterized in that, include: The data acquisition unit is used to acquire first data and second data that need to be stored synchronously. The first data is data generated before the batch data date, and the second data is data generated after the batch data date. The first storage unit is used to synchronously store the first data into a batch table of the first database system and the second database system; The second storage unit is used to synchronously store the second data in the real-time tables of the first database system and the second database system; The query route determination unit is used to determine the route query request based on the data time range in the user data query request and the date of the batch data; The data query unit is used to query the first data and the second data from the batch table and / or the real-time table in the first database system and / or the second database system based on the routing query request.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images