Data query method and device, equipment and storage medium

Abstract

The invention discloses a data query method, device and equipment and a storage medium, which are applied to the field of data query, and the method comprises the following steps: acquiring service table change data of a service table, and updating the service table change data to a wide table and a wide table master-slave table data graph according to an initial updating strategy; constructing a Markov chain decision model, and in a data updating process, determining a target updating strategy of the service table change data based on the Markov chain decision model; and updating the service table change data to the wide table and the wide table master-slave table data graph based on the target updating strategy. The wide table related data is converted into the graph structure by utilizing the data graph technology, efficient query and analysis are performed by utilizing the data graph, the wide table processing computing power is improved, quicker and more accurate wide table data construction is realized, the problems of performance bottleneck and resource consumption of the existing technical scheme can be solved, the Markov chain is introduced, and the wide table data construction efficiency is improved. The future data state can be predicted, and the uncertainty of data change time can be dealt with.

Description

Technical Field

[0001] The present invention relates to the field of data query, and in particular to a data query method, device, equipment and computer-readable storage medium. Background Art

[0002] With the increase in data volume and the complexity of data relationships, large-scale real-time statistical function queries based on analytical databases that associate multiple tables face performance bottlenecks. Since the physical model of the business system follows the three-paradigm design specification and the data redundancy is low, performance is limited when performing association analysis of multiple large tables, consuming a large amount of computing resources and causing performance problems. In addition, since statistical function queries require real-time performance, aggregating associated tables into wide tables for data query is inefficient and cannot meet real-time requirements. In addition, when the amount of data is large, a large amount of computing resources will be consumed. Summary of the invention

[0003] The purpose of the present invention is to provide a data query method, device, equipment and storage medium, which are applied to the field of data query. The method converts wide table related data into a graph structure by utilizing data graph technology, and uses the data graph for efficient query and analysis, thereby improving the wide table processing computing power, achieving faster and more accurate wide table data construction, and being able to solve the performance bottleneck and resource consumption problems of existing technical solutions.

[0004] In order to solve the above technical problems, the present invention provides a data query method, comprising:

[0005] Determine the business table required for data query needs in the business system, determine the wide table master and slave table from the business table, and construct a wide table and wide table master and slave table data map based on the wide table master and slave table;

[0006] When the data in the business table changes, the business table change data of the business table is obtained, and the business table change data is updated to the wide table and the wide table master-slave table data map according to the initial update strategy;

[0007] Constructing a Markov chain decision model, and in the data update process, determining a target update strategy for the business table change data based on the Markov chain decision model;

[0008] Based on the target update strategy, the business table change data is updated to the wide table and the wide table master-slave table data map;

[0009] When a data query request is received, a data query is performed in the wide table based on the data query request to obtain a data query result.

[0010] Optionally, based on the target update strategy, updating the business table change data to the wide table and the wide table master-slave table data map includes:

[0011] When the target state of the business table change data is the addition to the main table, the business table change data is updated to the wide table and the wide table master-slave table data map, the slave table data is pulled from the wide table master-slave table data map based on the business key, and the slave table data is updated to the wide table;

[0012] When the target state of the business table change data is main table update, the business table change data is updated to the wide table;

[0013] When the target state of the business table change data is that the slave table is newly added but not recorded in the master table, or the slave table is updated but not recorded in the master table, the business table change data is updated to the wide table master-slave table data map;

[0014] When the target state of the business table change data is that the slave table is added and the master table has been recorded or the slave table is updated and the master table has been recorded, the business table change data is first updated to the wide table, and then updated to the wide table master-slave table data map.

[0015] Optionally, determining a wide table master-slave table from the business table includes:

[0016] Determine the business table with the largest data volume and the highest query frequency as the wide table master table;

[0017] The remaining business tables that share a business primary key with the wide table master table are determined as wide table slave tables.

[0018] Optionally, the method may further include:

[0019] Obtain a wide table log and a wide table master-slave table data map log, and scan the wide table log and the wide table master-slave table data map log based on a sliding window algorithm;

[0020] When the data synchronization exception log is obtained through scanning, data compensation is performed on the wide table based on the master-slave table data map of the wide table.

[0021] Optionally, scanning the wide table log and the wide table master-slave table data graph log based on a sliding window algorithm includes:

[0022] The left boundary time point of the sliding window is determined as the current system time minus the preset time, and the right boundary time point of the sliding window is determined as the current system time;

[0023] The left boundary time point and the right boundary time point are updated at a preset frequency to complete the sliding scan of the sliding window on the wide table log and the wide table master-slave table data map log.

[0024] Optionally, when data in the business table changes, obtaining business table change data of the business table includes:

[0025] Setting a data probe of an event sensor in a business table of the business system;

[0026] When the data in the business table is detected to have changed through the data probe, initial business table change data of the business table is obtained and uploaded to the server of the event sensor;

[0027] Based on built-in rules, the initial business table change data is classified, filtered and screened to obtain the business table change data, and the business table change data is pushed to the built-in message queue of the server of the event sensor, so that the application end of the event sensor subscribes to the business table change data in the message queue and updates the wide table based on the business table change data.

[0028] Optionally, when data in the business table changes, obtaining business table change data of the business table includes:

[0029] When data in the target field in the business table is changed, the business table change data of the target field in the business table is acquired.

[0030] In order to solve the above technical problems, the present invention provides a data query device, comprising:

[0031] The first module is used to determine the business table required for data query needs in the business system, determine the wide table master and slave table from the business table, and construct a wide table and wide table master and slave table data map based on the wide table master and slave table;

[0032] The second module is used to obtain the business table change data of the business table when the data in the business table changes, and update the business table change data to the wide table and the master-slave table data map of the wide table according to the initial update strategy;

[0033] The third module is used to construct a Markov chain decision model, and during the data update process, determine the target update strategy of the business table change data based on the Markov chain decision model;

[0034] A fourth module is used to update the business table change data to the wide table and the wide table master-slave table data map based on the target update strategy;

[0035] The fifth module is used to perform data query in the wide table based on the data query request when receiving the data query request to obtain the data query result.

[0036] In order to solve the above technical problems, the present invention provides a data query device, comprising:

[0037] Memory, for storing computer programs;

[0038] A processor is used to implement the above-mentioned data query method when executing the computer program.

[0039] In order to solve the above technical problem, the present invention provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the above-mentioned data query method is implemented.

[0040] It can be seen that the method of the present invention determines the business table required for data query needs in the business system, determines the wide table master and slave table from the business table, and constructs the wide table and the wide table master and slave table data map based on the wide table master and slave table; when the data in the business table changes, the business table change data of the business table is obtained, and the business table change data is updated to the wide table and the wide table master and slave table data map according to the initial update strategy; a Markov chain decision model is constructed, and in the data update process, the target update strategy of the business table change data is determined based on the Markov chain decision model; based on the target update strategy, the business table change data is updated to the wide table and the wide table master and slave table data map; when a data query request is received, data query is performed in the wide table based on the data query request to obtain the data query result.

[0041] The present invention utilizes data graph technology to convert wide table related data into a graph structure, and utilizes data graphs for efficient query and analysis, thereby improving wide table processing computing power, achieving faster and more accurate wide table data construction, and being able to solve the performance bottleneck and resource consumption problems of existing technical solutions. In addition, the Markov chain is introduced to predict the future data state, so that the uncertainty of the master-slave table data change time in the wide table can be flexibly dealt with. Through the continuous learning and optimization of the Markov chain decision model, the wide table data writing and updating strategy can be accurately formulated, simplifying the complexity of data traversal, and greatly improving the streaming wide table data processing computing power. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0043] Figure 1 A flow chart of a data query method provided by an embodiment of the present invention;

[0044] Figure 2 An example of a wide table master-slave table data map provided by an embodiment of the present invention;

[0045] Figure 3 The present invention provides a structural block diagram of a data query device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] With the increase in data volume and the complexity of data relationships, large-scale real-time statistical function queries based on analytical databases that associate multiple tables face performance bottlenecks. Since the physical model of the business system follows the three-paradigm design specification and the data redundancy is low, performance is limited when performing association analysis of multiple large tables, consuming a large amount of computing resources and causing performance problems. In addition, since statistical function queries require real-time performance, aggregating associated tables into wide tables for data query is inefficient and cannot meet real-time requirements. In addition, when the amount of data is large, a large amount of computing resources will be consumed.

[0048] The existing method of real-time processing of online production data based on streaming wide tables is similar to the present invention. However, this method has some limitations, such as increased processing delays, excessive resource consumption, and difficulty in ensuring data consistency, which affect real-time performance and data processing efficiency. There are problems such as wide table data delay: since data needs to be periodically merged into multi-data source wide tables, data processing delays are long, reducing the real-time performance of data. Data delays will affect the real-time performance of data, making it impossible for data to reflect the current situation in a timely manner, affecting the accuracy and timeliness of business decisions; data redundancy and storage cost issues: when multiple data sources are transferred to single data source sub-tables in key-value pair format and merged into multi-data source wide tables, redundant data will occupy additional storage space and increase storage costs, especially when the amount of data is huge. In addition, additional work is required to maintain data consistency, increasing maintenance costs; data consistency issues: in the process of integrating multiple data sources, data changes in different data sources are independent events, and the time of streaming wide table consumption data change messages may be different, resulting in data inconsistency or data conflicts.

[0049] The method of the present invention aims to improve the processing computing power of wide table data: using data graph technology, converting wide table related data into a graph structure in the production and transaction link, using a graph database for efficient query and analysis, and being able to quickly find all related table data according to business fields, thereby achieving faster and more accurate wide table data processing and improving the processing computing power and effectiveness of wide table data; improving the query method of statistical function data: converting the original multi-table related query into a single-table query on the wide table, avoiding the large table multi-table related query after the business occurs, and reducing the consumption of analytical database computing resources, thereby improving the efficiency of statistical function data query; improving the accuracy of statistical function data: adopting a quasi-real-time anomaly detection and reconciliation mechanism to ensure that the data between the wide table and the transaction business table is always consistent, incrementally identifying inconsistent data through a sliding window algorithm, timely discovering and correcting data inconsistencies, and improving the accuracy of statistical function data query results.

[0050] In this application, the data writing and updating of the business wide table is driven by the transaction data change event. The streaming wide table processing system actively pulls the master table and slave table data from the data map for writing and updating based on the business key of the event. In view of the uncertainty of the change order of the original business table data corresponding to the master and slave tables in the business wide table, this mechanism can effectively ensure the integrity and availability of the wide table data. Here, the data map is more efficient than querying qualified data from a distributed relational database.

[0051] The following combination Figure 1 , Figure 1 A flowchart of a data query method provided by an embodiment of the present invention, the method may include:

[0052] S101: Determine the business table required for data query requirements in the business system, determine the wide table master and slave table from the business table, and construct a wide table and wide table master and slave table data map based on the wide table master and slave table.

[0053] This embodiment does not limit the execution subject of the data query method of this embodiment, and it can be set based on the actual application scenario, generally such as a server and a cloud server.

[0054] In this embodiment, there may be multiple business systems, and the data query method provided by this embodiment can be executed in each business system. In this embodiment, the business table required for the data query demand in the business system can be determined. In each business system, there may be multiple data query demands. The business table in this embodiment is a table storing the data required by the data query demand. This embodiment does not limit the number of business tables corresponding to each data query demand.

[0055] The business system in this embodiment may have multiple databases, and the business tables in this embodiment may come from the same or different databases.

[0056] After determining the business table required for data query needs in the business system, this embodiment can determine the master and slave tables of the wide table from the business table. The premise of data graph construction is to clarify the master and slave table division of the wide table. The master and slave table division is based on a comprehensive consideration of factors such as the data volume, data access frequency, and data dependency of the business table.

[0057] This embodiment does not limit the specific method of dividing the wide table into master and slave tables. Generally, in this embodiment, the business table with the largest data volume and the highest query frequency can be determined as the wide table master table; the remaining business tables that share the business primary key with the wide table master table can be determined as wide table slave tables.

[0058] Furthermore, this embodiment can construct a wide table and wide table master and slave table data map based on the wide table master and slave table. The data map can intuitively display the relationship between data entities and facilitate the analysis of complex data structures. In the data map, each node represents a data entity (such as a master table and a slave table), and the edge represents the relationship between entities. This embodiment can use the business primary key as the edge. The master table and the slave table can be connected through the business primary key, and the master tables can be connected through the sub-library key and the primary key. Figure 2 For example, for a transaction order, the transaction serial number can be used as the connection edge between the master table and the slave table. The data graph can show the complete transaction link and the associated data view.

[0059] Graph data can generally be stored in a dedicated graph database. When searching a wide master-slave table data graph, you can traverse it using a depth-first search algorithm. Specifically, start from the starting node and traverse down along a path until you can no longer traverse down. When you can no longer traverse down, return to the previous node and try other branches. Repeat the above steps until all nodes have been visited.

[0060] The wide table in this embodiment can be stored in the database after being generated. In this embodiment, the database storing the wide table can be an analytical database. This embodiment synchronizes the data in the distributed relational database to the wide table in the analytical database to convert the multi-table association query into a single-table query on the wide table in the analytical database, thereby greatly improving the data query efficiency of the statistical function and shortening the data response time.

[0061] S102: When data in the business table changes, obtain business table change data of the business table, and update the business table change data to the wide table and the wide table master-slave table data map according to the initial update strategy.

[0062] In this embodiment, when the data in the business table changes, the business table change data of the business table can be obtained and the business table change data can be changed to the wide table master-slave table data map. The business table change data can generally include data change information of operations such as adding, updating or deleting.

[0063] Since a business system may have multiple data query requirements, in this embodiment, there may be multiple wide table master-slave table data graphs under the business system. For each wide table master-slave table data graph, when the data in its corresponding business table changes, the business table change data of the business table can be obtained, and the business table change data can be changed to the corresponding wide table master-slave table data graph.

[0064] This embodiment does not limit the specific method of obtaining business table change data. Generally, a data probe of the event sensor can be set in the business table of the business system; when the data probe detects that the data in the business table has changed, the initial business table change data of the business table is obtained and uploaded to the server of the event sensor; based on the built-in rules, the initial business table change data is classified, filtered and screened to obtain the business table change data, and the business table change data is pushed to the built-in message queue of the server of the event sensor, so that the application end of the event sensor subscribes to the business table change data in the message queue, and updates the wide table based on the business table change data.

[0065] The event sensor is a tool for monitoring or detecting data changes in the native business tables of the application database of each business transaction center. In this embodiment, the number of master nodes in the multi-node subscription mode of the event sensor application end is one, that is, the event sensor application end supports only one node as the master node in the multi-node subscription mode, and the backup node will continue to work after the master node goes offline, ensuring stable operation in high-load scenarios.

[0066] In this embodiment, when the data method in the business table changes, the business table change data of the business table is obtained, and the business table change data is updated to the wide table and the wide table master-slave table data map according to the initial update strategy. This embodiment does not limit the specific update method of the initial update strategy, which can be consistent with the target update strategy.

[0067] The Markov chain decision model can reduce the complexity of traversal in the initial update strategy, save computing power and improve computing efficiency.

[0068] S103: Construct a Markov chain decision model. During the data update process, determine a target update strategy for the business table change data based on the Markov chain decision model.

[0069] This embodiment can construct a Markov chain decision model. During the data update process, a target update strategy for the business table change data is determined based on the Markov chain decision model.

[0070] Markov chain describes the transition process from one state to another in the state space and has the property of "no memory". Based on the change data pushed by the event sensor, the change of the target state of the change data is predicted and the decision model is constructed.

[0071] In view of the uncertainty of the change time of the original business table data corresponding to the master and slave tables in the wide table, a Markov chain model is constructed based on the change data pushed by the event sensor to determine the state space and state transition matrix, and a strategy for writing and updating wide table data is formulated according to the current state and transition probability.

[0072] First, the state space of the Markov chain decision model can be constructed. The state space can include a starting state and a target state. The starting state can be new addition to the main table, update of the main table, new addition to the slave table, and update of the slave table; the target state can be: new addition to the main table, update of the main table, new addition to the slave table but not recorded in the main table, update of the slave table but not recorded in the main table, new addition to the slave table and recorded in the main table, and update of the slave table and recorded in the main table.

[0073] In this embodiment, the service change data extracted during the data update process can be organized into a state transition record set of a Markov chain, each record including information such as the starting state, the target state, and the number of transitions.

[0074] A state transfer matrix can be further constructed, which contains the probability of each starting state transferring to each target state. In this embodiment, the state transfer matrix can be a 4×6 matrix, where each row of the matrix represents a starting state and each column represents a target state. The element P in the matrix ij represents the probability of transitioning from the starting state i to the target state j.

[0075] This embodiment may use a Markov chain model to formulate a strategy for writing and updating wide table data according to the current state and transition probability, that is, a target update strategy in this embodiment.

[0076] By introducing the Markov chain and predicting the future data status, we can flexibly deal with the uncertainty of the data change time of the master and slave tables in the wide table. Through the continuous learning and optimization of the Markov chain decision model, we can accurately formulate strategies for writing and updating wide table data, simplify the complexity of data traversal, and greatly improve the computing power of streaming wide table data processing.

[0077] S104: Based on the target update strategy, the business table change data is updated to the wide table and the wide table master-slave table data map.

[0078] This embodiment can use a target update strategy to update the business table change data to the wide table and the wide table master-slave table data map.

[0079] This embodiment does not limit the specific method of updating the business table change data to the wide table and the wide table master-slave table data map based on the target update strategy. The wide table is composed of the master table and the slave table. The core field of the wide table stores the most basic and most commonly used attribute information of the master table for quick query and access. The extended field of the wide table stores the extended information of the slave table. The addition or modification of the data in the master table and the slave table will be synchronously reflected in the wide table data structure to ensure the accuracy and integrity of the wide table data.

[0080] The specific target update strategy can be:

[0081] When the target state of the business table change data is added to the main table, the business table change data is updated to the wide table and the wide table master-slave table data map, the slave table data is pulled from the wide table master-slave table data map based on the business key, and the slave table data is updated to the wide table; specifically, when the wide table master table data is added, the new data is first updated to the wide table and the wide table master-slave table data map, then during the data query process, the main table data can be directly queried without waiting for the slave table data to be updated, thereby speeding up the query efficiency; further, the corresponding slave table related data is pulled from the wide table master-slave table data map according to the business primary key, and the slave table related data is updated to the wide table. If the corresponding slave table related data does not exist in the slave table, such as new or updated data in the slave table, the slave table data may not be updated.

[0082] When the target state of the business table change data is the main table update, the business table change data is updated to the wide table; specifically, when it involves the wide table main table data update, the wide table data update synchronization operation can be performed according to the wide table's sub-library key and primary key.

[0083] When the target state of the business table change data is that the slave table is added but the master table has no record, or the slave table is updated but the master table has no record, the business table change data is updated to the wide table master and slave table data map;

[0084] When the target state of the business table change data is that the slave table is added and the master table has been recorded, or the slave table is updated and the master table has been recorded, the business table change data is first updated to the wide table, and then updated to the wide table master-slave table data map.

[0085] Specifically, when the wide table is newly added or updated from the table data, there are two situations: the main table has been recorded and the main table has not been recorded, which respectively correspond to the main table data has been updated to the wide table and the main table data has not been updated to the wide table.

[0086] When the wide table has not recorded the newly added data of the master table, the update event of the slave table is ignored, and the newly added or updated data of the wide table slave table is temporarily stored in the wide table master and slave table data map. This is to avoid the situation that the slave table data is updated during the data query process, but the corresponding master table data is not updated, resulting in data query results that are inconsistent with the actual business, and the newly added slave table data is equivalent to redundant data, which affects the data query efficiency and the accuracy of the query results.

[0087] When the wide table has recorded the newly added data of the main table, the newly added data and updated data of the slave table can be directly updated to the wide table, and then the newly added data and updated data of the slave table can be directly updated to the wide table master-slave table data map.

[0088] S105: When a data query request is received, a data query is performed in the wide table based on the data query request to obtain a data query result.

[0089] In this embodiment, when a data query request is received, a data query can be performed in a wide table based on the data query request to obtain a data query result. This embodiment simplifies the original complex multi-table association query into a single table query on a wide table, greatly improving the data query efficiency of the statistical function and shortening the data response time.

[0090] Furthermore, this embodiment can introduce anomaly monitoring and fault-tolerance mechanisms to enable rapid recovery even in the face of abnormal situations, thereby ensuring uninterrupted data processing. Generally, a sliding window algorithm can be used to perform incremental quasi-real-time scanning of data graphs and wide table data, identify inconsistent data within a specified time range, and automatically trigger the data correction process to ensure data accuracy. At the same time, update the graph database records to maintain the timeliness and accuracy of the data graph. It supports flexible window size settings to meet different business monitoring needs, and its automated processing capabilities greatly reduce the burden of manual intervention and improve the efficiency and accuracy of data maintenance.

[0091] Specifically, this embodiment can obtain wide table logs and wide table master-slave table data map logs, and scan the wide table logs and wide table master-slave table data map logs based on a sliding window algorithm; when the scan obtains a data synchronization exception log, data compensation is performed on the wide table based on the wide table master-slave table data map.

[0092] This embodiment does not limit the specific method of scanning the wide table log and the wide table master-slave table data map log based on the sliding window algorithm. Generally, the left boundary time point of the sliding window can be determined as the current system time minus the preset time, and the right boundary time point of the sliding window can be determined as the current system time; the left boundary time point and the right boundary time point are updated at a preset frequency to complete the sliding scanning of the sliding window on the wide table log and the wide table master-slave table data map log. For example, the left boundary time point is the current system time minus 1 minute, and the right boundary time point is the current system time. The left boundary time point and the right boundary time point are updated at a fixed frequency of 5 seconds, so as to dynamically adjust the size and position of the window and realize the sliding of the window.

[0093] In this way, the system can capture the data between the master and slave tables and the wide table in the data map in near real time for comparison. Once data inconsistency is found, the data compensation mechanism is immediately triggered to automatically correct the difference to ensure the integrity and consistency of the wide table data.

[0094] Based on the above embodiments, the present invention uses data graph technology to convert wide table related data into a graph structure, and uses data graphs for efficient query and analysis, improves wide table processing computing power, and achieves faster and more accurate wide table data construction, which can solve the performance bottleneck and resource consumption problems of existing technical solutions. In addition, the Markov chain is introduced to predict the future data state, so that the uncertainty of the change time of the master and slave table data in the wide table can be flexibly responded to. Through the continuous learning and optimization of the Markov chain decision model, the wide table data writing and updating strategy can be accurately formulated, simplifying the complexity of data traversal, and greatly improving the streaming wide table data processing computing power.

[0095] The following combination Figure 3 , Figure 3 A structural block diagram of a data query device provided by an embodiment of the present invention, the device may include:

[0096] The first module 100 is used to determine the business table required for data query requirements in the business system, determine the wide table master and slave table from the business table, and construct a wide table and wide table master and slave table data map based on the wide table master and slave table;

[0097] The second module 200 is used to obtain the business table change data of the business table when the data in the business table changes, and update the business table change data to the wide table and the wide table master-slave table data map according to the initial update strategy;

[0098] The third module 300 is used to construct a Markov chain decision model, and during the data update process, determine the target update strategy of the business table change data based on the Markov chain decision model;

[0099] The fourth module 400 is used to update the business table change data to the wide table and the wide table master-slave table data map based on the target update strategy;

[0100] The fifth module 500 is used to perform data query in the wide table based on the data query request when receiving a data query request, and obtain a data query result.

[0101] Based on the above embodiments, the present invention uses data graph technology to convert wide table related data into a graph structure, and uses data graphs for efficient query and analysis, improves wide table processing computing power, and achieves faster and more accurate wide table data construction, which can solve the performance bottleneck and resource consumption problems of existing technical solutions. In addition, the Markov chain is introduced to predict the future data state, so that the uncertainty of the change time of the master and slave table data in the wide table can be flexibly responded to. Through the continuous learning and optimization of the Markov chain decision model, the wide table data writing and updating strategy can be accurately formulated, simplifying the complexity of data traversal, and greatly improving the streaming wide table data processing computing power.

[0102] Based on the above embodiment, the fourth module 400 may include:

[0103] The first unit is used for updating the business table change data to the wide table and the wide table master-slave table data map when the target state of the business table change data is the addition of the master table, pulling the slave table data from the wide table master-slave table data map based on the business key, and updating the slave table data to the wide table;

[0104] The second unit is used for updating the business table change data to the wide table when the target state of the business table change data is the main table update;

[0105] The third unit is used to update the business table change data to the wide table master-slave table data map when the target state of the business table change data is that the slave table is added but not recorded in the master table or the slave table is updated but not recorded in the master table;

[0106] The fourth unit is used to update the business table change data to the wide table first, and then update the business table change data to the wide table master-slave table data map when the target state of the business table change data is newly added to the slave table and recorded in the master table or updated in the slave table and recorded in the master table.

[0107] Based on the above embodiments, the first module 100 may include:

[0108] A fifth unit is used to determine the business table with the largest data volume and the highest query frequency as a wide table master table;

[0109] The sixth unit is used to determine the remaining business tables that share the business primary key with the wide table master table as wide table slave tables.

[0110] Based on the above embodiments, the device may further include:

[0111] A sixth module is used to obtain a wide table log and a wide table master-slave table data map log, and scan the wide table log and the wide table master-slave table data map log based on a sliding window algorithm;

[0112] The seventh module is used to perform data compensation on the wide table based on the master-slave table data map of the wide table when the data synchronization exception log is obtained through scanning.

[0113] Based on the above embodiments, the sixth module may include:

[0114] A seventh unit is used to determine the left boundary time point of the sliding window as the current system time minus the preset time, and determine the right boundary time point of the sliding window as the current system time;

[0115] The eighth unit is used to update the left boundary time point and the right boundary time point at a preset frequency to complete the sliding scan of the sliding window on the wide table log and the wide table master-slave table data map log.

[0116] Based on the above embodiments, the second module 200 may include:

[0117] A ninth unit is used to set a data probe of an event sensor in a business table of the business system;

[0118] A tenth unit is used for acquiring initial business table change data of the business table and uploading it to the server of the event sensor when the data in the business table is detected to be changed by the data probe;

[0119] The eleventh unit is used to classify, filter and screen the initial business table change data based on built-in rules to obtain the business table change data, and push the business table change data to the built-in message queue of the server of the event sensor, so that the application end of the event sensor subscribes to the business table change data in the message queue, and updates the wide table based on the business table change data.

[0120] Based on the above embodiments, the second module 200 includes:

[0121] The twelfth unit is configured to obtain the business table change data of the target field in the business table when data of the target field in the business table is changed.

[0122] Based on the above embodiments, the present invention further provides a data query device, which may include a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the steps provided in the above embodiments may be implemented. Of course, the device may also include various necessary network interfaces, power supplies, and other components.

[0123] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by an execution terminal or a processor, the method provided in the embodiment of the present invention can be implemented; the storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0124] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

Claims

1. A data query method, characterized in that: include: Determine the business table required for data query needs in the business system, determine the wide table master and slave table from the business table, and construct a wide table and wide table master and slave table data map based on the wide table master and slave table; When the data in the business table changes, the business table change data of the business table is obtained, and the business table change data is updated to the wide table and the wide table master-slave table data map according to the initial update strategy; Constructing a Markov chain decision model, and in the data update process, determining a target update strategy for the business table change data based on the Markov chain decision model; Based on the target update strategy, the business table change data is updated to the wide table and the wide table master-slave table data map; When a data query request is received, a data query is performed in the wide table based on the data query request to obtain a data query result.

2. The data query method according to claim 1, characterized in that: Based on the target update strategy, updating the business table change data to the wide table and the wide table master-slave table data map includes: When the target state of the business table change data is the addition to the main table, the business table change data is updated to the wide table and the wide table master-slave table data map, the slave table data is pulled from the wide table master-slave table data map based on the business key, and the slave table data is updated to the wide table; When the target state of the business table change data is main table update, the business table change data is updated to the wide table; When the target state of the business table change data is that the slave table is newly added but not recorded in the master table, or the slave table is updated but not recorded in the master table, the business table change data is updated to the wide table master-slave table data map; When the target state of the business table change data is that the slave table is added and the master table has been recorded or the slave table is updated and the master table has been recorded, the business table change data is first updated to the wide table, and then updated to the wide table master-slave table data map.

3. The data query method according to claim 1, characterized in that: Determining a wide table master and slave table from the business table includes: Determine the business table with the largest data volume and the highest query frequency as the wide table master table; The remaining business tables that share a business primary key with the wide table master table are determined as wide table slave tables.

4. The data query method according to claim 1, characterized in that: Also includes: Obtain a wide table log and a wide table master-slave table data map log, and scan the wide table log and the wide table master-slave table data map log based on a sliding window algorithm; When the data synchronization exception log is obtained through scanning, data compensation is performed on the wide table based on the master-slave table data map of the wide table.

5. The data query method according to claim 4, characterized in that: The scanning of the wide table log and the wide table master-slave table data graph log based on the sliding window algorithm includes: The left boundary time point of the sliding window is determined as the current system time minus the preset time, and the right boundary time point of the sliding window is determined as the current system time; The left boundary time point and the right boundary time point are updated at a preset frequency to complete the sliding scan of the sliding window on the wide table log and the wide table master-slave table data map log.

6. The data query method according to claim 1, characterized in that: When the data in the business table changes, obtaining the business table change data of the business table includes: Setting a data probe of an event sensor in a business table of the business system; When the data in the business table is detected to have changed through the data probe, initial business table change data of the business table is obtained and uploaded to the server of the event sensor; Based on built-in rules, the initial business table change data is classified, filtered and screened to obtain the business table change data, and the business table change data is pushed to the built-in message queue of the server of the event sensor, so that the application end of the event sensor subscribes to the business table change data in the message queue and updates the wide table based on the business table change data.

7. The data query method according to claim 1, characterized in that: When the data in the business table changes, obtaining the business table change data of the business table includes: When data in the target field in the business table is changed, the business table change data of the target field in the business table is acquired.

8. A data query device, characterized in that: include: The first module is used to determine the business table required for data query needs in the business system, determine the wide table master and slave table from the business table, and construct a wide table and wide table master and slave table data map based on the wide table master and slave table; The second module is used to obtain the business table change data of the business table when the data in the business table changes, and update the business table change data to the wide table and the master-slave table data map of the wide table according to the initial update strategy; The third module is used to construct a Markov chain decision model, and during the data update process, determine the target update strategy of the business table change data based on the Markov chain decision model; A fourth module is used to update the business table change data to the wide table and the wide table master-slave table data map based on the target update strategy; The fifth module is used to perform data query in the wide table based on the data query request when receiving the data query request to obtain the data query result.

9. A data query device, characterized in that: include: Memory, for storing computer programs; A processor, configured to implement the data query method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the data query method according to any one of claims 1 to 7 is implemented.

Images