A data query method and device, electronic equipment and computer readable medium

Abstract

The application discloses a data query method and device, electronic equipment and computer readable medium, relates to the technical field of computers, and a specific embodiment includes: receiving a data query request, obtaining a corresponding instruction type identifier and data timeliness identifier; positioning a target database table according to the instruction type identifier and the data timeliness identifier, and then determining the number of the target database table; in response to the number being multiple, obtaining query mode information corresponding to the instruction type identifier, determining a query statement assembly order based on the query mode information, assembling the query statement and the target database table based on the query statement assembly order, and generating a joint query statement; executing the joint query statement, and then outputting data obtained by querying. Thus, when data is queried, the range of the query is narrowed, and the data query is more accurate and faster.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and particularly relates to a data query method and device, electronic equipment and a computer readable medium. BACKGROUND

[0002] At present, a warehouse management system (WMS) warehouse system report system supports dynamic configuration of indexes and query conditions based on smartQuery component configuration xml, which meets the use of most scenarios; a database bottom layer uses Mysql to store business application data. System operation data needs to retain full data for nearly many years, and the existing report query system is prone to query timeout and data loss during operation.

[0003] In the process of implementing the present application, the inventors have found that at least the following problems exist in the prior art:

[0004] The Mysql database stores ten million level data, and the performance efficiency is relatively low when executing data query. SUMMARY

[0005] Therefore, the embodiments of the present application provide a data query method and device, electronic equipment and a computer readable medium, which can solve the problem that the existing Mysql database stores ten million level data, and the performance efficiency is relatively low when executing data query.

[0006] To achieve the above object, according to an aspect of the embodiments of the present application, a data query method is provided, comprising:

[0007] receiving a data query request, obtaining a corresponding instruction type identifier and data timeliness identifier;

[0008] locating a target database table according to the instruction type identifier and the data timeliness identifier, and then determining the number of the target database table;

[0009] in response to the number being multiple, obtaining query mode information corresponding to the instruction type identifier, determining query statement assembly order based on the query mode information, assembling the query statement and the target database table based on the query statement assembly order, to generate a joint query statement;

[0010] executing the joint query statement, and then outputting the data obtained by query.

[0011] Optionally, before locating the target database table according to the instruction type identifier and the data timeliness identifier, the method further comprises:

[0012] obtaining full data and deployment location identifier;

[0013] The full amount data is split based on the deployment location identifier to generate the sharded data;

[0014] The service identifier corresponding to the sharded data is determined, and then the sharded data is divided into buckets based on the service identifier to obtain the bucketed data;

[0015] The bucketed data is synchronized to the database table.

[0016] Optionally, the target database table is located according to the instruction type identifier and the data timeliness identifier, including:

[0017] From the synchronized database table, one or more database tables corresponding to the instruction type identifier and the data timeliness identifier are determined, and then the one or more database tables are determined as the target database table.

[0018] Optionally, the sharded data is divided into buckets based on the service identifier to obtain the bucketed data, including:

[0019] The field name corresponding to each sharded data is determined, and the time stamp of the sharded data with the same field name is determined;

[0020] The new data and the old data in the sharded data with the same field name are determined according to the time stamp;

[0021] The old data is replaced by the new data, and then the updated sharded data is obtained;

[0022] The data corresponding to the same service identifier in the updated sharded data is put into the same data bucket to obtain the bucketed data.

[0023] Optionally, the data corresponding to the same service identifier in the updated sharded data is put into the same data bucket to obtain the bucketed data, including:

[0024] The data corresponding to the same service identifier in the updated sharded data is executed for hash calculation, and then the result data of the hash calculation is stored in the same data bucket to obtain the bucketed data.

[0025] Optionally, the bucketed data is synchronized to the database table, including:

[0026] The data timeliness identifier corresponding to the bucketed data is determined;

[0027] The bucketed data is synchronized to the database table corresponding to the data timeliness identifier corresponding to the bucketed data.

[0028] In addition, the application also provides a data query device, including:

[0029] The receiving unit is configured to receive a data query request, and obtain a corresponding instruction type identifier and a data timeliness identifier;

[0030] The positioning unit is configured to locate the target database table according to the instruction type identifier and the data timeliness identifier, and determine the number of the target database table;

[0031] The joint query statement generation unit is configured to, in response to the number being multiple, acquire query mode information corresponding to the instruction type identifier, determine a query statement assembly order based on the query mode information, and assemble the query statement and the target database table based on the query statement assembly order to generate the joint query statement.

[0032] The execution unit is configured to execute the joint query statement, and output data obtained by the query.

[0033] Optionally, the apparatus further comprises a synchronization unit configured to:

[0034] acquire full data and a deployment location identifier;

[0035] split the full data based on the deployment location identifier to generate sharded data;

[0036] determine a business identifier corresponding to the sharded data, and perform bucketing on the sharded data based on the business identifier to obtain bucketed data;

[0037] synchronize the bucketed data to a database table.

[0038] Optionally, the positioning unit is further configured to:

[0039] determine one or more database tables corresponding to the instruction type identifier and the data timeliness identifier from the synchronized database table, and determine the one or more database tables as the target database table.

[0040] Optionally, the data query apparatus further comprises a bucketing unit configured to:

[0041] determine a field name corresponding to each piece of sharded data, and determine a timestamp of sharded data with the same field name;

[0042] determine new data and old data in the sharded data with the same field name according to the timestamp;

[0043] replace the old data with the new data to obtain updated sharded data;

[0044] put data corresponding to the same business identifier in the updated sharded data into the same data bucket to obtain the bucketed data.

[0045] Optionally, the bucketing unit is further configured to:

[0046] perform hash calculation on data corresponding to the same business identifier in the updated sharded data, and store result data of the hash calculation into the same data bucket to obtain the bucketed data.

[0047] Optionally, the synchronization unit is further configured to:

[0048] determine a data timeliness identifier corresponding to the bucketed data;

[0049] synchronize the bucketed data into a database table corresponding to the data timeliness identifier corresponding to the bucketed data.

[0050] In addition, the present application also provides a data query electronic device, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, so that the one or more processors implement the data query method as described above.

[0051] In addition, the present application also provides a computer readable medium, which stores a computer program, and the program is executed by a processor to implement the data query method as described above.

[0052] An embodiment of the above application has the following advantages or beneficial effects: the present application receives a data query request, obtains a corresponding instruction type identifier and data timeliness identifier, locates a target database table according to the instruction type identifier and the data timeliness identifier, and then determines the number of the target database table; in response to the number being multiple, obtains query mode information corresponding to the instruction type identifier, determines query statement assembly order based on the query mode information, assembles the query statement and the target database table based on the query statement assembly order, to generate a joint query statement; executes the joint query statement, and then outputs the data obtained by querying. Thus, when data is queried, the range of the query is narrowed, so that the data query is more accurate and faster.

[0053] The further effects of the above-mentioned non-conventional optional mode will be described in the following combined with the specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0054] The accompanying drawings serve to better understand the present application and do not constitute an improper limitation on the present application. Among them:

[0055] Figure 1 is a schematic diagram of the main process of the data query method according to the first embodiment of the present application;

[0056] Figure 2 is a schematic diagram of the main process of the data query method according to the second embodiment of the present application;

[0057] Figure 3 is a schematic diagram of the application scenario of the data query method according to the third embodiment of the present application;

[0058] Figure 4 is a schematic diagram of the main unit of the data query device according to the embodiment of the present application;

[0059] Figure 5 This is an exemplary system architecture diagram to which embodiments of this application can be applied;

[0060] Figure 6 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers in the embodiments of this application. Detailed Implementation

[0061] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description. The acquisition, storage, use, and processing of data in the technical solutions of this application all comply with relevant national laws and regulations.

[0062] Figure 1 This is a schematic diagram of the main flow of the data query method according to the first embodiment of this application, as shown below. Figure 1 As shown, the data query methods include:

[0063] Step S101: Receive a data query request and obtain the corresponding instruction type identifier and data timeliness identifier.

[0064] In this embodiment, the execution entity of the data query method (e.g., a server) can receive data query requests via wired or wireless connections. Specifically, the data query request can be a request to query historical data. After receiving the data query request, the execution entity can obtain the instruction type identifier carried in the data query request, where the instruction type refers to the type of the hint instruction. Instruction type identifiers, such as LY and ZX, represent routing and control, respectively. Routing is used to support dimension table queries, such as queries by date, merchant, or region. Control is used for caching query results, query timeouts, and multi-threaded segmented execution. The data timeliness identifier is used to characterize whether the data to be queried is real-time data or historical data.

[0065] Step S102: Locate the target database table based on the instruction type identifier and data timeliness identifier, and then determine the number of target database tables.

[0066] The routing instruction and the control instruction respectively have corresponding database tables, which are used to query or store data in the corresponding database tables. The database tables can also be divided into real-time database tables for storing real-time data and historical database tables for storing historical data. For example, the database tables corresponding to the hive cluster can store historical data.

[0067] The execution subject can determine the corresponding database tables according to the obtained instruction type identifier, such as database table A, database table B, and database table C. Then, the database tables that also satisfy the data timeliness identifier are determined in the database table A, the database table B, and the database table C, such as the database tables that satisfy the query of real-time data. If the database table A and the database table C satisfy, then the database table A and the database table C are the target database tables. In this embodiment, for example, the execution subject can determine that the number of target database tables is two.

[0068] In step S103, in response to the number being multiple, the query mode information corresponding to the instruction type identifier is obtained, the query statement assembly order is determined based on the query mode information, and the query statement and the target database table are assembled based on the query statement assembly order to generate a joint query statement.

[0069] The number of two or more can represent multiple. When the execution subject determines that the number of target database tables is multiple (for example, two), a joint query statement can be generated in combination with the target database table and the instruction type identifier (for example, when the instruction type identifier is LY, it represents a query by dimension table). For example: SELECT * FROM database table A, database table C. WHERE 1 or more connection conditions [AND query conditions].

[0070] Specifically, the query mode information corresponding to the instruction type identifier is obtained. For example, the query mode information can be querying by date, querying by merchant, querying by region, etc. The query statement assembly order is determined based on the query mode information, such as assembly by date, assembly by the first letter of the merchant name, assembly by the region name, etc. The query statement and the target database table are assembled based on the query statement assembly order to generate a joint query statement. For example, the data stored in the database table A is the latest date, and the data stored in the database table C is the data within 10 seconds. The order of the database table A in the joint query statement is placed in the database table C, so that when the joint query statement is executed, the data in the database table A is queried first, and then the data in the database table C is queried.

[0071] In step S104, the joint query statement is executed, and then the data obtained by querying is output.

[0072] After the execution subject generates the joint query statement, the execution subject can query the corresponding data from the corresponding database tables (for example, the database table A and the database table C) according to the order of the database table query set in the joint query statement.

[0073] The embodiment receives a data query request, obtains a corresponding instruction type identifier and a data timeliness identifier, locates a target database table according to the instruction type identifier and the data timeliness identifier, and then determines the number of the target database table. In response to the number being multiple, the embodiment obtains query mode information corresponding to the instruction type identifier, determines a query statement assembly order based on the query mode information, assembles the query statement and the target database table based on the query statement assembly order, generates a joint query statement, executes the joint query statement, and then outputs the data obtained by the query. Thus, the range of the query can be narrowed when the data is queried, and the data query is more accurate and faster.

[0074] Figure 2 FIG. 1 is a main flow schematic diagram of a data query method according to the second embodiment of the present application, as shown in FIG. 1, the data query method comprises the following steps. Figure 2

[0075] In step S201, full data and a deployment location identifier are obtained.

[0076] Before the data query is performed, the execution subject can first perform data processing and storage corresponding to the data query mode. The execution subject can first obtain full data and a deployment location identifier for the full data when the data processing and storage are performed. The full data can include various business table metadata and task configuration parameters. The business table metadata: a business table DDL statement is entered, the system is parsed based on a druid framework, table metadata dictionary information is generated, and the table metadata dictionary information is stored in a redis cache.

[0077] The task configuration parameters include:

[0078] Batch processing task information: the component supports two data sources, mysql and hive, the configuration parameters include batch processing sql information and data source connection information; real-time task message information: the component supports multiple message components, jdq, jmq, jmq4, kafka, and rocketmq, after the message is accessed, the data format is uniformly converted to jdwdata format in the component; task start parameters: parallelism, data sharding strategy, window time, data conversion strategy, data filtering strategy, and other task information.

[0079] ​The execution subject can obtain the corresponding deployment location identifier from the full volume data after obtaining the full volume data. The deployment location identifier is used to represent where to store the full volume data to the database table. For example, the deployment location identifier can be A and C, respectively representing storing the full volume data to the database table A and the database table C. As another example, the deployment location identifier can also be the institution number, the distribution center number, and the warehouse number corresponding to each data in the full volume data.

[0080] In step S202, the full volume data is split based on the deployment location identifier to generate the sharded data.

[0081] The execution subject can split the full volume data based on the deployment location identifier, for example, for the processing and storage of the medicine warehouse data, based on the institution number + distribution center number + warehouse number. That is, the full volume data is data-sharded based on the deployment location identifier (such as the institution number + distribution center number + warehouse number), and then each sharded data is obtained.

[0082] In step S203, the business identifier corresponding to the sharded data is determined, and then the sharded data is divided into buckets based on the business identifier to obtain the bucketed data.

[0083] The business identifier corresponding to each data in each sharded data can be different. After obtaining the sharded data, the execution subject can determine the business identifier (that is, the business key) corresponding to each sharded data, and divide each sharded data based on the business identifier corresponding to each sharded data. That is, the data in the sharded data corresponding to the same business identifier is divided into the same data bucket.

[0084] Specifically, the sharded data is divided into buckets based on the business identifier to obtain the bucketed data, including: determining the field name (such as the primary key corresponding to the sharded data) corresponding to each sharded data, determining the time stamp of the sharded data with the same field name (that is, determining the time stamp of the sharded data with the same primary key); determining the new data and the old data in the sharded data with the same field name according to the time stamp (that is, determining the new data and the old data in the sharded data with the same primary key); replacing the old data with the new data (that is, replacing the old data with the new data in the sharded data with the same primary key), and then obtaining the updated sharded data; and putting the data corresponding to the same business identifier in the updated sharded data into the same data bucket to obtain the bucketed data. The execution subject can further update the sharded data after replacing the old data with the new data in the sharded data with the same primary key, and put the data corresponding to the same business identifier in the updated sharded data into the same data bucket. This can improve the data writing performance and ensure the consistency of the data.

[0085] Specifically, the data corresponding to the same service identifier in the updated shard data is put into the same data bucket to obtain the bucketed data, including: performing hash calculation on the data corresponding to the same service identifier in the updated shard data, and then storing the result data of the hash calculation into the same data bucket to obtain the bucketed data. Thus, memory can be saved and space consumption can be reduced by using hash storage.

[0086] In step S204, the bucketed data is synchronized to the database table.

[0087] Specifically, the bucketed data is synchronized to the database table, including: determining the data timeliness identifier corresponding to the bucketed data, such as SS (representing real-time data) or LS (representing historical data); and synchronizing the bucketed data to the database table corresponding to the data timeliness identifier corresponding to the bucketed data. The execution subject can synchronize the data bucket containing real-time data to the database table storing real-time data, and synchronize the data bucket containing historical data to the database table storing historical data. This is for subsequent data query, based on the data timeliness identifier to quickly query the corresponding data.

[0088] In step S205, a data query request is received, and the corresponding instruction type identifier and data timeliness identifier are obtained.

[0089] In step S206, the target database table is located according to the instruction type identifier and the data timeliness identifier, and the number of target database tables is determined.

[0090] Specifically, the target database table is located according to the instruction type identifier and the data timeliness identifier, including: determining one or more database tables corresponding to the instruction type identifier and the data timeliness identifier from the synchronized database table, and then determining the one or more database tables as the target database table.

[0091] In step S207, in response to the number being multiple, the query mode information corresponding to the instruction type identifier is obtained, the query statement assembly order is determined based on the query mode information, the query statement and the target database table are assembled based on the query statement assembly order, and a joint query statement is generated.

[0092] In step S208, the joint query statement is executed, and the data obtained by the query is output.

[0093] The embodiments of the present application can be applied to the scene of optimizing the synchronization and processing of park logistics and warehousing data based on stream computing. By performing data bucketing based on the service identifier when storing data, and storing the bucketed data based on the data timeliness identifier corresponding to the bucketed data, the data query can be based on the data timeliness identifier to narrow the query range and make the data query more accurate and faster.

[0094] Figure 3 is a schematic diagram of an application scenario of a data query method according to a third embodiment of the present application. The data query method of the embodiments of the present application can be applied to a scenario of optimizing synchronization, processing and data query of park logistics and warehousing data based on stream computing. For example, as shown in Figure 3 , by using flink based on stream computing as a real-time computing engine and a TIDB storage engine, mass historical data can be stored and queried in the medicine GSP over-checking scenario. Figure 3 In the figure, "read" above the Hive cluster, binlog and jmq message is reading multiple data source types, which are all data sources for flink data processing. The Hive cluster mainly stores historical data, the binlog mainly refers to the binlog log of mysql, and the jmq is other business processing data. Smart-query is a report backend query service, which publishes a rest counterpart, is used to receive page requests, query data and return results. Flink is an open source real-time computing engine of apache. Flink-sql is a batch processing data API based on sql syntax, which is mainly used for batch processing and simple data synchronization. Flink-stream is an API for data processing through coding, which is more flexible than sql and can process more complex logic. Time window: mainly used to solve the problem of data disorder and temporary logical window in data aggregation, such as calculating the order quantity in 10s. The source in the source / sink window is the data reading access operator of flink engine, and the sink is the data output storage operator of flink engine. The generated flink-sql task information: when historical data needs to be processed, the historical data has been closed to hive, at this time, a flink-sql task can be created to pull historical data, and the historical data of multiple tables can be synchronized.

[0095] The user submits a data processing task, which refers to data processing and storage task. A data processing task is mainly used for data processing task of one or more report bottom tables. Main submission parameters: metadata: for example, if it is jmq, it is the field information of the message, if it is binlog or hive, it is the table field information. Task information: parameters required for running flink task script information, such as operator parallelism, checkpoint timeout and the like. The system parses the above configuration parameters into executable scripts of flink engine, and then publishes the scripts to flink cluster to execute data processing and storage. Before performing data query, the execution subject receives the data processing task, obtains the initialized business table metadata and task configuration parameters.

[0096] Among them, the business table metadata: enter the business table DDL statement, and the system is based on the druid framework to parse and generate table metadata dictionary information and store it in redis cache.

[0097] Task configuration parameters:

[0098] Batch task information: the component supports mysql and hive data sources, and the configuration parameters include batch sql information and data source connection information.

[0099] Real-time task message information: the component supports multiple message components jdq, jmq, jmq4, kafka, rocketmq, and after accessing the message, the component internally converts the data format to jdwdata format. Multiple messages: support configuring multiple messages in array form. Blood relationship: support join connection between multiple messages. When connecting messages, the default window is based on flink-state storage. Dimension table association: dimension table information supports storage based on redis or mysql;

[0100] Task start parameters: parallelism, data sharding strategy, window time, data conversion strategy, data filtering strategy, and task information.

[0101] Data filtering rules: in actual message processing, in order to avoid the influence of some business irrelevant data or repeated data, data filtering and cleaning are needed, the component is based on rule engine, loads user configured data filtering script, meets the data filtering requirements of different scenes.

[0102] Flink task metadata cache: based on redis as metadata cache storage, when flink task starts, read metadata information, if metadata changes, update through redis key space change event, do not need to restart or redeploy flink task.

[0103] Task submission: call bdp platform open Rest interface, start task, parameters include git address, code branch, start class information. After successful submission, flink task starts, task script automatically loads flink task metadata.

[0104] Data synchronization TIDB:

[0105] In order to improve the data writing performance and ensure the consistency of data, the component has made the following optimization processing to the data stream:

[0106] Data Sharding: Based on the flink bypass output feature, the data stream is split into different sub-streams. Since the upstream message is full data, but the warehouse system is deployed in different regions, when synchronizing data, it can be bypassed based on "organization number + distribution center number + warehouse number". Support custom implementation of sharding rules to meet other scenarios.

[0107] Data Bucket Merge: After receiving the data, the component splits the data into buckets based on the business key. Based on the business key, the data stream is keyBy bucketed, a 10s window is opened, and data merging is performed based on the merging strategy; Data merging strategy: based on the same specified business key (supporting multiple fields), compare by timestamp or primary key, new data covers old data in the same field. Based on the same specified business key (supporting multiple fields), aggregate and merge data. Field ignore: discard certain fields according to configuration information.

[0108] Fault Tolerance Processing: This mechanism is only applicable to database table primary key monotonic increment, if there is a blood relationship, then the primary table after join is used as the basis for judgment. WAL mechanism: an undo.log log file is written before each batch update. Flink does not start the checkPoint mechanism: after the flink task restarts, the undo.log file is scanned when writing data to compensate for data, and deleted. Flink starts the checkPoint mechanism: ignore and delete the undo.log file.

[0109] Smart-query report component extension: In a relational database, a database table is a collection of two-dimensional arrays that represent and store the relationships between data objects. It is composed of vertical columns and horizontal rows. The component implements declarative programming through custom hint instructions, rewrites or optimizes sql according to different instruction rules, and improves query performance.

[0110] Hint instructions are mainly divided into two categories according to their functions: routing and execution control. Routing: used to support dimension table query methods such as date, merchant, and region. Control: used for query result caching, query timeout, multi-threaded segmented execution, etc.

[0111] Execute hint instructions based on rule engine, decouple logic, and improve scalability.

[0112] Instruction definition: the component supports custom instructions based on SPI extension points.

[0113] Hint instruction prefix flag: " / *queryHint:* / "

[0114] Instruction type:

[0115] / *queryHint:Cache* / Supports sql level result cache

[0116] / *queryHint:Schema* / Rewrite of sql table

[0117] / *queryHint:multExec* / Multi-threaded segmented execution

[0118] Embodiments of the application can realize query optimization and control based on custom queryHint instruction set. Embodiments of the application can realize stream batch integrated processing based on flink real-time computing; operation and maintenance personnel only need to maintain a set of scripts, and the system automatically generates real-time tasks and batch processing tasks. The report library uses TIDB to replace mysql to store data, the TIDB engine supports mass data storage, and the query efficiency is better than mysql, and at the same time, TIDB supports mysql protocol, without changing the existing system architecture, the system upgrade can be completed. Realize the application of real-time computing in the warehouse scene, improve the system performance. It can meet the requirements of pharmaceutical warehouse GSP over-checking, realize the storage and query of historical data. It can replace the hive offline data processing data, reduce the cost of manual maintenance.

[0119] Figure 4 is a schematic diagram of the main unit of the data query device according to the embodiments of the application. As shown in Figure 4 , the data query device 400 includes a receiving unit 401, a positioning unit 402, a joint query statement generation unit 403 and an execution unit 404.

[0120] The receiving unit 401 is configured to receive a data query request, and obtain a corresponding instruction type identifier and data timeliness identifier.

[0121] The positioning unit 402 is configured to locate the target database table according to the instruction type identifier and the data timeliness identifier, and further determine the number of target database tables.

[0122] The joint query statement generation unit 403 is configured to, in response to the number being multiple, obtain query mode information corresponding to the instruction type identifier, determine query statement assembly order based on the query mode information, and assemble the query statement and the target database table based on the query statement assembly order to generate a joint query statement.

[0123] The execution unit 404 is configured to execute the joint query statement, and further output the data obtained by querying.

[0124] In some embodiments, the device further includes Figure 4The synchronization unit not shown in the figure is configured to: acquire the full volume data and the deployment location identifier; split the full volume data based on the deployment location identifier to generate the sharded data; determine the service identifier corresponding to the sharded data, and then perform bucketing on the sharded data based on the service identifier to obtain the bucketed data; and synchronize the bucketed data to the database table.

[0125] In some embodiments, the positioning unit 402 is further configured to: determine one or more database tables corresponding to the instruction type identifier and the data timeliness identifier from the synchronized database table, and then determine the one or more database tables as the target database table.

[0126] In some embodiments, the data query apparatus further includes Figure 4 The bucketing unit not shown in the figure is configured to: determine the field name corresponding to each piece of sharded data, determine the time stamp of the sharded data with the same field name; determine the new data and the old data in the sharded data with the same field name according to the time stamp; replace the old data with the new data, and then obtain the updated sharded data; and put the data corresponding to the same service identifier in the updated sharded data into the same data bucket to obtain the bucketed data.

[0127] In some embodiments, the bucketing unit is further configured to: perform hash calculation on the data corresponding to the same service identifier in the updated sharded data, and then store the result data of the hash calculation into the same data bucket to obtain the bucketed data.

[0128] In some embodiments, the synchronization unit is further configured to: determine the data timeliness identifier corresponding to the bucketed data; and synchronize the bucketed data to the database table corresponding to the data timeliness identifier corresponding to the bucketed data.

[0129] In some embodiments, the joint query statement generation unit 403 is further configured to: acquire the query mode information corresponding to the instruction type identifier; determine the query statement assembly order based on the query mode information; and assemble the query statement and the target database table based on the query statement assembly order to generate the joint query statement.

[0130] It should be noted that the data query method and the data query apparatus of the present application have a corresponding relationship in the specific implementation content, and therefore repeated content will not be described.

[0131] Figure 5 An exemplary system architecture 500 to which embodiments of the present application can be applied is shown.

[0132] As Figure 5As shown, the system architecture 500 can include terminal devices 501, 502, 503, a network 504 and a server 505. The network 504 is a medium for providing communication links between the terminal devices 501, 502, 503 and the server 505. The network 504 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0133] A user can use the terminal devices 501, 502, 503 to interact with the server 505 through the network 504 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 501, 502, 503, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0134] The terminal devices 501, 502, 503 can be various electronic devices with data query processing screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers and desktop computers, etc.

[0135] The server 505 can be a server providing various services, such as a background management server providing support for data query requests submitted by users using the terminal devices 501, 502, 503 (only as an example). The background management server can receive a data query request, obtain a corresponding instruction type identifier and a data timeliness identifier; locate a target database table according to the instruction type identifier and the data timeliness identifier, and then determine the number of target database tables; in response to the number being multiple, obtain query method information corresponding to the instruction type identifier, determine a query statement assembly order based on the query method information, assemble the query statement and the target database tables based on the query statement assembly order, to generate a joint query statement; execute the joint query statement, and then output the data obtained by the query. Thus, when data is queried, the scope of the query is narrowed, making the data query more accurate and faster.

[0136] It should be noted that the data query method provided by the embodiments of the present application is generally executed by the server 505, and correspondingly, the data query apparatus is generally provided in the server 505.

[0137] It should be understood that, Figure 5 The number of terminal devices, networks and servers in the above system architecture is only illustrative. According to the needs of implementation, there can be any number of terminal devices, networks and servers.

[0138] Reference will now be made to Figure 6 which shows a structural schematic diagram of a computer system 600 suitable for implementing the terminal device of the embodiments of the present application. Figure 6 The terminal device shown is only an example and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0139] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 602 or programs loaded from storage section 608 into random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of the computer system 600. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0140] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0141] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit (CPU) 601, it performs the functions defined above in the system of this application.

[0142] Note that the computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium or a combination thereof. The computer-readable storage medium can include, for example, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, the computer-readable storage medium can be any tangible medium that contains or stores a program used by an instruction execution system, apparatus, or device to function or operate. In the present application, the computer-readable signal medium can include a computer-readable storage medium or any computer-readable medium that propagates or transmits a program used by an instruction execution system, apparatus, or device to function or operate. The program code contained in the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber, RF, etc., or any suitable combination of the foregoing.

[0143] The flow diagrams and block diagrams in the drawings are illustrations of possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0144] The units described in the embodiments of the present application can be implemented in the form of software, or can be implemented in the form of hardware. The described units can also be arranged in a processor, for example, a processor can be described as including a receiving unit, a positioning unit, a joint query statement generating unit and an executing unit. In some cases, the names of these units do not constitute a limitation on the units themselves.

[0145] As another aspect, the present application also provides a computer readable medium, which can be included in the device described in the above embodiments, or can exist independently without being assembled into the device. The computer readable medium carries one or more programs, which, when executed by the device, cause the device to receive a data query request, obtain a corresponding instruction type identifier and data time effectiveness identifier, locate a target database table according to the instruction type identifier and the data time effectiveness identifier, and then determine the number of the target database table; in response to the number being multiple, obtain query mode information corresponding to the instruction type identifier, determine a query statement assembly order based on the query mode information, assemble the query statement and the target database table based on the query statement assembly order, to generate a joint query statement; execute the joint query statement, and then output the data obtained by the query. Thus, when data is queried, the range of the query is narrowed, and the data query is more accurate and faster.

[0146] According to the technical solutions of the embodiments of the present application, when data is queried, the range of the query can be narrowed, and the data query is more accurate and faster.

[0147] The above specific embodiments do not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made depending on design requirements and other factors. Any modification, equivalent replacement and improvement made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A data query method, characterized by, The method comprises: receiving a data query request, obtaining a corresponding instruction type identifier and data timeliness identifier, wherein the instruction type refers to the type of hint instruction, including routing instruction and control instruction, the routing instruction and the control instruction have their own corresponding database tables respectively, which are used to query data in their own corresponding database tables or store data in the corresponding database tables, and the data timeliness identifier is used to represent whether the data to be queried is real-time data or historical data; locating a target database table according to the instruction type identifier and the data timeliness identifier, and then determining the number of the target database table; in response to the number being multiple, obtaining query mode information corresponding to the instruction type identifier, determining query statement assembly order based on the query mode information, assembling the query statement and the target database table based on the query statement assembly order, and generating a joint query statement; executing the joint query statement, and then outputting the data obtained by querying.

2. The method of claim 1, wherein, Before the step of locating a target database table according to the instruction type identifier and the data timeliness identifier, the method further comprises: obtaining full data and deployment location identifier; based on the deployment location identifier, the full data is divided to generate sharded data; determining the business identifier corresponding to the sharded data, and then based on the business identifier, the sharded data is divided into buckets to obtain bucketed data; synchronizing the bucketed data to a database table.

3. The method of claim 2, wherein, The step of locating a target database table according to the instruction type identifier and the data timeliness identifier comprises: from the synchronized database table, one or more database tables corresponding to the instruction type identifier and the data timeliness identifier are determined, and then the one or more database tables are determined as the target database table.

4. The method of claim 2, wherein, The step of dividing the sharded data into buckets based on the business identifier to obtain bucketed data comprises: determining the field name corresponding to each sharded data, and determining the timestamp of the sharded data with the same field name; determining new data and old data in the sharded data with the same field name according to the timestamp; replacing the old data with the new data, and then obtaining updated sharded data; placing the data corresponding to the same business identifier in the updated sharded data into the same data bucket to obtain bucketed data.

5. The method of claim 4, wherein, The step of placing the data corresponding to the same business identifier in the updated sharded data into the same data bucket to obtain bucketed data comprises: performing hash calculation on the data corresponding to the same business identifier in the updated sharded data, and then storing the result data of hash calculation into the same data bucket to obtain bucketed data.

6. The method of claim 2, wherein, The step of synchronizing the bucketed data to a database table comprises: determining the data timeliness identifier corresponding to the bucketed data; synchronizing the bucketed data to the database table corresponding to the data timeliness identifier corresponding to the bucketed data.

7. A data query apparatus, characterized by comprising: The method comprises: The receiving unit is configured to receive a data query request, obtain a corresponding instruction type identifier and a data timeliness identifier, wherein the instruction type refers to the type of hint instruction, including a routing instruction and a control instruction, the routing instruction and the control instruction have respective corresponding database tables respectively, and the respective corresponding database tables are used to query data in the respective corresponding database tables or store data in the respective corresponding database tables, and the data timeliness identifier is used to represent whether the data to be queried is real-time data or historical data. The positioning unit is configured to locate a target database table according to the instruction type identifier and the data timeliness identifier, and then determine the number of the target database table. The joint query statement generation unit is configured to, in response to the number being multiple, obtain query mode information corresponding to the instruction type identifier, determine a query statement assembly sequence based on the query mode information, assemble the query statement and the target database table based on the query statement assembly sequence, and generate a joint query statement. The execution unit is configured to execute the joint query statement, and then output data obtained by querying.

8. The apparatus of claim 7, wherein, The apparatus further includes a synchronization unit configured to: obtain full-amount data and a deployment location identifier; split the full-amount data based on the deployment location identifier to generate sharded data; determine a business identifier corresponding to the sharded data, and then perform bucketing on the sharded data based on the business identifier to obtain bucketed data; synchronize the bucketed data to a database table.

9. The apparatus of claim 8, wherein, The positioning unit is further configured to: determine one or more database tables corresponding to the instruction type identifier and the data timeliness identifier from the synchronized database table, and then determine the one or more database tables as the target database table.

10. A data query electronic device, comprising: comprise: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-6.

11. A computer readable medium having stored thereon a computer program, characterized in that The program is executed by the processor to implement the method of any one of claims 1-6.

Images