Offline data real-time query method and device, equipment and storage medium

Abstract

The application discloses an offline data real-time query method and device, equipment and a storage medium, and belongs to the field of data processing. The method comprises the following steps: a scheduler determines a to-be-executed task according to task metadata in a metadata database, and sends the to-be-executed task to a task executor; the task metadata describes the requirement rules of a task for acquiring offline data by a business system; the task executor allocates the to-be-executed task to a task processor indicated by the task metadata; the task processor connects an offline database according to a real-time parameter variable value and the task metadata, and writes offline data meeting the task metadata in the offline database into a business database; and a service processor connects the business database according to a data query service request and service metadata, executes a data query statement, and outputs a query result; the service metadata describes the processing rules of the data query service on the data of the business database. According to the embodiment of the application, the research and development efficiency of the offline data real-time query of the business system can be improved.

Description

Technical Field

[0001] This application belongs to the field of data processing, and in particular relates to a method, apparatus, device and storage medium for real-time offline data query. Background Technology

[0002] Offline databases store a large amount of offline data, which can be used by business systems for secondary development and as a service provided externally. However, the volume of offline data is enormous, and processing it is slow, making it difficult to meet the requirements of real-time business queries. To retrieve data from offline databases, business systems can write code to interface with the offline database. After obtaining the data, it is loaded into the business system's business database, and then a real-time data query service is developed based on the business database. However, since different business systems have different business requirements, each requires independent manual development of code to interface with the offline database. The data query service also needs to handle a series of development logics such as input, database query, and result encapsulation, resulting in low development efficiency and excessively long development cycles for real-time offline data queries for various business systems. Summary of the Invention

[0003] This application provides a method, apparatus, device, and storage medium for real-time offline data query, which can improve the development efficiency of real-time offline data query in business systems.

[0004] In a first aspect, embodiments of this application provide a method for real-time offline data query, comprising: determining a task to be executed by a scheduler based on task metadata in a metadata database, and sending the task to be executed to a task executor, wherein the task metadata describes the requirement rules for a business system to obtain offline data; allocating the task to be executed to a task processor indicated by the task metadata by the task executor; obtaining real-time parameter variable values ​​by the task processor, connecting to an offline database based on the real-time parameter variable values ​​and the task metadata, and writing offline data conforming to the task metadata from the offline database into the business database of the business system; and connecting to the business database by a service processor based on a received data query service request and service metadata corresponding to the data query service request, executing the data query statement in the data query service request, and outputting the query result, wherein the service metadata describes the processing rules of the data query service for the data in the business database.

[0005] Secondly, embodiments of this application provide an offline data real-time query device, comprising: a scheduling module, configured to determine a task to be executed based on task metadata in a metadata database, and send the task to be executed to a task execution module, wherein the task metadata describes the requirement rules for a business system to obtain offline data; a task execution module, configured to assign the task to be executed to a task processing module indicated by the task metadata; a task processing module, configured to obtain real-time parameter variable values, connect to an offline database based on the real-time parameter variable values ​​and task metadata, and write offline data conforming to the task metadata from the offline database into the business database of the business system; and a service processing module, configured to connect to the business database based on a received data query service request and service metadata corresponding to the data query service request, execute the data query statement in the data query service request, and output the query result, wherein the service metadata describes the processing rules of the data query service on the data in the business database.

[0006] Thirdly, embodiments of this application provide an electronic device, including: a processor and a memory storing computer program instructions; the processor executes the computer program instructions to implement the offline data real-time query method of the first aspect.

[0007] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the offline data real-time query method of the first aspect.

[0008] This application provides a method, apparatus, device, and storage medium for real-time offline data query. A scheduler can trigger tasks to be executed to a task executor based on pre-configured task metadata. The task executor can then send the tasks to be executed to a task processor indicated by the task metadata. The task processor can connect to an offline database based on the task metadata, retrieve corresponding offline data, and write it to the business database of the business system. After the offline data is written to the business database, a service processor can receive data query services. Based on the service metadata corresponding to the data query service request, it connects to the business database, queries data using data query statements, and outputs the query results. The task metadata describes the requirements and rules for the business system to retrieve offline data; different tasks from different business systems can be described using task metadata with a unified format but different content. The service metadata describes the processing rules of the data query service on the data in the business database; different data query services provided by different business systems can be described using service metadata with a unified format but different content. By abstracting the differences in tasks across different business systems into configurable task metadata and the differences in data query services across different business systems into configurable service metadata, the process of applying offline data from offline databases to real-time queries can be completed based on task metadata and service metadata. This eliminates the need for manually developing separate code programs to interface with offline databases for different business systems, as well as for manually developing a series of code programs for input, query, and result encapsulation for data query services of different business systems. This improves the development efficiency of real-time offline data queries for business systems and shortens the development cycle. Attached Figure Description

[0009] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 A schematic diagram of the architecture of an example of an offline data real-time query system provided in an embodiment of this application;

[0011] Figure 2 A flowchart illustrating an embodiment of this application of a method for real-time offline data query;

[0012] Figure 3 This is a schematic diagram of the structure of an offline data real-time query method provided in another embodiment of this application;

[0013] Figure 4 This is a schematic diagram of the structure of an offline data real-time query device provided in an embodiment of this application;

[0014] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0015] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples. It should be noted that the acquisition, storage, use, and processing of information and data in the embodiments of this application are all authorized by users or relevant organizations and comply with the relevant provisions of national laws and regulations.

[0016] Offline databases store vast amounts of offline data, which can be used by business systems for secondary development and as a service provided externally. However, the sheer volume of offline data makes processing slow and difficult to meet the demands of real-time business queries. To retrieve data from offline databases, business systems can write code to interface with them, load the data into their own business database, and then develop real-time data query services based on that database. However, because different business systems have different functionalities, each requires independent manual development of code to interface with the offline database. The data query service also needs to handle a series of development logics, including input processing, database queries, and result encapsulation, resulting in low development efficiency and excessively long development cycles for real-time offline data queries across various business systems.

[0017] This application provides a method, apparatus, device, and storage medium for real-time offline data query. It configures the processing rules for offline data requirements of business system tasks as task metadata and the processing rules for data query service requirements as service metadata. This abstracts the differences in tasks across different business systems into configurable task metadata and the differences in data query service requirements across different business systems into configurable service metadata. Based on the task metadata and service metadata, the process of applying offline data from an offline database to real-time querying can be completed. This eliminates the need for manually developing separate code programs to interface with offline databases for different business systems, and also eliminates the need for manually developing a series of code programs for input, query, and result encapsulation for different data query services. This improves the development efficiency of real-time offline data query for business systems and shortens the development cycle.

[0018] The offline data real-time query method in this application embodiment can be executed by an offline data real-time query device, platform, system, etc. Furthermore, the offline data real-time query device, platform, system, etc., can be divided into multiple parts according to function, and these multiple parts work together to implement the offline data real-time query method in this application embodiment. For ease of understanding, this paper first takes the execution of the offline data real-time query method by an offline data real-time query system as an example to briefly explain the part of the offline data real-time query device used to work together to implement the offline data real-time query method in this application embodiment. The offline data real-time query system here can also be implemented as an offline data real-time query device, platform, etc., and is not limited here. Figure 1 A schematic diagram of the architecture of an example of an offline data real-time query system provided in an embodiment of this application is shown below. Figure 1 As shown, the offline data real-time query system may include a metadata configurator 11, a metadata database 12, a scheduler 13, a task executor 14, a task processor 15, a business database 16, and a service processor 17.

[0019] Metadata configurator 11 can be used to configure task metadata and service metadata. Both task metadata and service metadata belong to metadata, which describes data and can be used to support various functions. Task metadata can be used to describe the requirements rules for tasks that the business system needs to obtain offline data. Service metadata can be used to describe the processing rules of the data query service. Metadata configurator 11 can store the configured task metadata and service metadata in metadata database 12, that is, metadata database 12 is used to store task metadata and service metadata. Scheduler 13 can load task metadata from metadata database 12, thereby scheduling the corresponding task to task executor 14, and task executor 14 will assign the task to the corresponding task processor 15. Task processor 15 executes the task, interfaces with the offline database, performs data statistics, and writes data that meets the requirements rules of the business system for offline data into business database 16. Service processor 17 loads service metadata from metadata database 12, queries business database 16 according to the received data query service request and the corresponding service metadata, and obtains the query results obtained from querying business database 16.

[0020] The first aspect of this application provides a method for real-time offline data query, which can be applied to scenarios where business systems connect to offline databases to achieve real-time queries. This method can be executed by an offline real-time data query device, platform, system, etc. Figure 2 A flowchart of an offline data real-time query method provided in an embodiment of this application is shown below. Figure 2 As shown, the offline data real-time query method may include steps S201 to S204.

[0021] In step S201, the scheduler determines the task to be executed based on the task metadata in the metadata database and sends the task to be executed to the task executor.

[0022] The metadata database stores task metadata. Task metadata describes the requirements and rules for tasks that the business system uses to acquire offline data. Task metadata can be described through multiple items. In some examples, task metadata may include, but is not limited to, task identifier, task execution cycle, task execution content, task parameters, task processor, and input statement. In some cases, task metadata may also include, but is not limited to, database cleanup statement and / or task type. The task identifier item describes the task identifier, which may include the task name, task number, etc. The task execution cycle item describes the task's execution cycle. The task execution content item describes the task's execution content. The task parameters item describes the task's parameters. The task processor item describes the task processor that should execute the task. The input statement item describes the write statement used to write the offline data acquired by the task into the business database of the business system. The database cleanup statement item describes the cleanup statements related to the task that clean up at least a portion of the data in the business database. The task type item describes the task type. Task metadata has a uniform format but can vary in content. That is, different task metadata may contain the same items, but the values ​​of those items can differ. For example, the execution cycle described in task metadata A1 may differ from the execution cycle described in task metadata A2. Task metadata can be used to retrieve offline data required by a business system from an offline database. Task metadata corresponds to specific tasks; different tasks may have different corresponding task metadata. The requirements for the business system to retrieve offline data, as described in the task metadata, can include the timing of task execution. The scheduler can determine the tasks to be executed based on the timing of task execution described in the task metadata in the metadata database. These tasks may include those requiring the retrieval of offline data at the current time, as described in the task metadata. The scheduler can then send the tasks to be executed to the task executor, which will execute subsequent steps.

[0023] In some examples, tasks have a task execution cycle, which can be configured in the task metadata. Correspondingly, the scheduler can determine the task execution cycle based on the task metadata, and thus periodically send the tasks to be executed to the task executor.

[0024] In step S202, the task to be executed is assigned to the task processor indicated by the task metadata by the task executor.

[0025] The task metadata describes the requirements and rules for a business system to acquire offline data, which may include the task processor for executing the task. The task executor can determine the task processor indicated by the task metadata based on the task metadata corresponding to the task to be executed, and assign the task to the task processor indicated by the task metadata for processing.

[0026] In some examples, the scheduler may send the task to be executed along with its corresponding task metadata to the task executor, enabling the task executor to obtain the corresponding task metadata and assign the task to the appropriate task processor. In other examples, upon receiving a task to be executed, the task executor may retrieve the task metadata corresponding to the task from a metadata database, thereby assigning the task to the appropriate task processor.

[0027] In step S203, the real-time parameter variable values ​​are obtained through the task processor. Based on the real-time parameter variable values ​​and task metadata, the system connects to the offline database and writes the offline data that matches the task metadata into the business database of the business system.

[0028] The task metadata describes the requirements and rules for the task that the business system needs to obtain offline data. These rules may include variable information. Since the task has a certain degree of real-time nature, the task metadata is pre-configured. It is difficult to pre-configure the variable information in the task metadata. When configuring the task metadata, the variable information in the task metadata can be configured to a default value or be configured to be empty. When the task processor processes the task to be executed, it obtains the real-time parameter variable values ​​and uses the real-time parameter variable values ​​to update the task metadata obtained by the task processor. This allows the data query interface of the offline database to be called based on the updated task metadata, and the offline data that meets the requirements and rules in the task metadata is written into the business database of the business system.

[0029] In step S204, the service processor connects to the business database based on the received data query service request and the service metadata corresponding to the data query service request, executes the data query statement in the data query service request, and outputs the query result.

[0030] A data query service request is used to request real-time data queries from the business system. This request can be user-input. The request may include a service identifier, which identifies the data query service. Service metadata corresponds to the data query service; different data query services may correspond to different service metadata. A service processor can preload service metadata and retrieve the service metadata corresponding to the data query service request from the loaded metadata. For example, a service processor can configure a service data pool, which is used to load service metadata, and the service processor can retrieve the service metadata corresponding to the data query service request from the service data pool. Alternatively, upon receiving a data query service request, the service processor can query and retrieve the service metadata corresponding to the data query service request from the metadata database. Service metadata describes the processing rules of the data query service on the business database. Service metadata can describe these rules through multiple items. In some examples, service metadata may include, but is not limited to, service identifier items, service execution content items, input parameter items, output type items, and output parameter items. The service identifier item describes the identifier of the data query service and may include the service name and service number. The service execution content item describes the execution content of the data query service. The input parameter item describes the input parameters of the data query service. The output type item describes the output type of the data query service. The output parameter item describes the output parameters of the data query service. Items in the service metadata can also influence each other. For example, if the output type item in the service metadata describes an object as a complex entity, multiple execution content items can be configured in that service metadata to correspond to the object. The format of service metadata is uniform, but the content can differ. That is, different service metadata may contain the same items, but the values ​​of the items in different service metadata may differ. For example, the output type item in service metadata A3 describes a different output type than the output type item in service metadata A4.

[0031] Based on service metadata, real-time querying of offline data written to the business database can be achieved. The service processor can connect to the business database according to the content described in the service metadata corresponding to the data query service request, and execute specific data query services using data query statements to retrieve the corresponding data from the business database. The data query statements can be implemented as Structured Query Language (SQL) statements, but are not limited to them. A data query service request may include one or more data query statements, without limitation. After retrieving the corresponding data from the business database, the service processor can further assemble the retrieved data according to the output-related content in the data query service's data processing rules for the business database described in the service metadata, ensuring that the query results satisfy the data query service's data processing rules for the business database as described in the service metadata.

[0032] In this embodiment, a scheduler can trigger a task to be executed to a task executor based on pre-configured task metadata. The task executor can then send the task to be executed to a task processor indicated by the task metadata. The task processor can connect to an offline database based on the task metadata, retrieve corresponding offline data, and write it to the business database of the business system. After the offline data is written to the business database, a service processor can receive a data query service. Based on the service metadata corresponding to the data query service request, it connects to the business database, queries the data using a data query statement, and outputs the query results. The task metadata describes the requirements and rules for the business system to retrieve offline data; different tasks from different business systems can be described using task metadata with a unified format but different content. The service metadata describes the processing rules of the data query service on the data in the business database; different data query services provided by different business systems can be described using service metadata with a unified format but different content. By abstracting the differences in tasks across different business systems into configurable task metadata and the differences in data query services across different business systems into configurable service metadata, the process of applying offline data from offline databases to real-time queries can be completed based on task metadata and service metadata. This eliminates the need for manually developing separate code programs to interface with offline databases for different business systems, as well as for manually developing a series of code programs for input, query, and result encapsulation for data query services of different business systems. This improves the development efficiency of real-time offline data queries for business systems and shortens the development cycle.

[0033] In some embodiments, task metadata and service metadata can be configured through the metadata configurator before step S201, and the task metadata and service metadata can be stored in the metadata database so that they can be directly obtained from the metadata database during the real-time offline data query process without repeated configuration, thereby further improving the development efficiency of the business system's real-time offline data query.

[0034] In some embodiments, task metadata may include task execution cycle items, task processor items, task execution content items, and database entry statement items, while service metadata may include service identifier items, input parameter items, and service execution content items. Figure 3 This is a schematic diagram of the structure of an offline data real-time query method provided in another embodiment of this application. Figure 3 and Figure 2 The difference is that, Figure 2 Step S201 can be further refined as follows: Figure 3 Step S2011 in the process, Figure 2 Step S202 can be further refined as follows: Figure 3 Steps S2021 and S2022 in the process, Figure 2 Step S203 can be further refined as follows: Figure 3 Steps S2031 and S2032 in the process, Figure 2 Step S204 can be further refined as follows: Figure 3 Steps S2041 and S2043 in the process.

[0035] In step S2011, the scheduler queries the task execution cycle item of the task metadata in the metadata database, and sends the task corresponding to the task metadata of the task execution cycle indicated by the task execution cycle item at the current time as the task to be executed to the task executor.

[0036] The scheduler can query the task execution cycle items in the metadata of each task in the metadata database to obtain the task execution cycle corresponding to the task metadata. If the task execution cycle indicated by the task execution cycle item in the task metadata reaches the current time, the task corresponding to the task metadata can be identified as a task to be executed and sent to the task executor.

[0037] In step S2021, the task metadata corresponding to the task to be executed is obtained through the task executor.

[0038] The task executor can obtain task metadata corresponding to the task to be executed from the scheduler, or it can obtain task metadata corresponding to the task to be executed from the source database. Specifically, the task metadata corresponding to the task to be executed can be determined by the task identifier of the task to be executed.

[0039] In step S2022, the task executor assigns the task to be executed to the task processor indicated by the task processor item in the task metadata corresponding to the task to be executed.

[0040] The task processor entry in the task metadata of a task to be executed indicates the task processor that will handle the task. The task executor can read the task processor entry in the task metadata corresponding to the task to be executed, determine the task processor that will handle the task, and assign the task to that task processor for processing.

[0041] In step S2031, the real-time parameter variable values ​​are obtained through the task processor, and the variable values ​​in the task execution content items in the task metadata are replaced with the real-time parameter variable values ​​to obtain the updated task metadata.

[0042] The task to be executed has a certain degree of real-time requirement. Real-time parameter variable values ​​can include values ​​of parameters that reflect the real-time nature of the task, such as execution time and execution session. The task metadata is pre-configured. To ensure the real-time retrieval of offline data from the offline database by the task processor based on the task metadata, the variable values ​​in the task execution content items of the task metadata can be replaced with real-time parameter variable values ​​to obtain updated task metadata.

[0043] In step S2032, the task processor connects to the offline database based on the updated task metadata, obtains the result set data, and writes the result set data into the business database according to the input statement indicated by the input statement item in the task metadata.

[0044] The task processor has its own operational logic. Based on updated task metadata, it generates offline data statistical statements. The task processor then calls the offline database's query interface to connect and input the offline data statistical statements. These statements run in the offline database, retrieving the offline data required for the task to be executed. The offline data retrieved forms a result set. This result set includes offline data from the offline database that matches the task metadata; in other words, it includes the offline data obtained from the execution of the offline data statistical statements. After obtaining the result set, the task processor writes it to the corresponding business database according to the input statements specified in the task metadata.

[0045] In some examples, the result set data is first written from the offline database to the local Hadoop Distributed File System (Hadoop Distributed File System) of the offline database, and then retrieved from the Hadoop Distributed File System via a task processor. That is, retrieving the result set data in step S2032 can be specifically refined as: retrieving the result set data from the Hadoop Distributed File System via a task processor. Generally, the offline database contains a large amount of offline data, and the result set data also includes a large amount of offline data. The transfer speed of the result set data between the offline database and the Hadoop Distributed File System is very fast, and since the offline database and the Hadoop Distributed File System are located on the same local machine, the stability of data transfer between the offline database and the Hadoop Distributed File System is also high. Writing the result set data from the Hadoop Distributed File System to the business database is the underlying data transfer, and its data transfer speed is also very fast. Using the Hadoop Distributed File System as a relay node for result set data transfer can accelerate the speed of result set data copying and improve the stability of result set data copying.

[0046] In other examples, streaming data statistics can be used to directly read data from the data stream in the offline database. The step S2032 above, obtaining the result set data, can be further refined as follows: The result set data is obtained from the data stream generated by the offline database through a task processor. The result set data can be transferred without passing through the Hadoop Distributed File System.

[0047] In some embodiments, the task metadata may include cleanup statement items, which may indicate cleanup statements used to clean up at least a portion of the data in the business database. In the event of an anomaly when offline data matching the task metadata is written from the offline database to the business database, the cleanup statement indicated by the cleanup statement item in the task metadata is executed by the task processor to clean up at least a portion of the data in the business database. Anomalies in writing offline data to the business database can be of various types, with different causes and corresponding different measures. For example, if the anomaly is caused by poor communication during the write operation, the cleanup statement indicated by the cleanup statement item in the task metadata can be executed by the task processor to clear the corresponding data in the business database. Then, the insertion statement indicated by the insertion statement item in the task metadata can be re-executed to write the offline data matching the task metadata back to the business database. As another example, if the anomaly is caused by an error in the offline database's statistics of offline data matching the task metadata, the cleanup statement indicated by the cleanup statement item in the task metadata can be executed by the task processor to clear the corresponding data in the business database. Then, the offline database can be reconnected to obtain the offline data matching the task metadata and write it back to the business database.

[0048] In step S2041, the service processor searches the service data pool for service metadata whose service identifier, as indicated by the service identifier item, matches the service identifier in the data query service request.

[0049] The service processor can be configured with a service data pool, which includes service metadata. The service metadata in the service data pool comes from a metadata database. Service metadata can be pre-loaded from the metadata database into the service data pool to speed up the search for service metadata.

[0050] Service metadata includes a service identifier item. Service metadata whose service identifier matches the service identifier in the data query service request is the service metadata corresponding to the data query service request. In some examples, service metadata may also include a service version number item, which describes the version number of the data query service. The version number can also be carried in the data query service request. The service metadata that matches the service identifier indicated by the service identifier item in the service query service request and whose version number indicated by the service version number item matches the version number in the data query service request is identified as the service metadata corresponding to the data query service request.

[0051] In some examples, if the service metadata indicating the service identifier item is not found in the service data pool and the service identifier in the data query service request is consistent with the service identifier indicated by the service identifier item, it may be because the newly configured service metadata in the metadata database has not yet been loaded into the service data pool. The service processor can search in the metadata database. If the service metadata indicating the service identifier item is found in the metadata database and the service identifier in the data query service request is consistent with the service identifier indicated by the service identifier item, the service metadata is loaded into the service data pool, and subsequent steps are performed based on the service data.

[0052] In step S2042, the service processor verifies the validity of the request parameters in the data query service request according to the input parameter specifications indicated by the input parameter items in the service metadata.

[0053] The data query service request carries request parameters, which can be verified to confirm whether these parameters conform to the input parameter specifications indicated by the input parameter items in the service metadata determined in step S2041, thereby verifying the validity of the request parameters in the data query service request. If the request parameters conform to the input parameter specifications indicated by the input parameter items in the service metadata, the validity verification passes; if the request parameters do not conform to the input parameter specifications indicated by the input parameter items in the service metadata, the validity verification fails.

[0054] In step S2043, if the validity verification is passed, the service processor replaces the variable values ​​in the service execution content item in the service metadata with the request parameters, connects to the business database according to the updated service metadata, executes the data query statement, and outputs the query results.

[0055] Data query services also have a certain degree of real-time requirement, which can be reflected in the request parameters of the data query service request. While service metadata is pre-configured, the real-time variables within the service metadata items are difficult to pre-configure. Therefore, upon receiving a data query service request, the variable values ​​in the service execution content items of the service metadata can be replaced with request parameters that reflect the real-time nature of the data query service, resulting in updated service metadata.

[0056] Each service processor has its own operational logic. It can execute data query statements, process the data retrieved from the business database according to its own logic, and obtain the query results. The query results conform to the service metadata.

[0057] In some embodiments, service metadata includes output type items, and correspondingly, the query results output by the service processor conform to the output type items in the service metadata.

[0058] In some examples, if the output type field indicates that the output type includes any of the following: string, numeric, or boolean, the query result includes one record from the business database corresponding to the data query statement. String, numeric, and boolean types are all simple data types. If the output type field indicates that the output type is a simple data type, the queried data is one record from the business database (i.e., selectOne), which can be assembled into the query result.

[0059] In some examples, where the output type indicated by the output type item includes an array of objects, the query results include an N-dimensional table or multi-way tree containing records from the business database corresponding to the data query statement, where N is an integer greater than or equal to 2. An array of objects is an array of objects. If the output type includes an array of objects, the data retrieved from the business database may include multiple records (i.e., a selectList). All records retrieved from the business database may have more than two attributes, which cannot be represented by a simple array. Therefore, the records retrieved from the business database can be assembled into a multi-dimensional table or multi-way tree as the query results.

[0060] In some examples, where the output type indicated in the output type field includes a numeric array or a string array, the query results include an array containing records from the business database corresponding to the data query statement. A numeric array is an array of numbers, and a string array is an array of strings. The output type includes either a numeric array or a string array. The data retrieved from the business database may include multiple records (i.e., a selectList), which can be represented by an array. The data retrieved from the business database can be assembled into an array as the query results.

[0061] In some examples, when the output type indicated by the output type item includes objects and the number of data query statements is greater than two, the query results include a tree object containing records from the business database corresponding to each data query statement. Objects are relatively complex entities, and when the output type includes objects, multiple data query statements can be supported. When there are multiple data query statements, they are executed separately. When executing each data query statement, two scenarios may occur: First, if the data query statement has only one output parameter, recursion can be used, calling the processing logic for output types including string, number, and boolean types to obtain the records found in the business database for that data query statement. Second, if the data query statement has two or more output parameters, the data found in the business database for each data query statement can be assembled into an object. Finally, the data found by each data query statement is merged to form a tree object.

[0062] For example, Table 1 below shows the query results for a tree structure:

[0063] Table 1

[0064]

[0065]

[0066] In this table, w1 and w2 are child nodes of w, and e1 and e2 are child nodes of e. The operations in Table 1 are not shown; in practice, operations may include details, marking, adding child nodes, deletion, etc., which will not be elaborated upon here.

[0067] By configuring the output type items in the service metadata, various requirements for the output format of query results can be met by the data query service. Furthermore, the task processor can automatically assemble query results that meet the requirements of the current data query service based on the service metadata, resulting in high availability and applicability to various application scenarios. Moreover, the process of executing the data query service can be configured through the service metadata, satisfying data query services across different business system architectures and offering enhanced scalability.

[0068] The second aspect of this application provides an offline data real-time query device. Figure 4 This is a schematic diagram of the structure of an offline data real-time query device provided in an embodiment of this application, as shown below. Figure 4 As shown, the offline data real-time query device 300 may include a scheduling module 301, a task execution module 302, a task processing module 303, and a service processing module 304.

[0069] The scheduling module 301 can determine the tasks to be executed based on the task metadata in the metadata database and send the tasks to be executed to the task execution module 302. The task metadata is used to describe the requirements and rules for the task of the business system to obtain offline data.

[0070] The task execution module 302 can be used to assign tasks to be executed to the task processing module 303 indicated by the task metadata.

[0071] The task processing module 303 can be used to obtain real-time parameter variable values, connect to the offline database based on the real-time parameter variable values ​​and task metadata, and write the offline data that matches the task metadata from the offline database into the business database of the business system.

[0072] The service processing module 304 can connect to the business database, execute the data query statements in the data query service request, and output the query results based on the received data query service request and the corresponding service metadata. The service metadata describes the processing rules of the data query service on the data in the business database.

[0073] In this embodiment, a scheduler can trigger a task to be executed to a task executor based on pre-configured task metadata. The task executor can then send the task to be executed to a task processor indicated by the task metadata. The task processor can connect to an offline database based on the task metadata, retrieve corresponding offline data, and write it to the business database of the business system. After the offline data is written to the business database, a service processor can receive a data query service. Based on the service metadata corresponding to the data query service request, it connects to the business database, queries the data using a data query statement, and outputs the query results. The task metadata describes the requirements and rules for the business system to retrieve offline data; different tasks from different business systems can be described using task metadata with a unified format but different content. The service metadata describes the processing rules of the data query service on the data in the business database; different data query services provided by different business systems can be described using service metadata with a unified format but different content. By abstracting the differences in tasks across different business systems into configurable task metadata and the differences in data query services across different business systems into configurable service metadata, the process of applying offline data from offline databases to real-time queries can be completed based on task metadata and service metadata. This eliminates the need for manually developing separate code programs to interface with offline databases for different business systems, as well as for manually developing a series of code programs for input, query, and result encapsulation for data query services of different business systems. This improves the development efficiency of real-time offline data queries for business systems and shortens the development cycle.

[0074] In some embodiments, the offline data real-time query device 300 described above may further include a metadata configuration module. The metadata configuration module can be used to: configure task metadata and service metadata through a metadata configurator, and store the task metadata and service metadata in a metadata database.

[0075] Task metadata may include task identifier, task execution cycle, task execution content, task parameters, task processor, and data entry statement.

[0076] Service metadata includes service identifier items, service execution content items, input parameter items, output type items, and output parameter items.

[0077] In some examples, task metadata includes task execution cycle items. The scheduling module 301 can specifically be used to: query the task execution cycle items in the task metadata of the metadata database, and send the task corresponding to the task metadata whose task execution cycle, as indicated by the task execution cycle item, reaches the current time as a task to be executed to the task execution module 302.

[0078] In some examples, the task metadata includes a task processor item. The task execution module 302 may be specifically used to: obtain the task metadata corresponding to the task to be executed; and assign the task to be executed to the task processing module 303 indicated by the task processor item in the task metadata corresponding to the task to be executed.

[0079] In some examples, task metadata includes task execution content items and database entry statement items. Task processing module 303 can be specifically used to: obtain real-time parameter variable values; replace the variable values ​​in the task execution content items of the task metadata with the real-time parameter variable values ​​to obtain updated task metadata; connect to the offline database based on the updated task metadata, obtain result set data, and write the result set data to the business database according to the database entry statement indicated by the database entry statement item in the task metadata. The result set data includes offline data in the offline database that matches the task metadata corresponding to the task to be executed.

[0080] In some examples, result set data is written from the offline database to the local Hadoop Distributed File System (HDFS) of the offline database. Task processing module 303 can specifically be used to retrieve result set data from the Hadoop Distributed File System via a task processor.

[0081] In some examples, task processing module 303 may be specifically used to: retrieve result set data from a data stream generated by an offline database via a task processor.

[0082] In some examples, the task metadata includes cleanup statement items. The task processing module 303 can also be used to: in the event of an anomaly in writing offline data conforming to the task metadata into the business database, execute cleanup statements indicated by the cleanup statement items in the task metadata to clean up at least a portion of the data in the business database.

[0083] In some examples, service metadata includes a service identifier item, input parameter items, and service execution content items. Service processing module 304 is configured with a service data pool, which includes service metadata. Specifically, service processing module 304 can be used to: search the service data pool for service metadata whose service identifier, as indicated by the service identifier item, matches the service identifier in the data query service request; the service metadata in the service data pool comes from a metadata database; validate the request parameters in the data query service request according to the input parameter specifications indicated by the input parameter items in the service metadata; if the validation passes, replace the variable values ​​in the service execution content items of the service metadata with the request parameters, connect to the business database based on the updated service metadata, execute the data query statement, and output the query results.

[0084] In some embodiments, service metadata includes output type items.

[0085] If the output type specified in the output type item includes any one of string type, numeric type, or boolean type, the query result includes one record in the business database corresponding to the data query statement.

[0086] If the output type indicated in the output type item includes an array of objects, the query results include an N-dimensional table or multi-way tree containing records in the business database corresponding to the data query statement, where N is an integer greater than or equal to 2.

[0087] If the output type indicated in the output type field includes a numeric array or a string array, the query results include an array containing records from the business database that correspond to the data query statement.

[0088] If the output type indicated by the output type item includes objects and the number of data query statements is greater than two, the query results include a tree object containing records in the business database corresponding to each data query statement.

[0089] A third aspect of this application also provides an electronic device. Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 5 As shown, the electronic device 400 includes a memory 401, a processor 402, and a computer program stored in the memory 401 and executable on the processor 402.

[0090] In some examples, the processor 402 described above may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits that may be configured to implement the embodiments of this application.

[0091] Memory 401 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the offline data real-time query method according to embodiments of this application.

[0092] The processor 402 reads the executable program code stored in the memory 401 to run the computer program corresponding to the executable program code, so as to implement the offline data real-time query method in the above embodiment.

[0093] In some examples, electronic device 400 may also include a communication interface 403 and a bus 404. For example, Figure 5 As shown, the memory 401, processor 402, and communication interface 403 are connected through bus 404 and complete communication with each other.

[0094] The communication interface 403 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application. Input devices and / or output devices can also be connected through the communication interface 403.

[0095] Bus 404 includes hardware, software, or both, that couples components of electronic device 400 together. For example, and not limitingly, bus 404 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-E) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 404 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application considers any suitable bus or interconnection.

[0096] A fourth aspect of this application also provides a computer-readable storage medium storing computer program instructions. When these computer program instructions are executed by a processor, they can implement the offline data real-time query method described in the above embodiments and achieve the same technical effect. To avoid repetition, further details are omitted here. The aforementioned computer-readable storage medium may include non-transitory computer-readable storage media, such as read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, etc., and is not limited thereto.

[0097] This application provides a computer program product. When the instructions in the computer program product are executed by the processor of an electronic device, the electronic device performs the offline data real-time query method described in the above embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0098] It should be clarified that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. For the device embodiments, equipment embodiments, computer-readable storage medium embodiments, and computer program product embodiments, the relevant parts can be referred to the description section of the method embodiments. This application is not limited to the specific steps and structures described above and shown in the figures. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application. Furthermore, for the sake of brevity, detailed descriptions of known methods and techniques are omitted here.

[0099] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0100] Those skilled in the art will understand that the above embodiments are exemplary and not restrictive. Different technical features appearing in different embodiments can be combined to achieve beneficial effects. Based on a study of the drawings, specification, and claims, those skilled in the art should be able to understand and implement other variations of the disclosed embodiments. In the claims, the term "comprising" does not exclude other means or steps; the quantifier "a" does not exclude a plurality; the terms "first" and "second" are used to identify names and not to indicate any particular order. No reference numerals in the claims should be construed as limiting the scope of protection. The functionality of multiple parts appearing in the claims can be implemented by a single hardware or software module. The appearance of certain technical features in different dependent claims does not mean that these technical features cannot be combined to achieve beneficial effects.

Claims

1. A method for real-time offline data query, characterized in that, include: Configure task metadata and service metadata through the metadata configurator, and store task metadata and service metadata in the metadata database. The different tasks of obtaining offline data in different business systems are abstracted into configurable task metadata, and the different data query services in different business systems are abstracted into configurable service metadata. The scheduler determines the tasks to be executed based on the task metadata in the metadata database and sends the tasks to be executed to the task executor. The task metadata is used to describe the requirements and rules of the business system to obtain offline data. The task executor assigns the task to be executed to the task processor indicated by the task metadata. The task processor obtains real-time parameter variable values, connects to the offline database based on the real-time parameter variable values ​​and task metadata, and writes offline data that conforms to the task metadata into the business database of the business system. The offline data that conforms to the task metadata in the offline database is statistically obtained by running an offline data statistical statement in the offline database. The offline data statistical statement is generated by the task processor according to its own running logic. The service processor connects to the business database based on the received data query service request and the service metadata corresponding to the data query service request, executes the data query statement in the data query service request, and outputs the query results. The service metadata is used to describe the data processing rules of the data query service on the data of the business database.

2. The method according to claim 1, characterized in that, Task metadata includes task execution cycle items, The step of determining the task to be executed by the scheduler based on the task metadata in the metadata database and sending the task to be executed to the task executor includes: The scheduler queries the task execution cycle item of the task metadata in the metadata database, and sends the task corresponding to the task metadata of the task execution cycle indicated by the task execution cycle item at the current time as the task to be executed to the task executor.

3. The method according to claim 1, characterized in that, Task metadata includes task processor items, The step of assigning the task to be executed to the task processor indicated by the task metadata through the task executor includes: The task executor obtains the task metadata corresponding to the task to be executed. The task executor assigns the task to be executed to the task processor indicated by the task processor item in the task metadata corresponding to the task to be executed.

4. The method according to claim 1, characterized in that, Task metadata includes task execution content items and database entry statement items. The step of obtaining real-time parameter variable values ​​through the task processor, connecting to an offline database based on the real-time parameter variable values ​​and task metadata, and writing offline data matching the task metadata from the offline database into the business database of the business system includes: The task processor obtains the real-time parameter variable values ​​and uses the real-time parameter variable values ​​to replace the variable values ​​in the task execution content items in the task metadata to obtain the updated task metadata. The task processor connects to the offline database based on the updated task metadata, obtains result set data, and writes the result set data into the business database according to the input statement indicated by the input statement item in the task metadata. The result set data includes offline data in the offline database that matches the task metadata corresponding to the task to be executed.

5. The method according to claim 4, characterized in that, The result set data is written from the offline database to the local Hadoop distributed file system of the offline database. The obtained result set data includes: The result set data is obtained from the Hadoop Distributed File System through the task processor.

6. The method according to claim 4, characterized in that, The obtained result set data includes: The task processor retrieves the result set data from the data stream generated by the offline database.

7. The method according to claim 1, characterized in that, Task metadata includes cleanup statement items. The method further includes: In the event of an anomaly in the offline database where offline data conforming to the task metadata is written to the business database, the task processor executes the cleanup statement indicated in the cleanup statement item in the task metadata to clean up at least a portion of the data in the business database.

8. The method according to claim 1, characterized in that, Service metadata includes service identifier items, input parameter items, and service execution content items. The service processor is configured with a service data pool, which includes service metadata. The process of connecting to the business database via the service processor based on the received data query service request and the service metadata corresponding to the data query service request, executing the data query statement in the data query service request, and outputting the query result includes: The service processor searches the service data pool for service metadata whose service identifier, as indicated by the service identifier item, matches the service identifier in the data query service request. The service metadata in the service data pool comes from the metadata database. The service processor verifies the validity of the request parameters in the data query service request according to the input parameter specifications indicated by the input parameter items in the service metadata. If the validity verification is passed, the service processor uses the request parameters to replace the variable values ​​in the service execution content item of the service metadata, connects to the business database according to the updated service metadata, executes the data query statement, and outputs the query result.

9. The method according to claim 1, characterized in that, Service metadata includes output type items; If the output type indicated by the output type item includes any one of string type, numeric type, or boolean type, the query result includes one record in the business database corresponding to the data query statement; When the output type indicated by the output type item includes an object array, the query result includes an N-dimensional table or multi-way tree containing records in the business database corresponding to the data query statement, where N is an integer greater than or equal to 2; If the output type indicated by the output type item includes a numeric array or a string array, the query result includes an array containing records in the business database corresponding to the data query statement; If the output type indicated by the output type item includes an object and the number of the data query statements is greater than two, the query result includes a tree object containing records in the business database corresponding to each of the data query statements.

10. The method according to any one of claims 1 to 9, characterized in that, in, Task metadata includes task identifier, task execution cycle, task execution content, task parameters, task processor, and data entry statement. Service metadata includes service identifier items, service execution content items, input parameter items, output type items, and output parameter items.

11. An offline data real-time query device, characterized in that, include: The metadata configuration module is used to configure task metadata and service metadata through the metadata configurator, and store the task metadata and service metadata in the metadata database. It abstracts the different tasks of obtaining offline data in different business systems into configurable task metadata, and abstracts the different data query services of different business systems into configurable service metadata. The scheduling module is used to determine the task to be executed based on the task metadata in the metadata database and send the task to be executed to the task execution module. The task metadata is used to describe the requirements and rules of the task for the business system to obtain offline data. The task execution module is used to assign the task to be executed to the task processing module indicated by the task metadata; The task processing module is used to obtain real-time parameter variable values, connect to the offline database based on the real-time parameter variable values ​​and task metadata, and write offline data that conforms to the task metadata into the business database of the business system. The offline data that conforms to the task metadata in the offline database is obtained by running an offline data statistical statement in the offline database. The offline data statistical statement is generated by the task processing module according to its own operating logic. The service processing module is used to connect to the business database based on the received data query service request and the service metadata corresponding to the data query service request, execute the data query statement in the data query service request, and output the query result. The service metadata is used to describe the processing rules of the data query service on the data of the business database.

12. An electronic device, characterized in that, include: Processor and memory storing computer program instructions; When the processor executes the computer program instructions, it implements the offline data real-time query method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the offline data real-time query method as described in any one of claims 1 to 10.

Images