Data query method and device, electronic equipment and storage medium

Abstract

The invention discloses a data query method and device, electronic equipment and a storage medium, and the method comprises the steps: receiving a data query request, and segmenting a main task of the data query request into two different subtasks according to the time consumption of a returned result; executing a sub-task in the main task, and storing an execution result and the total information of the transaction into a library; executing asynchronous sub-tasks in the main task by adopting a sub-thread, sending a request to a data source, obtaining a result file name, and taking the file name and a data source label as a key library; according to a monitoring result of the message queue, receiving message information obtained according to a file name and data source information package in the message; and querying a database according to the file name and the data source to obtain subtask information temporarily stored in the database, forwarding message information to a target server in which the transaction to be continued is located according to a stored server mark, and returning a query result. On one hand, multi-data source query is realized based on a task management mechanism, and on the other hand, the query efficiency is improved.

Description

Technical Field

[0001] The present application relates to the technical field of data query, and in particular to a data query method, device, electronic device, and storage medium. Background Art

[0002] In the era of big data, the amount of data is growing, and the sources of data are becoming more complex. Compared with the past, query services often involve massive amounts of data and have to face more complex and diverse data sources.

[0003] In order to bridge the differences between data sources and reduce the impact of unstable data sources, it is inevitable to increase resource consumption. This consumption is not only traditional resources such as memory and threads, but also a huge waste of time. These are the challenges faced by data queries from multiple data sources. Summary of the invention

[0004] The embodiments of the present application provide a data query method, device, electronic device, and storage medium to implement multi-data source query through a new task management mechanism.

[0005] The present application embodiment adopts the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides a data query method, wherein the method comprises:

[0007] Receive a data query request, and divide the main task of the data query request into two different subtasks according to the time taken to return the result, wherein the two different subtasks include a query task in which the data source can directly return the result or directly download the result, and another query task in which the requester is actively notified to obtain the result file by means of message notification and represented by an asynchronous task;

[0008] Execute the subtasks in the main task, and store the execution results and overall transaction information in the database. The overall transaction information at least includes the number of subtasks, the number of completed asynchronous subtasks, and the file name of the final result file. The execution results include the execution results of the total transaction;

[0009] A sub-thread is used to execute the asynchronous sub-task in the main task and send a request to the data source to obtain the result file name, and the file name plus the data source number is used as the key to store the result file in the library. The data source is also used to upload the result file to the enterprise data bus EDB, and the EDB is used to generate a corresponding message and put it into the message queue;

[0010] According to the monitoring result of the message queue, receiving the message information packaged according to the file name and data source information in the message;

[0011] The database is queried according to the file name and data source to obtain the subtask information temporarily stored in the database, and the message information is forwarded to the target server where the transaction to be continued is located according to the stored server mark, and the query result is returned.

[0012] In some embodiments, the target server includes a first server, and the method further includes:

[0013] Query the database through the first server to obtain subtask context information, download the file from the EDB, parse and process it, and then store it in the warehouse;

[0014] The overall transaction status is updated through the first server. If the first server executes itself without exception, its own status is changed to successful and the number of subtasks executed is increased by one;

[0015] If the first server itself executes abnormally, the overall transaction status is changed to failure.

[0016] In some embodiments, the method further comprises:

[0017] When the first server executes successfully, the number of subtasks is set to M, and the number of subtasks that have been executed is set to N, and the condition for completing the plus-one execution operation in the first server is that M is not equal to N+1;

[0018] If the update fails, the subtask is determined to be the last completed subtask, and the final file is generated based on the result file name and the total transaction context, and then uploaded to the EDB;

[0019] In the case of execution exception, the conditions for modifying the total transaction status to failure include that the original status of the total transaction is being executed. If the update fails, the current task is judged to be the first subtask that failed to execute, and an exception file is generated in combination with the result file name and uploaded to the EDB.

[0020] In some embodiments, before returning the query result, the method further includes:

[0021] Start a scheduled task, traverse the database at each fixed time point, and directly set the timed-out and unprocessed tasks as failed and generate an exception file.

[0022] In some embodiments, the method of using a sub-thread to execute the asynchronous sub-task in the main task and sending a request to a data source to obtain a result file name, and storing the result file in a database using the file name plus the data source number as a key includes:

[0023] Using the first thread to execute the subtasks in the main task, and storing the execution results and overall transaction information in a database;

[0024] If the first thread finishes waiting, sub-threads are established to respectively execute the asynchronous sub-tasks in the main task and send requests to the data source and terminate the first thread;

[0025] The result file name is obtained by sending a request to the data source, and the file name plus the data source number is used as the key to store it in the database, recording the execution status of the subtask, context information, corresponding file name, and server flag.

[0026] In some embodiments, the receiving, according to the monitoring result of the message queue, the message information packaged according to the file name and data source information in the message includes:

[0027] A message gateway is used to monitor messages in the message queue, and the file name and data source information in the message are packaged into a message according to the configuration of the apollo configuration center and sent to the second server, which belongs to the same cluster as the first server.

[0028] In some embodiments, the second server is further configured to:

[0029] The database is queried according to the file name and data source to obtain the subtask information temporarily stored in the database, and the message is forwarded to the target server where the transaction to be continued is located according to the stored server flag, and the subtask message status is updated to received, and the target server includes the first server.

[0030] In a second aspect, an embodiment of the present application further provides a data query device, wherein the device comprises:

[0031] A receiving module is used to receive a data query request, and divide the main task of the data query request into two different subtasks according to the time taken to return the result, wherein the two different subtasks include a query task in which the data source can directly return the result or directly download the result, and another query task in which the requester is actively notified to obtain the result file by means of message notification and represented by an asynchronous task;

[0032] An execution module, used to execute the subtasks in the main task, and store the execution results and overall information of the transaction in a database, wherein the overall information of the transaction at least includes the number of subtasks, the number of completed asynchronous subtasks, and the file name of the final result file, and the execution result includes the execution result of the total transaction;

[0033] The storage module is used to use a sub-thread to execute the asynchronous sub-task in the main task and send a request to the data source to obtain the result file name, and store the result file in the database using the file name plus the data source number as the key. The data source is also used to upload the result file to the enterprise data bus EDB, and the EDB is used to generate a corresponding message and put it into the message queue;

[0034] A monitoring module, used for receiving the message information packaged according to the file name and data source information in the message according to the monitoring result of the message queue;

[0035] The query module is used to query the database according to the file name and data source to obtain the subtask information temporarily stored in the database, and forward the message information to the target server where the transaction to be continued is located according to the stored server mark, and return the query result.

[0036] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor; and a memory arranged to store computer executable instructions, wherein the executable instructions, when executed, cause the processor to perform the above method.

[0037] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by an electronic device including multiple application programs, the electronic device executes the above method.

[0038] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: receiving a data query request, dividing the main task of the data query request into two different subtasks according to the time taken to return the result, the two different subtasks include a query task in which the data source can directly return the result or directly download the query task, and another query task that actively notifies the requesting party to obtain the result file by means of message notification and is represented by an asynchronous task. Those that take a short time and can obtain the result synchronously are classified into one category; those that take a long time and cannot obtain the result synchronously are classified into another category by message notification processing. Then, the subtasks in the main task are executed, and the execution results and the overall information of the transaction are stored in the database. The overall information of the transaction includes at least the number of subtasks, the number of completed asynchronous subtasks, and the file name of the final result file. The execution result includes the execution result of the total transaction. The life cycle of the asynchronous query subtask is paused until the name of the result file is obtained, and the context and the main transaction information are stored in the database for use in subsequent tasks. Further, a sub-thread is used to execute the asynchronous sub-task in the main task and send a request to the data source to obtain the result file name, and the file name plus the data source number is used as the key to enter the library. The data source is also used to upload the result file to the enterprise data bus EDB, and the EDB is used to generate a corresponding message and put it into the message queue. According to the monitoring result of the message queue, the message information obtained by packaging according to the file name and data source information in the message is received, and the database is queried according to the file name and data source to obtain the sub-task information temporarily stored in the database. At the same time, the message information is forwarded to the target server where the transaction to be continued is located according to the stored server mark, and the query result is returned. Thus, all sub-tasks and main transactions are scheduled by updating the status of sub-tasks and main transactions in the database. Only when all sub-tasks are executed without exception will the suspended main transaction be continued. Through the above method, the query task is segmented according to the data source, and the strategy mode is used to enable it to access multiple data sources more flexibly. At the same time, the mutual waiting between each sub-query is effectively avoided, and the waste of thread resources and memory resources is avoided to the greatest extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0040] Figure 1 A schematic diagram of a data query method in an embodiment of the present application;

[0041] Figure 2 This is a schematic diagram of the implementation principle of the data query method in the embodiment of the present application;

[0042] Figure 3This is a schematic diagram of the structure of a data query device in an embodiment of the present application;

[0043] Figure 4 This is a schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0045] How to bridge the gap between data sources, avoid the adverse effects of instability, and save computer resources and time as much as possible. In related technologies, the conventional approach is equipment upgrade and algorithm optimization. The former uses computing and memory resources to make up for the lack of time as much as possible, while the latter reduces resource consumption as much as possible from the storage method and aggregation algorithm. Another solution is to use elasticSearch, but the shortcomings of using elasticSearch are:

[0046] First of all, the consumption of storage resources and computing resources is itself a major pain point for multi-data source queries. Although there are many ways to improve resource utilization, the threshold for resource consumption itself is very high.

[0047] Secondly, the essence of elasticSearch technology is to solve problems through preprocessing, which is very advantageous for querying existing data. However, if you want to ensure the speed of real-time data query, you need to sacrifice CPU and network resources.

[0048] Finally, although elasticSearch technology is excellent in data processing itself, it still cannot solve the problem that it takes a long time to return data from multiple data sources. The data source that takes a long time will become the shortcoming of the query and slow down the overall query progress.

[0049] In view of the above shortcomings, a data query method is provided in the embodiment of the present application, which divides the query task according to the data source, and cooperates with the strategy mode to enable it to access multiple data sources more flexibly, and can support more diverse data formats than the data preprocessing solution. In addition, it can effectively avoid mutual waiting between sub-queries, and avoid the waste of thread resources and memory resources to the greatest extent.

[0050] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0051] The present application embodiment provides a data query method, such as Figure 1 As shown, a schematic diagram of a data query method in an embodiment of the present application is provided, and the method at least includes the following steps S110 to S140:

[0052] Step S110, receiving a data query request, and dividing the main task of the data query request into two different subtasks according to the time taken to return the results. The two different subtasks include a query task in which the data source can directly return the results or directly download the results, and another query task that actively notifies the requesting party to obtain the result file by means of message notification and is represented by an asynchronous task.

[0053] A data query request refers to a query on different data sources or the same data source. After receiving the data query request, the main task of the data query request is divided into two different subtasks according to the time it takes to return the data query result.

[0054] Optionally, in addition to the time taken to return the result, it is also possible to consider whether the returned result can be obtained synchronously or not, and to divide the task according to whether the result can be obtained synchronously.

[0055] Specifically, two different subtasks include a query task where a data source can directly return results or directly download the results, and such tasks can be aggregated together as a main task for execution and storage. Another query task that uses message notification to actively notify the requester to obtain the result file and is represented by an asynchronous task can use message notification to actively notify the requester to obtain the result file and is represented by an asynchronous subtask.

[0056] Step S120, execute the subtasks in the main task, and store the execution results and the overall information of the transaction in the database. The overall information of the transaction includes at least the number of subtasks, the number of completed asynchronous subtasks, and the file name of the final result file. The execution results include the execution results of the total transaction.

[0057] Execute the subtasks in the main task, and store the execution results and the overall transaction information in the database. The main content includes but is not limited to the number of asynchronous subtasks, the number of completed asynchronous subtasks, the file name of the final result file, and the execution results of the total transaction.

[0058] Step S130, use a child thread to execute the asynchronous subtask in the main task and send a request to the data source to obtain the result file name, and use the file name plus the data source number as the key to enter the library. The data source is also used to upload the result file to the enterprise data bus EDB, and the EDB is used to generate a corresponding message and put it into the message queue.

[0059] Sub-threads are used to execute asynchronous sub-tasks. Multiple sub-threads can be used to execute multiple asynchronous sub-tasks. Furthermore, sub-threads execute asynchronous sub-tasks respectively, send requests to data sources, obtain result file names, and store them in the database using the file name plus the data source number as the key, recording the execution status of the sub-task, context information, and the corresponding file name and server flag.

[0060] After the data source uploads the result file to EDB (Enterprise Data Bus), EDB generates a corresponding message and puts it into the message queue.

[0061] It can be understood that the execution status of the subtask is used as a status result of whether the subtask itself is executed.

[0062] Context information refers to the task information when the task is interrupted, mainly including but not limited to: task number, contamination source number (abnormal source), upstream system number, downstream system number, task type, and key business information that can continue the task. Through the context information, the task number, contamination source number (abnormal source), upstream system number, downstream system number, task type, and key business information that can continue the task can be determined.

[0063] The corresponding file name is used to record the file name in the subtask.

[0064] Server logo, used to mark different server locations.

[0065] Step S140: receiving the message information packaged according to the file name and data source information in the message according to the monitoring result of the message queue.

[0066] The message gateway listens to the messages in the message queue, and according to the configuration of the apollo configuration center, packages the file name and data source information in the message into a message and sends it to the same or different servers in the distributed service architecture.

[0067] Step S150, query the database according to the file name and data source to obtain the subtask information temporarily stored in the database, and forward the message information to the target server where the transaction to be continued is located according to the stored server mark, and return the query result.

[0068] The server queries the database according to the file name and data source to obtain the subtask information temporarily stored in the database. It not only forwards the message to the server where the transaction to be continued is located according to the stored server flag, but also updates the subtask message status to received to avoid repeated message processing. The server queries the database, obtains the subtask context information, downloads the file from the EDB (Enterprise Data Bus), parses and processes it, and then puts it into the warehouse. The previous task was terminated. The data to be processed is placed in the corresponding database, and the information returned from the outside needs to be retrieved. At this point, the query result is returned.

[0069] It should be noted that the above servers may be the same or different servers. When they are different servers, they need to be forwarded between the servers where the transactions to be continued exist.

[0070] Through the above method, the query tasks are divided according to the data source, and combined with the strategy mode, it can access multiple data sources more flexibly and support more diverse data formats than the data preprocessing solution.

[0071] The above method effectively avoids mutual waiting between subqueries, and avoids the waste of thread resources and memory resources to the greatest extent. It is also convenient to implement without complex and sophisticated optimization algorithms and architecture design.

[0072] like Figure 2 As shown, it is a schematic diagram of the implementation principle of the data query method in the embodiment of the present application, which specifically includes: server 1 (first server), server 2 (second server), local database, thread 1, EDB (enterprise data bus), data source, and its implementation steps are:

[0073] S1. Server 1 requests data query and starts thread 1

[0074] S2. Thread 1 returns the result file name.

[0075] S3. Thread 1 starts task splitting.

[0076] S4. Release and execute the main task, which includes multiple main tasks, Task 1, Task 2, and Task 3. In addition, Task 1, Task 2, and Task 3 may have different task sockets. Subtasks include multiple subtasks A, subtask B, and subtask C.

[0077] S5. The main task is completed and the main thread continues to execute.

[0078] S6. The query results executed by the main task are temporarily stored in the local database.

[0079] S7. Open up to thread 2.A-2.Z, and start child thread 2A.

[0080] S8. Issue and execute subtask AZ through subthread 2A.

[0081] S9. Subtask AZ queries data source AZ respectively.

[0082] S10. Return the file name to each child thread.

[0083] S11. The execution progress of each sub-thread is temporarily stored in the local database.

[0084] SA. Upload the result files to EDB separately.

[0085] SB. The message gateway transmits the file upload message to the receiving server according to the configuration information.

[0086] SC. Forward to the server where the pending transaction exists.

[0087] SD. Get the progress of subtask A according to file name A.

[0088] SE. Download the result file and parse it into the database.

[0089] SF. Query the database for the number of successful and failed tasks.

[0090] SG.1. The subtask is executed successfully and is the last one to be completed.

[0091] SG.2. The task execution process fails and it is the first one to fail.

[0092] SH. Generate successful or abnormal result files and upload them to EDB.

[0093] In one embodiment of the present application, the target server includes a first server, and the method further includes: querying a database through the first server to obtain subtask context information, downloading files from the EDB, parsing and processing them, and then putting them into storage; updating the overall transaction status through the first server, if the first server itself executes without exception, its own status is changed to successful and the number of subtasks executed is increased by one; if the first server itself executes abnormally, the overall transaction status is changed to failure.

[0094] The first server updates the overall transaction status. If there is no abnormality in its own execution, it changes its own status to successful and increases the number of subtasks completed by one. If there is an abnormality in the first server's own execution, it attempts to change the overall transaction status to failed.

[0095] In one embodiment of the present application, the method also includes: when the execution is successful, the number of subtasks is set to M, and the number of subtasks that have been executed is set to N, and the condition for completing the plus-one execution operation in the first server is that M is not equal to N+1; if the update fails, the subtask is judged to be the last subtask to be executed, and the final file is generated in combination with the result file name and the total transaction context, and then uploaded to the EDB; in the case of abnormal execution, the condition for modifying the total transaction status to failure includes that the original status of the total transaction is being executed. If the update fails, the current task is judged to be the first subtask that failed to execute, and an abnormal file is generated in combination with the result file name and uploaded to the EDB.

[0096] In order to avoid read and write problems under multi-threading and query operations, the first server adopts new update conditions and adopts a mechanism of replacing query with update.

[0097] Specifically, when the first server executes successfully, assuming that the number of subtasks is M and the number of subtasks that have been completed is N, the condition for completing the plus-one operation is that M is not equal to N+1.

[0098] If the update fails, it means that this subtask in the first server is the last subtask to be completed. The final file is generated and uploaded to EDB (Enterprise Data Bus) in combination with the result file name and the total transaction context.

[0099] In the case of execution exception, the condition for changing the total transaction status to failure is that the original status of the total transaction is being executed. If the update fails, it means that this task is the first subtask that failed to execute. Combined with the result file name, an exception file is generated and uploaded to EDB (Enterprise Data Bus).

[0100] In one embodiment of the present application, before returning the query results, it also includes: starting a scheduled task, traversing the database at each fixed time point, and directly setting the timed-out and unprocessed tasks as failed and generating an exception file.

[0101] By starting the scheduled task, the database is traversed at each fixed time point, and the timed-out and unprocessed tasks are directly set as failed and an exception file is generated.

[0102] In one embodiment of the present application, the method of using a sub-thread to execute the asynchronous sub-task in the main task and sending a request to the data source to obtain the result file name, and storing it in the database with the file name plus the data source number as the key, includes: using a first thread to execute the sub-task in the main task, and storing the execution result and the overall information of the transaction in the database; if the first thread ends waiting, establishing a sub-thread to execute the asynchronous sub-task in the main task respectively and sending a request to the data source and terminating the first thread; obtaining the result file name by sending a request to the data source, storing it in the database with the file name plus the data source number as the key, and recording the execution status of the sub-task, context information, corresponding file name, and server flag.

[0103] The first thread is started to execute the subtasks in the main task, and the execution results and the overall information of the transaction are stored in the database. The first thread ends the waiting, and a subthread is opened to execute the asynchronous subtask, and the first thread is terminated. The subthreads execute the asynchronous subtasks respectively. A request is sent to the data source to obtain the result file name, and the file name plus the data source number is used as the key to store it in the database, and the execution status of the subtask, context information, and the corresponding file name and server flag are recorded.

[0104] In one embodiment of the present application, based on the monitoring result of the message queue, receiving the message information packaged according to the file name and data source information in the message, includes: using a message gateway to monitor the messages in the message queue, and packaging the file name and data source information in the message into a message according to the configuration of the apollo configuration center and sending it to a second server, the second server and the first server belong to the same cluster.

[0105] The message gateway monitors the message in the message queue, and according to the configuration of the apollo configuration center, packages the file name and data source information in the message into a message and sends it to the second server.

[0106] In one embodiment of the present application, the second server is also used to query the database according to the file name and data source to obtain the subtask information temporarily stored in the database, forward the message to the target server where the transaction to be continued is located according to the stored server flag, and update the subtask message status to received, and the target server includes the first server.

[0107] The second server queries the database according to the file name and data source to obtain the subtask information temporarily stored in the database. On the one hand, it forwards the message to the first server where the transaction is to be continued according to the stored server flag, and on the other hand, it updates the subtask message status to received to avoid repeated message processing. The second server will not repeatedly consume messages in the message queue and lock it.

[0108] The present application embodiment also provides a data query device 300, such as Figure 3 As shown, a schematic diagram of the structure of a data query device in an embodiment of the present application is provided, wherein the data query device 300 at least includes: a receiving module 310, an execution module 320, a storage module 330, a monitoring module 340 and a query module 350, wherein:

[0109] In one embodiment of the present application, the receiving module 310 is specifically used to: receive a data query request, and divide the main task of the data query request into two different subtasks according to the time taken to return the results, the two different subtasks including a query task in which a data source can directly return results or directly download the results, and another query task that actively notifies the requesting party to obtain the result file by means of a message notification and is represented by an asynchronous task.

[0110] A data query request refers to a query on different data sources or the same data source. After receiving the data query request, the main task of the data query request is divided into two different subtasks according to the time it takes to return the data query result.

[0111] Optionally, in addition to the time taken to return the result, it is also possible to consider whether the returned result can be obtained synchronously or not, and to divide the task according to whether the result can be obtained synchronously.

[0112] Specifically, two different subtasks include a query task where a data source can directly return results or directly download the results, and such tasks can be aggregated together as a main task for execution and storage. Another query task that uses message notification to actively notify the requester to obtain the result file and is represented by an asynchronous task can use message notification to actively notify the requester to obtain the result file and is represented by an asynchronous subtask.

[0113] In one embodiment of the present application, the execution module 320 is specifically used to: execute the subtasks in the main task, and store the execution results and the overall information of the transaction in a database, wherein the overall information of the transaction includes at least the number of subtasks, the number of completed asynchronous subtasks, and the file name of the final result file, and the execution result includes the execution result of the total transaction.

[0114] Execute the subtasks in the main task, and store the execution results and the overall transaction information in the database. The main content includes but is not limited to the number of asynchronous subtasks, the number of completed asynchronous subtasks, the file name of the final result file, and the execution results of the total transaction.

[0115] In one embodiment of the present application, the storage module 330 is specifically used to: use a sub-thread to execute the asynchronous sub-task in the main task and send a request to the data source, obtain the result file name, and store it in the database using the file name plus the data source number as the key. The data source is also used to upload the result file to the enterprise data bus EDB, and the EDB is used to generate a corresponding message and put it into the message queue.

[0116] Sub-threads are used to execute asynchronous sub-tasks. Multiple sub-threads can be used to execute multiple asynchronous sub-tasks. Furthermore, sub-threads execute asynchronous sub-tasks respectively, send requests to data sources, obtain result file names, and store them in the database using the file name plus the data source number as the key, recording the execution status of the sub-task, context information, and the corresponding file name and server flag.

[0117] After the data source uploads the result file to EDB (Enterprise Data Bus), EDB generates a corresponding message and puts it into the message queue.

[0118] It can be understood that the execution status of the subtask is used as a status result of whether the subtask itself is executed.

[0119] In one embodiment of the present application, the monitoring module 340 is specifically used to: receive the message information packaged according to the file name and data source information in the message according to the monitoring result of the message queue.

[0120] The message gateway listens to the messages in the message queue, and according to the configuration of the apollo configuration center, packages the file name and data source information in the message into a message and sends it to the same or different servers in the distributed service architecture.

[0121] In one embodiment of the present application, the query module 350 is specifically used to: query the database according to the file name and data source to obtain the subtask information temporarily stored in the database, and at the same time forward the message information to the target server where the transaction to be continued is located according to the stored server flag, and return the query result.

[0122] The server queries the database according to the file name and data source to obtain the subtask information temporarily stored in the database. It not only forwards the message to the server where the transaction to be continued is located according to the stored server flag, but also updates the subtask message status to received to avoid repeated message processing. The server queries the database, obtains the subtask context information, downloads the file from the EDB (Enterprise Data Bus), parses and processes it, and then puts it into the warehouse. The previous task was terminated. The data to be processed is placed in the corresponding database, and the information returned from the outside needs to be retrieved. At this point, the query result is returned.

[0123] It should be noted that the above servers may be the same or different servers. When they are different servers, they need to be forwarded between the servers where the transactions to be continued exist.

[0124] In one embodiment of the present application, the target server includes a first server and further includes: an update module for

[0125] Query the database through the first server to obtain subtask context information, download the file from the EDB, parse and process it, and then store it in the warehouse;

[0126] The overall transaction status is updated through the first server. If the first server executes itself without exception, its own status is changed to successful and the number of subtasks executed is increased by one;

[0127] If the first server itself executes abnormally, the overall transaction status is changed to failure.

[0128] In one embodiment of the present application, the update module is also used to

[0129] When the first server executes successfully, the number of subtasks is set to M, and the number of subtasks that have been executed is set to N, and the condition for completing the plus-one execution operation in the first server is that M is not equal to N+1;

[0130] If the update fails, the subtask is determined to be the last completed subtask, and the final file is generated based on the result file name and the total transaction context, and then uploaded to the EDB;

[0131] In the case of execution exception, the conditions for modifying the total transaction status to failure include that the original status of the total transaction is being executed. If the update fails, the current task is judged to be the first subtask that failed to execute, and an exception file is generated in combination with the result file name and uploaded to the EDB.

[0132] In one embodiment of the present application, it also includes: a timing task module for

[0133] Start a scheduled task, traverse the database at each fixed time point, and directly set the timed-out and unprocessed tasks as failed and generate an exception file.

[0134] In one embodiment of the present application, the storage module 320 is used to

[0135] Using the first thread to execute the subtasks in the main task, and storing the execution results and overall transaction information in a database;

[0136] If the first thread finishes waiting, sub-threads are established to respectively execute the asynchronous sub-tasks in the main task and send requests to the data source and terminate the first thread;

[0137] The result file name is obtained by sending a request to the data source, and the file name plus the data source number is used as the key to store it in the database, recording the execution status of the subtask, context information, corresponding file name, and server flag.

[0138] In one embodiment of the present application, the monitoring module 340 is also used to

[0139] A message gateway is used to monitor messages in the message queue, and the file name and data source information in the message are packaged into a message according to the configuration of the apollo configuration center and sent to the second server, which belongs to the same cluster as the first server.

[0140] In one embodiment of the present application, the second server is also used to

[0141] The database is queried according to the file name and data source to obtain the subtask information temporarily stored in the database, and the message is forwarded to the target server where the transaction to be continued is located according to the stored server flag, and the subtask message status is updated to received, and the target server includes the first server.

[0142] It can be understood that the above-mentioned data query device can implement each step of the data query method provided in the above-mentioned embodiment, and the relevant explanations about the data query method are applicable to the data query device, which will not be repeated here.

[0143] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 4 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0144] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0145] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0146] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a data query device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0147] Receive a data query request, and divide the main task of the data query request into two different subtasks according to the time taken to return the result, wherein the two different subtasks include a query task in which the data source can directly return the result or directly download the result, and another query task in which the requester is actively notified to obtain the result file by means of message notification and represented by an asynchronous task;

[0148] Execute the subtasks in the main task, and store the execution results and overall transaction information in the database. The overall transaction information at least includes the number of subtasks, the number of completed asynchronous subtasks, and the file name of the final result file. The execution results include the execution results of the total transaction;

[0149] A sub-thread is used to execute the asynchronous sub-task in the main task and send a request to the data source to obtain the result file name, and the file name plus the data source number is used as the key to store the result file in the library. The data source is also used to upload the result file to the enterprise data bus EDB, and the EDB is used to generate a corresponding message and put it into the message queue;

[0150] According to the monitoring result of the message queue, receiving the message information packaged according to the file name and data source information in the message;

[0151] The database is queried according to the file name and data source to obtain the subtask information temporarily stored in the database, and the message information is forwarded to the target server where the transaction to be continued is located according to the stored server mark, and the query result is returned.

[0152] The above application Figure 1The method performed by the data query device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0153] The electronic device may also perform Figure 1 The method executed by the data query device in Figure 1 The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.

[0154] The present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by an electronic device including multiple application programs, enable the electronic device to execute Figure 1 The method executed by the data query device in the embodiment shown is specifically used to execute:

[0155] Receive a data query request, and divide the main task of the data query request into two different subtasks according to the time taken to return the result, wherein the two different subtasks include a query task in which the data source can directly return the result or directly download the result, and another query task in which the requester is actively notified to obtain the result file by means of message notification and represented by an asynchronous task;

[0156] Execute the subtasks in the main task, and store the execution results and overall transaction information in the database. The overall transaction information at least includes the number of subtasks, the number of completed asynchronous subtasks, and the file name of the final result file. The execution results include the execution results of the total transaction;

[0157] A sub-thread is used to execute the asynchronous sub-task in the main task and send a request to the data source to obtain the result file name, and the file name plus the data source number is used as the key to store the result file in the library. The data source is also used to upload the result file to the enterprise data bus EDB, and the EDB is used to generate a corresponding message and put it into the message queue;

[0158] According to the monitoring result of the message queue, receiving the message information packaged according to the file name and data source information in the message;

[0159] The database is queried according to the file name and data source to obtain the subtask information temporarily stored in the database, and the message information is forwarded to the target server where the transaction to be continued is located according to the stored server mark, and the query result is returned.

[0160] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0161] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0162] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0163] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0164] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0165] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0166] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0167] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0168] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0169] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A data query method, wherein: The method comprises: Receive a data query request, and divide the main task of the data query request into two different subtasks according to the time taken to return the result, wherein the two different subtasks include a query task in which the data source can directly return the result or directly download the result, and another query task in which the requester is actively notified to obtain the result file by means of message notification and represented by an asynchronous task; Execute the subtasks in the main task, and store the execution results and overall transaction information in the database. The overall transaction information at least includes the number of subtasks, the number of completed asynchronous subtasks, and the file name of the final result file. The execution results include the execution results of the total transaction; A sub-thread is used to execute the asynchronous sub-task in the main task and send a request to the data source to obtain the result file name, and the file name plus the data source number is used as the key to store the result file in the library. The data source is also used to upload the result file to the enterprise data bus EDB, and the EDB is used to generate a corresponding message and put it into the message queue; According to the monitoring result of the message queue, receiving the message information packaged according to the file name and data source information in the message; The database is queried according to the file name and data source to obtain the subtask information temporarily stored in the database, and the message information is forwarded to the target server where the transaction to be continued is located according to the stored server mark, and the query result is returned.

2. The method of claim 1, wherein: The target server includes a first server, and the method further includes: Query the database through the first server to obtain subtask context information, download the file from the EDB, parse and process it, and then store it in the warehouse; The overall transaction status is updated through the first server. If the first server executes itself without exception, its own status is changed to successful and the number of subtasks executed is increased by one; If the first server itself executes abnormally, the overall transaction status is changed to failure.

3. The method of claim 2, wherein: The method further comprises: When the first server executes successfully, the number of subtasks is set to M, and the number of subtasks that have been executed is set to N, and the condition for completing the plus-one execution operation in the first server is that M is not equal to N+1; If the update fails, the subtask is determined to be the last completed subtask, and the final file is generated based on the result file name and the total transaction context, and then uploaded to the EDB; In the case of execution exception, the conditions for modifying the total transaction status to failure include that the original status of the total transaction is being executed. If the update fails, the current task is judged to be the first subtask that failed to execute, and an exception file is generated in combination with the result file name and uploaded to the EDB.

4. The method of claim 1, wherein: Before returning the query result, the method further includes: Start a scheduled task, traverse the database at each fixed time point, and directly set the timed-out and unprocessed tasks as failed and generate an exception file.

5. The method of claim 1, wherein: The method of using a sub-thread to execute the asynchronous sub-task in the main task and sending a request to the data source to obtain the result file name, and storing the result file in the database using the file name plus the data source number as the key, includes: Using the first thread to execute the subtasks in the main task, and storing the execution results and overall transaction information in a database; If the first thread finishes waiting, sub-threads are established to respectively execute the asynchronous sub-tasks in the main task and send requests to the data source and terminate the first thread; The result file name is obtained by sending a request to the data source, and the file name plus the data source number is used as the key to store it in the database, recording the execution status of the subtask, context information, corresponding file name, and server flag.

6. The method of claim 1, wherein: The receiving, according to the monitoring result of the message queue, the message information packaged according to the file name and data source information in the message includes: A message gateway is used to monitor messages in the message queue, and the file name and data source information in the message are packaged into a message according to the configuration of the apollo configuration center and sent to the second server, which belongs to the same cluster as the first server.

7. The method of claim 6, wherein: The second server is also used for The database is queried according to the file name and data source to obtain the subtask information temporarily stored in the database, and the message is forwarded to the target server where the transaction to be continued is located according to the stored server flag, and the subtask message status is updated to received, and the target server includes the first server.

8. A data query device, wherein: The device comprises: A receiving module is used to receive a data query request, and divide the main task of the data query request into two different subtasks according to the time taken to return the result, wherein the two different subtasks include a query task in which the data source can directly return the result or directly download the result, and another query task in which the requester is actively notified to obtain the result file by means of message notification and represented by an asynchronous task; An execution module, used to execute the subtasks in the main task, and store the execution results and overall information of the transaction in a database, wherein the overall information of the transaction at least includes the number of subtasks, the number of completed asynchronous subtasks, and the file name of the final result file, and the execution result includes the execution result of the total transaction; The storage module is used to use a sub-thread to execute the asynchronous sub-task in the main task and send a request to the data source to obtain the result file name, and store the result file in the database using the file name plus the data source number as the key. The data source is also used to upload the result file to the enterprise data bus EDB, and the EDB is used to generate a corresponding message and put it into the message queue; A monitoring module, used for receiving the message information packaged according to the file name and data source information in the message according to the monitoring result of the message queue; The query module is used to query the database according to the file name and data source to obtain the subtask information temporarily stored in the database, and forward the message information to the target server where the transaction to be continued is located according to the stored server mark, and return the query result.

9. An electronic device, comprising: processor; as well as A memory arranged to store computer executable instructions, which when executed cause the processor to perform the method of any one of claims 1 to 7.

10. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including a plurality of application programs, causes the electronic device to execute any one of the methods of claims 1 to 7.

Images