Cross-data-source data access method and terminal

Abstract

The invention discloses a cross-data-source data access method and terminal, and the method comprises the steps: obtaining a data access request which comprises target data; generating an access task according to the target data and the data source corresponding to the target data, and running the access task to obtain an actual value of the target data; if the data access request comprises the association relationship of the target data, constructing a merging task according to the association relationship; and executing the merging task to obtain an associated value, and returning the actual value and the associated value through an output task. According to the method, the data acquisition process is divided into three independent tasks, so that a plurality of different data access requests can be executed in parallel, meanwhile, the three independent tasks can be realized in advance, direct running can be realized only by receiving corresponding parameters, technical personnel do not need to independently write codes for each query, and the efficiency is improved. After a demand side inputs a data access request, required data can be automatically obtained across data sources.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a method and a terminal for accessing data across data sources. Background Art

[0002] The current big data cross-data source data acquisition solution is still mainly manual, that is, the demand side connects with the technical staff → the technical staff integrates and sorts the data from different data sources into the same data source → writes SQL statements to analyze and acquire data → deletes the temporary data after the analysis is completed. The technical staff needs to write the corresponding SQL statements to read the data according to the different requirements provided by the demand side, which is time-consuming, labor-intensive and prone to errors. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a method and a terminal for accessing data across data sources, so as to realize automatic acquisition of required data across data sources.

[0004] In order to solve the above technical problems, a technical solution adopted by the present invention is:

[0005] A method for accessing data across data sources, comprising the steps of:

[0006] Obtaining a data access request, wherein the data access request includes target data;

[0007] Generate a data acquisition task according to the target data and a data source corresponding to the target data, and execute the data acquisition task to obtain an actual value of the target data;

[0008] If the data access request includes the association relationship of the target data, constructing a merging task according to the association relationship;

[0009] The merge task is executed to obtain the associated value, and the actual value and the associated value are returned through the output task.

[0010] In order to solve the above technical problems, another technical solution adopted by the present invention is:

[0011] A terminal for accessing data across data sources comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0012] Obtaining a data access request, wherein the data access request includes target data;

[0013] Generate a data acquisition task according to the target data and a data source corresponding to the target data, and execute the data acquisition task to obtain an actual value of the target data;

[0014] If the data access request includes the association relationship of the target data, constructing a merging task according to the association relationship;

[0015] The merge task is executed to obtain the associated value, and the actual value and the associated value are returned through the output task.

[0016] The beneficial effects of the present invention are as follows: a data acquisition task is generated according to the data source corresponding to the target data in the data access request. If different target data correspond to different data sources, the corresponding data acquisition tasks can be automatically generated. After the data acquisition, if the data access request also contains the association relationship between the target data, a merging task is constructed to merge the target data according to the association relationship to obtain the association value. Finally, the actual value and the association value are returned through the output task, and the process of obtaining data is divided into three independent tasks. Then, multiple different data access requests can be executed in parallel. At the same time, the three independent tasks can be implemented in advance and can be directly run by receiving the corresponding parameters. There is no need for technical personnel to write separate code for each query. After the demander inputs the data access request, the required data can be automatically obtained across data sources. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A flowchart of a method for accessing data across data sources according to an embodiment of the present invention;

[0018] Figure 2 A data acquisition task processing flow chart of an embodiment of the present invention;

[0019] Figure 3 A flowchart of a method for accessing data across data sources in an actual scenario according to an embodiment of the present invention;

[0020] Figure 4 A schematic diagram of state changes of a state machine according to an embodiment of the present invention;

[0021] Figure 5 A schematic diagram of the structure of a terminal for accessing data across data sources according to an embodiment of the present invention;

[0022] Description of labels:

[0023] 1. A terminal for accessing data across data sources; 2. A processor; 3. A memory. DETAILED DESCRIPTION

[0024] In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is an explanation in combination with the implementation modes and the accompanying drawings.

[0025] Please refer to Figure 1 , a method for accessing data across data sources, comprising the steps of:

[0026] Obtaining a data access request, wherein the data access request includes target data;

[0027] Generate a data acquisition task according to the target data and a data source corresponding to the target data, and execute the data acquisition task to obtain an actual value of the target data;

[0028] If the data access request includes the association relationship of the target data, constructing a merging task according to the association relationship;

[0029] The merge task is executed to obtain the associated value, and the actual value and the associated value are returned through the output task.

[0030] From the above description, it can be seen that the beneficial effects of the present invention are: a data acquisition task is generated according to the data source corresponding to the target data in the data access request. If different target data correspond to different data sources, the corresponding data acquisition tasks can be automatically generated. After the data acquisition, if the data access request also contains the association relationship between the target data, a merge task is constructed to merge the target data according to the association relationship to obtain the associated value. Finally, the actual value and the associated value are returned through the output task, and the data acquisition process is divided into three independent tasks. Then, multiple different data access requests can be executed in parallel. At the same time, the three independent tasks can be implemented in advance, and they can be directly run by receiving the corresponding parameters. There is no need for technical personnel to write separate code for each query. After the demander inputs the data access request, it can automatically obtain the required data across data sources.

[0031] Furthermore, it also includes:

[0032] Obtaining a to-be-executed task corresponding to the data access request from a preset task queue, where the tasks in the same preset task queue are of the same type;

[0033] Determine the processing status of the task to be executed, and if all subtasks have been executed, modify the state machine of the task to be executed to a success state;

[0034] If the subtask fails to execute, determine whether the number of retries of the subtask has reached the upper limit, and if so, modify the state machine of the task to be executed to a failed state;

[0035] The tasks to be executed include the data acquisition task, the merging task or the output task.

[0036] From the above description, we can see that each task type corresponds to a task queue. Since the resources used by each task type are different, they are placed in the task queue respectively, which can control the concurrency number corresponding to each task type and prevent the task from crashing due to the sudden occupation of a large number of resources when the task volume is large. During the task process, each task corresponds to a state machine, and the state machine is maintained according to the processing status of the task, which can ensure that the corresponding tasks can be executed sequentially in an access request.

[0037] Further, generating a data acquisition task according to the target data and a data source corresponding to the target data includes:

[0038] If the number of data sources is greater than 1, the number of data items corresponding to the actual value is obtained;

[0039] Estimate the total amount of data of the actual value according to the number of data items;

[0040] Determine whether the current memory resources are greater than the total amount of data, and if so, generate a data acquisition task in the memory according to the target data and the data source corresponding to the target data;

[0041] Otherwise, a data acquisition task is generated in the middle layer according to the target data and the data source corresponding to the target data.

[0042] From the above description, it can be seen that when it is necessary to merge the actual values ​​obtained from multiple data sources, after the data acquisition task obtains the actual value, the total amount of data is estimated based on the number of data items of the actual value. If the total amount of data is greater than the memory resources, it means that the memory resources may not be enough to process the next merge task. At this time, generating a merge task in the middle layer can ensure the processing efficiency of the merge task.

[0043] Further, the target data includes field restriction conditions;

[0044] The generating of the data acquisition task according to the target data and the data source corresponding to the target data comprises:

[0045] Generate a data acquisition task according to the field restriction condition and the SQL statement template of the data source corresponding to the target data.

[0046] From the above description, it can be seen that according to the field restriction conditions and the data source, the required target data can be located, and the data acquisition task can be accurately performed to obtain the required data.

[0047] Further, returning the actual value and the associated value through the output task includes:

[0048] The actual value and the associated value are converted into a front-end adaptation format, and the converted actual value and the associated value are returned through a preset interface.

[0049] From the above description, it can be seen that different data access requests may correspond to different formats. When returning actual values ​​and associated values, they are converted into corresponding adaptation formats to ensure the display effect of the front end.

[0050] Please refer to Figure 5 , a terminal for accessing data across data sources, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0051] Obtaining a data access request, wherein the data access request includes target data;

[0052] Generate a data acquisition task according to the target data and a data source corresponding to the target data, and execute the data acquisition task to obtain an actual value of the target data;

[0053] If the data access request includes the association relationship of the target data, constructing a merging task according to the association relationship;

[0054] The merge task is executed to obtain the associated value, and the actual value and the associated value are returned through the output task.

[0055] The beneficial effects of the present invention are as follows: a data acquisition task is generated according to the data source corresponding to the target data in the data access request. If different target data correspond to different data sources, the corresponding data acquisition tasks can be automatically generated. After the data acquisition, if the data access request also contains the association relationship between the target data, a merging task is constructed to merge the target data according to the association relationship to obtain the association value. Finally, the actual value and the association value are returned through the output task, and the process of obtaining data is divided into three independent tasks. Then, multiple different data access requests can be executed in parallel. At the same time, the three independent tasks can be implemented in advance and can be directly run by receiving the corresponding parameters. There is no need for technical personnel to write separate code for each query. After the demander inputs the data access request, the required data can be automatically obtained across data sources.

[0056] Furthermore, it also includes:

[0057] Obtaining a to-be-executed task corresponding to the data access request from a preset task queue, where the tasks in the same preset task queue are of the same type;

[0058] Determine the processing status of the task to be executed, and if all subtasks have been executed, modify the state machine of the task to be executed to a success state;

[0059] If the subtask fails to execute, determine whether the number of retries of the subtask has reached the upper limit, and if so, modify the state machine of the task to be executed to a failed state;

[0060] The tasks to be executed include the data acquisition task, the merging task or the output task.

[0061] From the above description, we can see that each task type corresponds to a task queue. Since the resources used by each task type are different, they are placed in the task queue respectively, which can control the concurrency number corresponding to each task type and prevent the task from crashing due to the sudden occupation of a large number of resources when the task volume is large. During the task process, each task corresponds to a state machine, and the state machine is maintained according to the processing status of the task, which can ensure that the corresponding tasks can be executed sequentially in an access request.

[0062] Further, generating a data acquisition task according to the target data and a data source corresponding to the target data includes:

[0063] If the number of data sources is greater than 1, the number of data items corresponding to the actual value is obtained;

[0064] Estimate the total amount of data of the actual value according to the number of data items;

[0065] Determine whether the current memory resources are greater than the total amount of data, and if so, generate a data acquisition task in the memory according to the target data and the data source corresponding to the target data;

[0066] Otherwise, a data acquisition task is generated in the middle layer according to the target data and the data source corresponding to the target data.

[0067] From the above description, it can be seen that when it is necessary to merge the actual values ​​obtained from multiple data sources, after the data acquisition task obtains the actual value, the total amount of data is estimated based on the number of data items of the actual value. If the total amount of data is greater than the memory resources, it means that the memory resources may not be enough to process the next merge task. At this time, generating a merge task in the middle layer can ensure the processing efficiency of the merge task.

[0068] Further, the target data includes field restriction conditions;

[0069] The generating of the data acquisition task according to the target data and the data source corresponding to the target data comprises:

[0070] Generate a data acquisition task according to the field restriction condition and the SQL statement template of the data source corresponding to the target data.

[0071] From the above description, it can be seen that according to the field restriction conditions and the data source, the required target data can be located, and the data acquisition task can be accurately performed to obtain the required data.

[0072] Further, returning the actual value and the associated value through the output task includes:

[0073] The actual value and the associated value are converted into a front-end adaptation format, and the converted actual value and the associated value are returned through a preset interface.

[0074] From the above description, it can be seen that different data access requests may correspond to different formats. When returning actual values ​​and associated values, they are converted into corresponding adaptation formats to ensure the display effect of the front end.

[0075] The above-mentioned method and terminal for accessing data across data sources of the present invention can be applied to situations where data from multiple different data sources need to be obtained simultaneously, which will be described below through specific implementation methods.

[0076] Please refer to Figure 1-3 , Embodiment 1 of the present invention is:

[0077] A method for accessing data across data sources, specifically comprising:

[0078] S1. Obtain a data access request, where the data access request includes target data;

[0079] In an optional embodiment, the target data includes field restrictions;

[0080] S2, generating a data acquisition task according to the target data and a data source corresponding to the target data, and running the data acquisition task to obtain an actual value of the target data, including:

[0081] Generate a data acquisition task according to the field restriction condition and the SQL statement template of the data source corresponding to the target data; for example, if the field restriction condition is a date, the entire data on the corresponding date is the actual value to be acquired;

[0082] Wherein, generating a data acquisition task according to the target data and a data source corresponding to the target data includes:

[0083] S21. If the number of the data sources is greater than 1, obtaining the number of data items corresponding to the actual value;

[0084] If the number of data sources is equal to 1, directly obtain the actual value through SQL statements, such as through JDBC;

[0085] In an optional implementation, a preset SQL statement is filled in according to the field restriction condition to obtain the number of data items corresponding to different data sources;

[0086] S22, estimating the total amount of data of the actual value according to the number of data items; that is, being able to obtain an estimate of the total amount of data obtained from all data sources, the amount of data that each data item may contain can be directly preset, and the preset amount of data can be used for estimation without actually obtaining the data; the preset amount of data can be obtained by obtaining historical data or set by technical personnel based on experience, and is not limited here;

[0087] S23, determining whether the resources of the current memory are greater than the total amount of data, if so, generating a data acquisition task in the memory according to the target data and the data source corresponding to the target data; otherwise, generating a data acquisition task in the middle layer according to the target data and the data source corresponding to the target data;

[0088] In an optional implementation, the middle layer is Flink;

[0089] S3. If the data access request includes the association relationship of the target data, constructing a merge task according to the association relationship, including:

[0090] S4, executing the merge task to obtain the associated value, and returning the actual value and the associated value through the output task, including:

[0091] The actual value and the associated value are converted into a front-end adaptation format, and the converted actual value and the associated value are returned through a preset interface.

[0092] Please refer to Figure 4 , Embodiment 2 of the present invention is:

[0093] A method for accessing data across data sources, which differs from the first embodiment in that:

[0094] Also includes:

[0095] Step 1: Obtain (take out) a to-be-executed task corresponding to the data access request from a preset task queue, where the tasks in the preset task queue are of the same type;

[0096] In an optional implementation, all pending tasks in the task queue include an identifier of a data access request, and the processing result of each pending task also includes the data access request identifier; then before executing the merge task, the process includes: determining whether an execution result of a data acquisition task having the same data access request identifier as the merge task is obtained, and if so, executing the merge task; before executing the output task, the process includes: determining whether an execution result of a merge task having the same data access request identifier as the output task is obtained, and if so, executing the output task;

[0097] Step 2: determine the processing status of the task to be executed. If all subtasks have been executed, modify the state machine of the task to be executed to a successful state; if there are unexecuted subtasks, continue to execute the subtasks in the task to be executed;

[0098] In an optional implementation, if there are no multiple independent subtasks in the task to be executed, if the processing status of the task to be executed is completed, the state machine of the task to be executed is modified to a successful state; if it is not completed, the task to be executed continues to be executed;

[0099] Step 3, determine the processing status of the subtask. If the subtask fails to execute, determine whether the number of retries of the subtask has reached the upper limit. If so, modify the state machine of the task to be executed to a failed state; if the number of retries has not reached the upper limit, increase the number of retries by 1 and put the task to be executed back into the task queue; because even if an error occurs in the subtask, it may affect the execution result of the entire task to be executed, so re-executing the entire task to be executed can further ensure the accuracy of data acquisition.

[0100] The tasks to be executed include the data acquisition tasks, merging tasks or output tasks, that is, they include three types.

[0101] Please refer to Figure 5 , Embodiment 3 of the present invention is:

[0102] A terminal 1 for accessing data across data sources comprises a processor 2, a memory 3 and a computer program stored in the memory 3 and executable on the processor 2. When the processor 2 executes the computer program, each step in the first embodiment is implemented.

[0103] In summary, the present invention provides a method and terminal for accessing data across data sources, combined with visual self-service data retrieval, to simplify the data development process as follows: the technician configures the required data source parameters and table calculation SQL at one time → the demander configures multi-table association conditions and query conditions → instantly and automatically generates reports. The technician is only responsible for configuring the basic data source information and query SQL, and the demander can complete the cross-data source data retrieval requirements by themselves, which greatly saves manpower expenses; and in the execution process, the state machine is introduced in conjunction with the task queue to split a complete task of processing data access requests into data retrieval tasks, merging tasks, and output tasks for separate management and control. The number of concurrent tasks of different types can be adjusted according to the processing time and difficulty of different tasks, and the processing order of tasks can be controlled by the state machine to avoid errors due to disordered order, thereby improving processing efficiency.

[0104] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for accessing data across data sources, It is characterized in that Includes steps: Obtaining a data access request, wherein the data access request includes target data; Generate a data acquisition task according to the target data and a data source corresponding to the target data, and execute the data acquisition task to obtain an actual value of the target data; If the data access request includes the association relationship of the target data, constructing a merging task according to the association relationship; The merge task is executed to obtain the associated value, and the actual value and the associated value are returned through the output task.

2. A method for accessing data across data sources according to claim 1, It is characterized in that Also includes: Obtaining a to-be-executed task corresponding to the data access request from a preset task queue, where the tasks in the same preset task queue are of the same type; Determine the processing status of the task to be executed, and if all subtasks have been executed, modify the state machine of the task to be executed to a success state; If the subtask fails to execute, determine whether the number of retries of the subtask has reached the upper limit, and if so, modify the state machine of the task to be executed to a failed state; The tasks to be executed include the data acquisition task, the merging task or the output task.

3. A method for accessing data across data sources according to claim 1, It is characterized in that The generating of the data acquisition task according to the target data and the data source corresponding to the target data comprises: If the number of data sources is greater than 1, the number of data items corresponding to the actual value is obtained; Estimate the total amount of data of the actual value according to the number of data items; Determine whether the current memory resources are greater than the total amount of data, and if so, generate a data acquisition task in the memory according to the target data and the data source corresponding to the target data; Otherwise, a data acquisition task is generated in the middle layer according to the target data and the data source corresponding to the target data.

4. A method for accessing data across data sources according to claim 1, It is characterized in that The target data includes field restriction conditions; The generating of the data acquisition task according to the target data and the data source corresponding to the target data comprises: Generate a data acquisition task according to the field restriction condition and the SQL statement template of the data source corresponding to the target data.

5. A method for accessing data across data sources according to claim 1, It is characterized in that The returning the actual value and the associated value by outputting the task includes: The actual value and the associated value are converted into a front-end adaptation format, and the converted actual value and the associated value are returned through a preset interface.

6. A terminal for accessing data across data sources, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the following steps are implemented: Obtaining a data access request, wherein the data access request includes target data; Generate a data acquisition task according to the target data and a data source corresponding to the target data, and execute the data acquisition task to obtain an actual value of the target data; If the data access request includes the association relationship of the target data, constructing a merging task according to the association relationship; The merge task is executed to obtain the associated value, and the actual value and the associated value are returned through the output task.

7. A terminal for accessing data across data sources according to claim 6, It is characterized in that Also includes: Obtaining a to-be-executed task corresponding to the data access request from a preset task queue, where the tasks in the same preset task queue are of the same type; Determine the processing status of the task to be executed, and if all subtasks have been executed, modify the state machine of the task to be executed to a success state; If the subtask fails to execute, determine whether the number of retries of the subtask has reached the upper limit, and if so, modify the state machine of the task to be executed to a failed state; The tasks to be executed include the data acquisition task, the merging task or the output task.

8. A terminal for accessing data across data sources according to claim 6, It is characterized in that The generating of the data acquisition task according to the target data and the data source corresponding to the target data comprises: If the number of data sources is greater than 1, the number of data items corresponding to the actual value is obtained; Estimate the total amount of data of the actual value according to the number of data items; Determine whether the current memory resources are greater than the total amount of data, and if so, generate a data acquisition task in the memory according to the target data and the data source corresponding to the target data; Otherwise, a data acquisition task is generated in the middle layer according to the target data and the data source corresponding to the target data.

9. A terminal for accessing data across data sources according to claim 6, It is characterized in that The target data includes field restriction conditions; The generating of the data acquisition task according to the target data and the data source corresponding to the target data comprises: Generate a data acquisition task according to the field restriction condition and the SQL statement template of the data source corresponding to the target data.

10. A terminal for accessing data across data sources according to claim 6, It is characterized in that The returning the actual value and the associated value by outputting the task includes: The actual value and the associated value are converted into a front-end adaptation format, and the converted actual value and the associated value are returned through a preset interface.

Images