Data processing method and device, computer device and storage medium

Abstract

The present disclosure relates to a data processing method, device, computer equipment and storage medium. The method comprises logically dividing a processing procedure of data processing of original data, determining real tasks and virtual tasks in the data processing procedure; wherein the real tasks comprise tasks of data processing in the processing procedure, and the virtual tasks comprise tasks without data processing in the processing procedure; running the virtual tasks according to pre-configured task parameters corresponding to the virtual tasks, to obtain running results of the virtual tasks; judging virtual task relationships according to the running results of the virtual tasks and logical operators corresponding to the real tasks; and running the real tasks in response to a pass result of the virtual task relationship judgment. The present method can prevent business logic processing procedure and non-business logic processing procedure from being mixed, and prevent processing code from being redundant.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a data processing method, apparatus, computer equipment, and storage medium. Background Technology

[0002] With the development of big data, various business systems generate massive amounts of data. This vast amount of data requires logical processing to obtain the data we actually need; this logical processing is often referred to as batch processing.

[0003] Currently, batch data processing typically includes both business logic and non-business logic processing. In practice, batch data processing usually focuses on business logic, interspersed with non-business logic processing. This leads to logical inconsistencies and the possibility that non-business logic processing may correspond to multiple business logic operations, resulting in redundant code and reduced data processing speed. Summary of the Invention

[0004] Therefore, it is necessary to provide a data processing method, apparatus, computer equipment, or storage medium that will not cause confusion between business logic processing and non-business logic processing, or result in redundant processing code, in order to address the aforementioned technical problems.

[0005] Firstly, this disclosure provides a data processing method. The method includes:

[0006] The data processing procedure for raw data is logically divided to determine the real tasks and virtual tasks in the data processing procedure; wherein, the real tasks include: the tasks that perform data processing in the processing procedure, and the virtual tasks include: the tasks that do not require data processing in the processing procedure.

[0007] The virtual task is run according to the pre-configured task parameters corresponding to the virtual task, and the running result of the virtual task is obtained.

[0008] The virtual task relationship is determined based on the execution result of the virtual task and the logical operator corresponding to the real task.

[0009] If the virtual task relationship determination result is passed, the real task is executed.

[0010] In one embodiment, the virtual task includes: a completion signal determination; and the step of running the virtual task according to pre-configured task parameters corresponding to the virtual task to obtain the running result of the virtual task includes:

[0011] Read the pre-set signal task parameters, which include: the execution statement for the polling completion signal, the number of polls, and the polling time. The completion signal includes: a signal indicating that the business system has completed processing the original data.

[0012] The polling completion signal is polled based on the executed statement, polling count, and polling time to determine whether the completion signal exists and obtain the corresponding signal result.

[0013] In one embodiment, the virtual task includes: file location determination; and the step of running the virtual task according to pre-configured task parameters corresponding to the virtual task to obtain the running result of the virtual task includes:

[0014] Read the pre-set file task parameters, which include: file path, file name, number of polling iterations, and polling time; wherein, the file includes: the file in the original data that needs to be processed;

[0015] Based on the file path, file name, number of polling iterations, and polling time, determine whether the file exists and obtain the corresponding file result.

[0016] In one embodiment, the virtual task includes: information collection; and the step of running the virtual task according to pre-configured task parameters corresponding to the virtual task, and obtaining the running result of the virtual task, includes:

[0017] Read the pre-set information collection parameters;

[0018] The first data parameters of the data table are obtained according to the information collection parameters, wherein the data table includes: a table storing the original data;

[0019] After running the actual task, the method further includes:

[0020] The second data parameters of the data table are obtained based on the information collection parameters;

[0021] By comparing the first data parameters and the second data parameters, the changes in the data table after running the actual task are determined.

[0022] In one embodiment, the method further includes:

[0023] Based on the logical relationship between the real tasks and the virtual tasks, an entity link graph is created, wherein the directed acyclic graph is used to show the logical relationship between the real tasks and the virtual tasks, as well as the running status during the data processing.

[0024] In one embodiment, the step of determining the virtual task relationship based on the execution result of the virtual task and the logical operator corresponding to the real task includes:

[0025] Based on the virtual tasks required to run the real tasks, logical operators are determined, including: AND operator and OR operator;

[0026] The virtual task relationship is determined by using the logical operators and the execution results of the virtual tasks.

[0027] Secondly, this disclosure also provides a data processing apparatus. The apparatus includes:

[0028] The task partitioning module is used to logically partition the data processing process of the raw data, and determine the real tasks and virtual tasks in the data processing process; wherein, the real tasks include: the tasks that perform data processing in the processing process, and the virtual tasks include: the tasks that do not require data processing in the processing process.

[0029] The virtual task execution module is used to run the virtual task according to the pre-configured task parameters corresponding to the virtual task, and obtain the execution result of the virtual task;

[0030] The relationship judgment module is used to judge the relationship between virtual tasks based on the running results of the virtual tasks and the corresponding logical operators.

[0031] The real task execution module is used to execute the real task in response to the virtual task relationship judgment result being passed.

[0032] Thirdly, this disclosure also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of any of the above-described method embodiments.

[0033] Fourthly, this disclosure also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of any of the above-described method embodiments.

[0034] Fifthly, this disclosure also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of any of the above-described method embodiments.

[0035] In the above embodiments, the data processing procedure for raw data is logically divided to determine real tasks and virtual tasks within the data processing process. The virtual tasks are run according to pre-configured task parameters to obtain their results, ensuring that virtual and real tasks are processed separately to avoid business logic confusion. Furthermore, a virtual task relationship judgment is performed based on the virtual task's result and the corresponding logical operator of the real task; if the virtual task relationship judgment result is successful, the real task is run. The virtual task relationship judgment is performed only after the virtual task has completed; if successful, the real task is then run. Each virtual task only needs to be run once, and the corresponding real task can call the result of the virtual task executed once, avoiding repeated execution of virtual tasks, preventing redundancy in processing code, and improving processing speed during data processing. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the specific embodiments of this disclosure or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0037] Figure 1 This is a schematic diagram illustrating the application environment of the data processing method in one embodiment;

[0038] Figure 2 This is a flowchart illustrating a data processing method in one embodiment;

[0039] Figure 3 This is a flowchart illustrating step S204 in one embodiment;

[0040] Figure 4 This is a flowchart illustrating another step S204 in one embodiment;

[0041] Figure 5 This is a flowchart illustrating another step S204 and a data comparison step in one embodiment;

[0042] Figure 6 This is a schematic diagram of a DAG diagram in one embodiment;

[0043] Figure 7 This is a flowchart illustrating step S206 in one embodiment;

[0044] Figure 8 This is a flowchart illustrating the data processing method in another embodiment;

[0045] Figure 9This is a schematic block diagram of the data processing method apparatus in one embodiment;

[0046] Figure 10 This is a schematic diagram of the internal structure of a computer device in one embodiment. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this disclosure.

[0048] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings herein are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0049] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0050] As described in the background section, in actual batch processing, most batch processing tasks rely on the arrival signals of data from the business system. Once the business system has finished processing the data, it sends a completion signal to the batch processing task, which then performs the batch processing. The main task in batch processing is to process a large amount of raw, unprocessed data through specified logic to obtain the required data. This process of processing data through specified logic to obtain the required data is called business logic. Because the raw data required in actual batch processing comes from different systems and channels, the arrival times of the raw data are inconsistent. Therefore, there is a process of waiting for some raw data during batch processing, which is the aforementioned need to receive the completion signal. If the signal value of the arrived raw data (i.e., the completion signal) is polled, actual data logic processing can be performed, i.e., the actual task process (i.e., business logic processing). Therefore, the processes of waiting for raw data signals and judging signal values, which are not part of the actual logical processing, can be called non-business logic.

[0051] However, in the current data batch processing process, as exemplified above, business logic processing is often interspersed with non-business logic processing (waiting for signals, waiting for raw data, etc.), which may lead to the following problems: (1) If multiple different business logic processes share the same raw data, the same non-business logic waiting for raw data needs to be added to the corresponding multiple business logic processes, which will cause corresponding code redundancy. (2) Non-business logic is usually loaded during the business logic processing process and there is no unified standard management. (3) When the data batch processing is interrupted, non-business logic processing is often interspersed during the business logic processing process. Therefore, it is necessary to analyze each stage of the batch processing process to troubleshoot errors, which is relatively difficult.

[0052] Therefore, to solve the above problems, this disclosure provides a data processing method that can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with business server 104 via a network. Terminal 102 first receives raw data transmitted by business server 104. Terminal 102 logically divides the data processing of the raw data, determining real tasks and virtual tasks in the data processing process. Real tasks include tasks that perform data processing during the process, and virtual tasks include tasks that do not require data processing during the process. Terminal 102 runs the virtual tasks according to pre-configured task parameters and obtains the results of the virtual tasks. Terminal 102 determines the virtual task relationship based on the results of the virtual tasks and the logical operators corresponding to the real tasks. If the virtual task relationship determination result is successful, terminal 102 runs the real tasks. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, etc. Business server 104 can be implemented using a standalone server or a server cluster composed of multiple servers.

[0053] In one embodiment, such as Figure 2 As shown, a data processing method is provided, which can be applied to... Figure 1 Taking terminal 102 as an example, the explanation includes the following steps:

[0054] S202, the data processing procedure for the raw data is logically divided to determine the real tasks and virtual tasks in the data processing procedure; wherein, the real tasks include: the tasks that perform data processing in the processing procedure, and the virtual tasks include: the tasks that do not require data processing in the processing procedure.

[0055] In some embodiments of this disclosure, data processing typically refers to the process of processing a large amount of raw, unprocessed data through specified logic to obtain the data actually needed; this can generally be understood as batch data processing. The raw data is usually generated by an upstream business system. During data processing, the actual data processing steps are typically business logic, such as data cleaning, processing, and filtering, while non-business logic includes tasks that do not involve actual processing, such as detecting file locations, detecting signals, or checking other environmental parameters. Business logic is typically a real task, while non-business logic is typically a virtual task. The raw data is typically data that requires batch processing.

[0056] Specifically, the system receives raw data transmitted from upstream business systems. This raw data needs to be processed. Therefore, the data processing is logically divided according to the processing steps, determining the business logic and non-business logic within the data processing process. Typically, the business logic is executed after the non-business logic. Virtual tasks are determined based on the non-business logic. Real tasks are determined based on the business logic. Multiple virtual tasks and multiple real tasks can exist.

[0057] S204, run the virtual task according to the pre-configured task parameters corresponding to the virtual task, and obtain the running result of the virtual task.

[0058] The task parameters for a virtual task can be parameters involved in running the virtual task. For example, if a virtual task checks whether original data exists, its corresponding task parameters could be: the name of the original data, the storage location of the original data, etc., which can be used to determine whether the original data exists. It is understood that the above task parameters are only for illustrative purposes, and the corresponding task parameters will vary depending on the type of virtual task.

[0059] Specifically, based on the pre-configured task parameters for each virtual task, the corresponding virtual task can be run to obtain the running results for each virtual task.

[0060] S206, determine the virtual task relationship based on the running result of the virtual task and the logical operator corresponding to the real task.

[0061] The logical operators can be determined based on the type of the actual task and may include operators such as AND and OR. For example, if running an actual task requires either virtual task A or virtual task B to pass, then the logical operator for that actual task could be the OR operator. Virtual task relationship judgment typically involves determining the relationship between virtual tasks corresponding to the actual task. For instance, if virtual task A passes and virtual task B fails, and the operator is OR, then the corresponding virtual task relationship judgment result would be "pass".

[0062] Specifically, before running a real task, it is necessary to find the virtual task corresponding to the real task, determine the logical operator corresponding to the real task based on the virtual task, and then perform operations between the real task and the corresponding virtual task based on the logical operator to determine the virtual task relationship and obtain the virtual task relationship determination result.

[0063] S208, in response to the virtual task relationship judgment result being passed, the real task is executed.

[0064] Specifically, when the result of the virtual task relationship judgment is "pass", it can be determined that all the necessary conditions for running the real task have been met, and therefore the real task can be run.

[0065] In some exemplary embodiments, for example, the virtual tasks required before running the real task are: detecting a completion signal, detecting raw data A, and detecting raw data B. Therefore, after detecting the completion signal, detecting raw data A, and detecting raw data B, the virtual task judgment result can be determined to be successful, and the real task can be run.

[0066] S210, in response to the virtual task relationship judgment result being unsuccessful, the execution of the real task is stopped.

[0067] In the above data processing method, the data processing procedure for raw data is logically divided to determine the real tasks and virtual tasks in the data processing process. The virtual tasks are run according to the pre-configured task parameters, and the results of the virtual tasks are obtained, ensuring that virtual and real tasks are processed separately to avoid confusion in business logic. Furthermore, a virtual task relationship judgment is performed based on the results of the virtual tasks and the logical operators corresponding to the real tasks; if the virtual task relationship judgment result is successful, the real task is run. The virtual task relationship judgment is performed only after the virtual tasks have completed, and the real task is run only if it is successful. Each virtual task only needs to be run once, and the corresponding real task can call the result of the virtual task executed once, avoiding repeated execution of virtual tasks, preventing redundancy in processing code, and improving the processing speed in the data processing process.

[0068] In one embodiment, such as Figure 3 As shown, the virtual task includes: completion signal determination; running the virtual task according to the pre-configured task parameters corresponding to the virtual task; and obtaining the running result of the virtual task, including:

[0069] S302, Read the pre-set signal task parameters, which include: the execution statement for the polling completion signal, the number of polling cycles, and the polling time. The completion signal includes: a signal indicating that the business system has completed processing the original data.

[0070] S304, based on the execution statement, polling count, and polling time of the polling completion signal, poll the completion signal to determine whether the completion signal exists, and obtain the corresponding signal result.

[0071] The completion signal is typically used to determine whether the upstream business system has finished processing the raw data. A completion signal is generated when the upstream business system has completed processing the raw data. The execution statement for polling the completion signal is usually a specific statement that reads the signal, typically in code form. The signal task parameters are usually parameters used to determine whether a completion signal exists. Polling is generally a way for the CPU to decide how to provide services to peripheral devices, also known as "Programmed I / O".

[0072] Specifically, the raw data required in the actual batch processing comes from different business systems and channels, resulting in inconsistent arrival times. Therefore, there is a waiting period for some raw data during batch processing, which is the waiting for the raw signal in this example. This is because data processing can only proceed after the upstream business system has completed processing the raw data. Therefore, it is necessary to determine whether a completion signal exists to establish the initial conditions for data processing. This can be done by setting signal task parameters and then reading them. Based on the execution statement, polling count, and polling time specified in the signal task parameters, the execution statement for polling the completion signal is used to check if a completion signal exists in the business system within the specified polling count and time, obtaining the corresponding signal result. If a completion signal exists, the corresponding signal result is "Completion signal exists." If no completion signal exists, the corresponding signal result is "Completion signal does not exist." When a completion signal exists, the raw data required for data processing can be obtained from the upstream business system.

[0073] Additionally, it is understandable that business systems can also include systems that perform batch data processing, and the original data can be the data processed by the batch data processing system.

[0074] In some embodiments, if the business system is a batch processing system that performs batch data processing, the virtual task may further include generating a completion signal after the batch processing system completes its actual task. The data processed by the batch processing actual task can also be the raw data of other batch processing systems, so generating a completion signal can inform other systems that the relevant raw data is ready.

[0075] In this embodiment, by judging the completion signal, it is possible to determine whether the corresponding upstream business system has completed processing the original data. When it is necessary to process the original data that has been processed by multiple upstream business systems and the order of data processing is different, the corresponding real task can be executed based solely on the signal result, without the need for real and virtual tasks to be interspersed, ensuring that the business processing logic is not confused.

[0076] In one embodiment, such as Figure 4As shown, the virtual task includes: file location determination; running the virtual task according to the pre-configured task parameters corresponding to the virtual task; and obtaining the running result of the virtual task, including:

[0077] S402, Read the pre-set file task parameters, the file task parameters being: file path, file name, number of polling attempts, and polling time; wherein, the file includes: the file in the original data that needs to be processed.

[0078] S404, determine whether the file exists based on the file path, file name, number of polling attempts, and polling time, and obtain the corresponding file result.

[0079] In some embodiments of this disclosure, the files are often necessary for subsequent batch processing tasks, and the data stored in these files is primarily used for subsequent data processing. Once a file reaches a specific path location, the virtual task will poll for the file's existence. The subsequent real task configuring the batch data processing logic unit will then begin processing. The file task parameter can be a parameter used to determine whether the file exists.

[0080] Specifically, once a completion signal is detected in the upstream business process, it can be determined that the upstream business system has completed processing the raw data. Therefore, it is necessary to retrieve the raw data. After retrieving the raw data, it is necessary to determine whether the existence of the corresponding raw data was successfully obtained, i.e., whether it exists. Therefore, pre-set file task parameters can be read. Based on the file path, file name, polling count, and polling time, the file is searched for to see if it exists within the specified polling time and polling count. If it exists, the corresponding file result can be considered as existing. Subsequently, data processing can be performed on the file. If it does not exist, the corresponding file result can be considered as not existing.

[0081] In this embodiment, before performing actual data processing tasks—that is, before executing all virtual tasks—it is determined whether the file to be processed by the actual task exists, rather than determining it separately before each actual task. Distinguishing between actual and virtual tasks ensures that business logic remains clear. Furthermore, when multiple actual tasks correspond to the same virtual task, the virtual task only needs to be executed once, and the result saved. This result can then be used for subsequent judgments when executing actual tasks, reducing redundancy in data processing code and improving processing speed.

[0082] In one embodiment, such as Figure 5 As shown, the virtual task includes: information collection; running the virtual task according to the pre-configured task parameters corresponding to the virtual task; and obtaining the running result of the virtual task, including:

[0083] S502, read the pre-set information collection parameters;

[0084] S504, obtain the first data parameters of the data table according to the information collection parameters, wherein the data table includes: a table storing the original data;

[0085] In this case, the storage medium for storing the data is typically a database-type storage medium. Information collection parameters are typically parameters of functions built into the storage medium, and may include, for example, the table owner, table name, sampling percentage, parallelism, collection of index information, and whether the object is invalid.

[0086] Specifically, before batch processing of data, the files that need to be processed in batches are usually stored in tables on database-type storage media. The first data parameters in the data table before batch processing need to be collected according to the information collection parameters.

[0087] After running the actual task, the method further includes:

[0088] S506, Obtain the second data parameters of the data table according to the information collection parameters;

[0089] S508, compare the first data parameter and the second data parameter to determine the data changes in the data table after running the actual task.

[0090] Specifically, after collecting the first set of data parameters before batch data processing, the data in the database-type storage medium will change after the actual task is completed. Therefore, after the actual task is completed, the second set of data parameters for the data table can be collected based on the information collection parameters. By comparing the first and second data parameters, the changes in the data table in the database-type storage medium before and after the batch task execution can be statistically analyzed. Furthermore, based on these changes in the data table, it can be determined which data was processed during the data processing phase.

[0091] In this embodiment, changes in the data table can identify the data processed during the data processing. When a problem occurs in the actual task, the changes in the data table can be used to determine which data caused the problem and locate the abnormal data.

[0092] In one embodiment, the method further includes:

[0093] Based on the logical relationship between the real tasks and the virtual tasks, an entity link graph is created, wherein the directed acyclic graph is used to show the logical relationship between the real tasks and the virtual tasks, as well as the running status during the data processing.

[0094] The entity connection graph can be a DAG (Directed Acyclic Graph), which can represent the control flow in the data processing process, or an ER (Entity Relationship) graph, as long as it can describe the logical relationship between real and virtual tasks. The running status can be which real or virtual task it is currently running on.

[0095] Specifically, based on the logical relationship between virtual tasks and real tasks, such as the results of multiple virtual tasks supporting the execution of a single real task, or multiple real tasks using the execution result of a single virtual task, a unified DAG graph can be formed. The DAG graph can then be used to display the running status of the data processing process in real time.

[0096] In some exemplary embodiments, such as Figure 6 The diagram shows the Directed Acyclic Graph (DAG) between virtual tasks and real tasks. After virtual task S1 completes and passes execution independently, but virtual task S2 has not yet been executed, real task A1 can be run. After both virtual tasks S1 and S2 have completed and passed execution, real task B1 can be run. After real task A1 has completed execution and virtual task S2 has completed and passed execution, real task C1 can be run.

[0097] In this embodiment, the entity link diagram can determine the relationship between real tasks and virtual tasks, and can determine the running status of the current data processing process in real time. When a fault occurs or data processing stops, the entity link diagram can determine which virtual task or real task is currently stopped, and the problem can be repaired in a timely manner.

[0098] In one embodiment, such as Figure 7 As shown, the step of determining the virtual task relationship based on the execution result of the virtual task and the logical operator corresponding to the real task includes:

[0099] S602, determine logical operators based on the virtual tasks required to run the real task, the logical operators including: AND operator and OR operator;

[0100] S604, using the logical operators and the execution results of the virtual tasks to determine the virtual task relationship, and determining the result of the virtual task relationship determination.

[0101] Specifically, the virtual tasks required to run a real task can include one or more. When multiple virtual tasks are required to run a real task, the logical operators need to be determined. Continuing with... Figure 6For example, the virtual task required to run real task A1 can be virtual task S1. The virtual tasks required to run real task B1 are virtual task S1 and virtual task S2. If either virtual task S1 or virtual task S2 is successful enough to run real task B1, then the corresponding logical operator can be the AND operator. If both virtual tasks S1 and virtual task S2 are successful before the real task can run, then the corresponding logical operator can also be the AND operator. The result is determined based on the established logical operator and the virtual-virtual task relationship.

[0102] In addition, the virtual tasks required to run real tasks can be configured through a visual front-end interface, which can include task relationship configuration and task relationship parameters. Task relationship configuration (including but not limited to logical operations such as AND (&) and OR (|)) and task relationship parameters are dynamically configurable relationships. For example, the task relationship configuration: {0} | {1}, with corresponding task parameters: procbatch1, procbatch2. This indicates a dynamic task OR relationship, and the parameter represents the virtual task name of the object.

[0103] In some exemplary embodiments, the determination of a virtual task as a completion signal is used as an example. If the completion signal exists, the result can be true; if the completion signal does not exist, the result can be false. Logical operators are actually based on true or false for result calculation; true & true = true, true | true = true, true & false = false, true | false = true. The execution of a real task depends on the arrival of two raw data. Execution can only proceed after both raw data arrive. In this case, two virtual tasks can be configured corresponding to the arrival of the two raw data. For example, if the first raw data arrives on time, the virtual task execution result is true; if the second raw data does not arrive on time, the virtual task execution result is false. The result of the operation of the two virtual tasks is true & false = false. That is, the result of the virtual task judgment is failure. When there are more than two virtual tasks, the operation method is the same as described above.

[0104] In this embodiment, the execution results of multiple virtual tasks required for the real task can be determined by using logical operators. It can be determined whether the execution result of the corresponding virtual task meets the conditions. If the conditions are met, the real task is executed. The results of the real task and the virtual task can be distinguished.

[0105] In one embodiment, this disclosure also provides another data processing method, such as... Figure 8 As shown, it includes the following steps:

[0106] S802, the data processing procedure for the raw data is logically divided to determine the real tasks and virtual tasks in the data processing procedure; wherein, the real tasks include: the tasks that perform data processing in the processing procedure, and the virtual tasks include: the tasks that do not require data processing in the processing procedure.

[0107] S804, the virtual task includes: completion signal judgment, reading pre-set signal task parameters, the signal task parameters including: the execution statement for polling the completion signal, the number of polling times and the polling time, wherein the completion signal includes: the signal that the business system has completed processing the original data;

[0108] S806: Poll the completion signal according to the execution statement, polling count and polling time, determine whether the completion signal exists, and obtain the corresponding signal result.

[0109] S808, the virtual task includes: file location determination. Pre-set file task parameters are read, including: file path, file name, number of polling iterations, and polling time; wherein, the file includes: the file in the original data that needs data processing.

[0110] S810: Determine whether the file exists based on the file path, file name, number of polling iterations, and polling time, and obtain the corresponding file result.

[0111] S812, the virtual task includes: information collection, reading pre-set information collection parameters;

[0112] S814, obtain the first data parameters of the data table according to the information collection parameters, wherein the data table includes: a table storing the original data.

[0113] S816, determine logical operators based on the virtual tasks required to run the real task, the logical operators including: AND operator and OR operator.

[0114] S818, using the logical operator and the execution result of the virtual task, the virtual task relationship is determined, and the result of the virtual task relationship determination is determined.

[0115] S820, in response to the virtual task relationship judgment result being passed, the real task is executed.

[0116] S822, Obtain the second data parameters of the data table according to the information collection parameters.

[0117] S824, compare the first data parameter and the second data parameter to determine the data changes in the data table after running the actual task.

[0118] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0119] Based on the same inventive concept, this disclosure also provides a data processing apparatus for implementing the data processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data processing apparatus embodiments provided below can be found in the limitations of the data processing method described above, and will not be repeated here.

[0120] In one embodiment, such as Figure 9 As shown, a data processing device 900 is provided, including: a task partitioning module 902, a virtual task execution module 904, a relationship judgment module 906, and a real task execution module 908, wherein:

[0121] The task partitioning module 902 is used to logically partition the data processing process of the raw data and determine the real tasks and virtual tasks in the data processing process; wherein, the real tasks include: the tasks that perform data processing in the processing process, and the virtual tasks include: the tasks that do not require data processing in the processing process.

[0122] The virtual task execution module 904 is used to run the virtual task according to the pre-configured task parameters corresponding to the virtual task, and obtain the execution result of the virtual task;

[0123] The relationship judgment module 906 is used to judge the virtual task relationship based on the running result of the virtual task and the corresponding logical operator;

[0124] The real task execution module 908 is used to run the real task in response to the virtual task relationship judgment result being passed.

[0125] In one embodiment of the device, the virtual task includes: completion signal determination; the virtual task execution module 904 includes:

[0126] The signal task parameter reading module is used to read pre-set signal task parameters, which include: the execution statement of the polling completion signal, the number of polling times, and the polling time. The completion signal includes: a signal that the business system has completed processing the original data.

[0127] The completion signal determination module is used to poll the completion signal according to the execution statement, polling number and polling time, determine whether the completion signal exists, and obtain the corresponding signal result.

[0128] In one embodiment of the device, the virtual task includes: file location determination; the virtual task execution module 904 includes:

[0129] The file task parameter reading module is used to read pre-set file task parameters, which include: file path, file name, number of polling attempts, and polling time; wherein, the file includes: the file in the original data that needs to be processed.

[0130] The file determination module is used to determine whether the file exists based on the file path, file name, number of polling cycles, and polling time, and to obtain the corresponding file result.

[0131] In one embodiment of the device, the virtual task includes: information collection; the virtual task execution module 904 includes:

[0132] The parameter reading module is used to read pre-set information collection parameters;

[0133] The first parameter acquisition module is used to acquire the first data parameters of the data table based on the information collection parameters, wherein the data table includes a table storing the original data.

[0134] The device further includes a second parameter acquisition module, used to acquire second data parameters of the data table based on the information collection parameters.

[0135] The parameter comparison module is used to compare the first data parameter and the second data parameter to determine the data changes in the data table after running the actual task.

[0136] In one embodiment of the device, the device further includes: a logical relationship connection module, used to create an entity link graph based on the logical relationship between the real task and the virtual task, wherein the directed acyclic graph is used to display the logical relationship between the real task and the virtual task, as well as the running status during the data processing.

[0137] In one embodiment of the device, the relationship determination module 906 includes: a logical operator determination module, used to determine logical operators based on the virtual tasks required to run the real task, wherein the logical operators include: AND operator and OR operator;

[0138] The relationship judgment submodule is used to perform virtual task relationship judgment using the logical operators and the execution results of the virtual tasks, and to determine the result of the virtual task relationship judgment.

[0139] Each module in the aforementioned data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0140] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 10 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores raw data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a data processing method.

[0141] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present disclosure and does not constitute a limitation on the computer device to which the present disclosure is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0142] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in any of the above method embodiments.

[0143] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in any of the above method embodiments.

[0144] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0145] It should be noted that the data involved in this disclosure (including the original data mentioned in this disclosure) are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0146] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this disclosure can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this disclosure may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this disclosure may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0147] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0148] The embodiments described above are merely illustrative of several implementations of this disclosure, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent disclosure. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this disclosure, and these all fall within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the appended claims.

Claims

1. A data processing method, characterized in that, The method includes: The data processing procedure for raw data is logically divided to determine the real tasks and virtual tasks in the data processing procedure; wherein, the real tasks include: the tasks that perform data processing in the processing procedure, and the virtual tasks include: the tasks that do not require data processing in the processing procedure. The virtual task is run according to the pre-configured task parameters corresponding to the virtual task, and the running result of the virtual task is obtained. The virtual task relationship is determined based on the execution result of the virtual task and the logical operator corresponding to the real task. If the virtual task relationship determination result is passed, the real task is executed; The virtual task includes: information collection; and running the virtual task according to pre-configured task parameters to obtain the running result of the virtual task, including: Read the pre-set information collection parameters; The first data parameters of the data table are obtained according to the information collection parameters, wherein the data table includes: a table storing the original data; After running the actual task, the method further includes: The second data parameters of the data table are obtained based on the information collection parameters; By comparing the first data parameters and the second data parameters, the changes in the data table after running the actual task are determined.

2. The method according to claim 1, characterized in that, The virtual task includes: completion signal determination; running the virtual task according to the pre-configured task parameters corresponding to the virtual task; and obtaining the running result of the virtual task, including: Read the pre-set signal task parameters, which include: the execution statement for the polling completion signal, the number of polls, and the polling time. The completion signal includes: a signal indicating that the business system has completed processing the original data. The polling completion signal is polled based on the executed statement, polling count, and polling time to determine whether the completion signal exists and obtain the corresponding signal result.

3. The method according to claim 2, characterized in that, The virtual task includes: file location determination; running the virtual task according to the pre-configured task parameters corresponding to the virtual task; and obtaining the running result of the virtual task, including: Read the pre-set file task parameters, which include: file path, file name, number of polling iterations, and polling time; wherein, the file includes: the file in the original data that needs to be processed; Based on the file path, file name, number of polling iterations, and polling time, determine whether the file exists and obtain the corresponding file result.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Based on the logical relationship between the real tasks and the virtual tasks, a directed acyclic graph is created, wherein the directed acyclic graph is used to show the logical relationship between the real tasks and the virtual tasks, as well as the running status during the data processing.

5. The method according to claim 1, characterized in that, The step of determining the virtual task relationship based on the execution result of the virtual task and the logical operator corresponding to the real task includes: Based on the virtual tasks required to run the real tasks, logical operators are determined, including: AND operator and OR operator; The virtual task relationship is determined by using the logical operators and the execution results of the virtual tasks.

6. A data processing apparatus, characterized in that, The device includes: The task partitioning module is used to logically partition the data processing process of the raw data, and determine the real tasks and virtual tasks in the data processing process; wherein, the real tasks include: the tasks that perform data processing in the processing process, and the virtual tasks include: the tasks that do not require data processing in the processing process. The virtual task execution module is used to run the virtual task according to the pre-configured task parameters corresponding to the virtual task, and obtain the execution result of the virtual task; The relationship judgment module is used to judge the relationship between virtual tasks based on the running results of the virtual tasks and the corresponding logical operators. The real task execution module is used to run the real task in response to the virtual task relationship judgment result being passed. The virtual task includes: information collection; the virtual task execution module includes: The parameter reading module is used to read pre-set information collection parameters; The first parameter acquisition module is used to acquire the first data parameters of the data table according to the information collection parameters, wherein the data table includes: a table storing the original data; The data processing device further includes: The second parameter acquisition module is used to acquire the second data parameters of the data table based on the information collection parameters. The parameter comparison module is used to compare the first data parameter and the second data parameter to determine the data changes in the data table after running the actual task.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images