Data processing method

Abstract

The embodiment of the invention provides a data processing method which is applied to a data calculation engine and comprises the following steps: acquiring a code file associated with a data processing task, and determining operator data by traversing the code file; analyzing the operator data to obtain data source information and data target information; constructing a data relation table corresponding to the data processing task according to the data source information, the data target information and identification information in the operator data; and generating an execution plan corresponding to the data processing task based on the data relation table, and storing data associated with the data processing task from a source end to a target end by executing the execution plan.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of data processing technology, and in particular, to a data processing method. Background Art

[0002] With the development of computer and Internet technology, data island problem has become a key issue affecting business processing efficiency. Data island problem will bring great difficulties to business-related data mining, report development, etc. In order to solve this problem, the concept of data warehouse came into being. Data warehouse can extract data scattered in different nodes from the original data source through certain processing technology, and enter the data warehouse uniformly through processing and integration, laying the foundation for subsequent decision-making. In this process, ETL (Extract-Transform-Load) technology plays a vital role. Through ETL technology, scattered data can be integrated into a standard and unified data source, thereby supporting downstream business use. In the prior art, in the process of using Spark to complete data ETL, the Spark framework calculates its pre-dependencies through the Action operator and generates an execution plan, and implements ETL processing by submitting it to the Spark cluster. However, in actual application scenarios, since a large amount of code is involved in the development process and there are dependencies between each other, in order to improve the execution speed of ETL, in most cases, users are required to manually arrange the execution plan. This processing method is extremely inefficient and requires users to have certain professional knowledge. Therefore, an effective solution is urgently needed to solve the above problems. Summary of the invention

[0003] In view of this, the embodiments of this specification provide a data processing method. One or more embodiments of this specification also relate to a data processing device, a computing device, a computer-readable storage medium and a computer program product to solve the technical defects existing in the prior art.

[0004] According to a first aspect of an embodiment of this specification, a data processing method is provided, which is applied to a data computing engine, and includes: Obtaining a code file associated with the data processing task, and determining operator data by traversing the code file; Analyze the operator data to obtain data source information and data target information; Constructing a data relationship table corresponding to the data processing task according to the data source information, the data target information and the identification information in the operator data; An execution plan corresponding to the data processing task is generated based on the data relationship table, and data associated with the data processing task is stored from the source end to the target end by executing the execution plan.

[0005] According to a second aspect of an embodiment of this specification, another data processing method is provided, including: Obtaining a code file associated with the data processing task, and determining operator data by traversing the code file; Analyze the operator data to obtain data source information and data target information; Constructing a data relationship table corresponding to the data processing task according to the data source information, the data target information and the identification information in the operator data; The data relationship table is used to store the data associated with the data processing task from the source end to the target end.

[0006] According to a third aspect of an embodiment of this specification, there is provided a data processing device, which is applied to a data computing engine, and includes: An acquisition module is configured to acquire a code file associated with a data processing task and determine operator data by traversing the code file; A parsing module, configured to parse the operator data to obtain data source information and data target information; A construction module, configured to construct a data relationship table corresponding to the data processing task according to the data source information, the data target information and the identification information in the operator data; The execution module is configured to generate an execution plan corresponding to the data processing task based on the data relationship table, and store the data associated with the data processing task from the source end to the target end by executing the execution plan.

[0007] According to a fourth aspect of an embodiment of this specification, another data processing device is provided, including: A file acquisition module is configured to acquire a code file associated with a data processing task and determine operator data by traversing the code file; A data parsing module is configured to parse the operator data to obtain data source information and data target information; A relationship table building module is configured to build a data relationship table corresponding to the data processing task according to the data source information, the data target information and the identification information in the operator data; The data relationship table is used to store the data associated with the data processing task from the source end to the target end.

[0008] According to a fifth aspect of an embodiment of this specification, a computing device is provided, including: Memory and processor; The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the above data processing method are implemented.

[0009] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the above-mentioned data processing method are implemented.

[0010] According to a seventh aspect of the embodiments of this specification, a computer program product is provided, comprising a computer program or instructions, which implement the steps of the above-mentioned data processing method when executed by a processor.

[0011] The data processing method provided in this embodiment, in order to realize the automatic construction of the execution plan corresponding to the data processing task and quickly complete the storage of data from the source end to the target end, can obtain the code file associated with the data processing task in the Spark application construction stage, at this time, the operator data can be determined by traversing the code file; the relevant execution code and its corresponding identification information can be recorded through the operator data. Then, the data source information and data target information can be extracted from the operator data by parsing the operator data; and then the data relationship table corresponding to the data processing task can be constructed according to the data source information, the data target information and the identification information in the operator data; the data relationship table records the relationship between the data source, the data target and the identification information, and supports the task execution stage. The execution plan corresponding to the data processing task can be generated based on the data relationship table, and then the data associated with the data processing task can be stored from the source end to the target end by executing the execution plan. The construction of the execution plan can be automatically completed without the need for human arrangement, and the construction of the execution plan depends on the data relationship table, which can ensure that when the execution plan is executed, the data is quickly and accurately stored from the source end to the target end, so as to facilitate the use of downstream business. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a schematic diagram of a data processing method provided by an embodiment of this specification; Figure 2 is a flow chart of a data processing method provided by an embodiment of this specification; Figure 3 is a flow chart of another data processing method provided by an embodiment of this specification; Figure 4 is a processing flow chart of a data processing method provided by an embodiment of this specification; Figure 5 is a structural schematic diagram of a data processing device provided by an embodiment of this specification; Figure 6is a schematic diagram of the structure of another data processing device provided by an embodiment of this specification; Figure 7 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0013] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0014] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0015] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0016] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0017] First, the terms involved in one or more embodiments of this specification are explained.

[0018] ETL (Extract-Transform-Load): It is the abbreviation of Extract, Transform, and Load, which represents a method of data integration. It mainly includes three steps: data extraction, data transformation, and data loading. Among them, data extraction is to extract data from various data sources (such as relational databases, files, Web services, message queues, etc.). Data extraction can use various technologies, such as database query, file reading, API call, etc. Data transformation is to clean, transform and integrate the extracted data to make it suitable for the data structure and format of the target system. Data transformation can include data cleaning (such as removing duplicate records, null value processing, outlier detection and processing, etc.), data normalization, data merging, data splitting and other operations. The transformation process can use technologies such as SQL query, script programming, ETL tools, etc. Data loading is to load the transformed data into the target system (such as data warehouse, data mart, relational database, etc.). The loading process can select appropriate loading strategies according to business needs, such as batch loading (suitable for periodic processing of large amounts of data) or real-time loading (suitable for scenarios that require instant analysis).

[0019] Spark: It is a fast and general large-scale data computing engine. It is based on memory computing, which improves the real-time performance of data processing in a big data environment while ensuring high fault tolerance and high scalability. Spark supports multiple computing modes, including batch processing, stream processing, interactive query, machine learning, and graph computing. It is implemented in Scala and provides a rich API interface to facilitate users to process and analyze big data.

[0020] In this specification, a data processing method is provided. This specification also relates to a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail one by one in the following embodiments.

[0021] See also Figure 1The schematic diagram shown, in order to realize the automatic construction of the execution plan corresponding to the data processing task and quickly complete the storage of data from the source end to the target end, the code file associated with the data processing task can be obtained in the Spark application construction stage, at this time, the operator data can be determined by traversing the code file; the relevant execution code and its corresponding identification information can be recorded through the operator data. Then, the data source information and data target information can be extracted from the operator data by parsing the operator data; and then the data relationship table corresponding to the data processing task can be constructed according to the data source information, the data target information and the identification information in the operator data; the data relationship table records the relationship between the data source, the data target and the identification information, and supports the task execution stage. The execution plan corresponding to the data processing task can be generated based on the data relationship table, and then the data associated with the data processing task can be stored from the source end to the target end by executing the execution plan. The construction of the execution plan can be automatically completed without the need for manual arrangement, and the construction of the execution plan depends on the data relationship table. It can ensure that when the execution plan is executed, the data is quickly and accurately stored from the source end to the target end, so as to facilitate the use of downstream businesses.

[0022] See also Figure 2 , Figure 2 A flow chart of a data processing method provided according to an embodiment of the present specification is shown. The method is applied to a data computing engine and specifically includes the following steps.

[0023] Step S202, obtaining a code file associated with the data processing task, and determining operator data by traversing the code file.

[0024] The data processing method provided in this embodiment is applied to a data computing engine to implement data ETL processing using the data computing engine. It can be applied to any scenario, such as a cross-platform data integration scenario (used to integrate and use data from different platforms), a log data analysis scenario (integrating logs from different servers for system performance analysis, etc.), or a business data backup scenario (backing up data generated in business scenarios), etc. This embodiment does not make any limitations here.

[0025] Specifically, the data computing engine refers to Spark, and the data processing task is the task corresponding to ETL processing, which is used to store data from the source end to the target end for use by downstream businesses. Correspondingly, the code file refers to a file consisting of the code that the data computing engine needs to execute when the currently triggered data processing task needs to be executed. Correspondingly, the operator data refers to a data record constructed by extracting the execution code corresponding to the operator based on the traversal result and combining it with the identification information after traversing the code file. It is stored in the memory based on the characteristics of the data computing engine for subsequent use.

[0026] Based on this, in order to realize the automatic construction of the execution plan corresponding to the data processing task and quickly complete the storage of data from the source end to the target end, the code file associated with the data processing task can be obtained in the Spark application construction stage. At this time, the operator data can be determined by traversing the code file; the relevant execution code and its corresponding identification information can be recorded through the operator data. Then, the data source information and data target information can be extracted from the operator data by parsing the operator data; and then the data relationship table corresponding to the data processing task can be constructed according to the data source information, data target information and identification information in the operator data; the relationship between the data source, data target and identification information can be recorded through the data relationship table, which supports the generation of the execution plan corresponding to the data processing task based on the data relationship table during the task execution stage, and then the data associated with the data processing task can be stored from the source end to the target end by executing the execution plan.

[0027] Furthermore, when traversing the code lines contained in the code file, in order to ensure the integrity of the operator data generated subsequently, the code lines containing the first operator information and the second operator information can be selected as the basis for determining the execution code, and the operator data can be constructed in combination with the identification information. In this embodiment, the specific implementation method is as follows: The code lines contained in the code file are traversed according to preset traversal rules; a code line containing first operator information is selected as a start code line according to the traversal result, and a code line containing second operator information is selected as an end code line; identification information is determined according to the start code line and the end code line, and an execution code corresponding to the identification information is extracted from the code file; operator data is constructed based on the identification information and the execution code.

[0028] Specifically, the traversal rule specifically refers to the rule of traversing the code lines contained in the code file in order and line by line, which can avoid missing information and causing the data processing task to fail to complete. Correspondingly, the first operator information specifically refers to the information that characterizes the start of the operator execution. If the code line contains the operator information, the code line can be marked as the start code line. The first operator information includes but is not limited to the read operator or the jdbc operator. Correspondingly, the second operator information specifically refers to the information that characterizes the end of the operator execution. If the code line contains the operator, the code line can be marked as the end code line. The second operator information includes but is not limited to the save operator or the write operator. Correspondingly, the execution code specifically refers to the code that matches the operators composed of all the code lines between the start code line and the end code line.

[0029] Based on this, after obtaining the code file, in order to obtain the operator data to facilitate the subsequent construction of the data relationship table, the code lines contained in the code file can be traversed according to the preset traversal rules, so as to select the code line containing the first operator information as the starting code line according to the traversal result, and select the code line containing the second operator information as the ending code line; since each operator corresponds to an execution logic, in order to realize that it can be stored as a record, the identification information can be determined according to the starting code line and the ending code line, and the execution code corresponding to the identification information can be extracted from the code file; thereby, the operator data is constructed based on the identification information and the execution code. When all the operator data corresponding to the data processing task are constructed, they can be used to construct the data relationship table, thereby automatically constructing an execution plan for executing the data processing task.

[0030] In summary, by determining the operator data by traversing the code file line by line in order, it can be ensured that the operator data can record the execution code and identification information, which makes it easier to complete the construction of the data relationship table based on the execution code and identification information in subsequent processing, so as to automatically complete the construction and execution of the execution plan.

[0031] Furthermore, when determining the identification information and the execution code, the problem of missing code lines cannot be avoided, and a sequential reading method can be used to complete the task. In this embodiment, the specific implementation method is as follows: Based on the start code line, the code lines contained in the code file are read in a set order to obtain identification information; when the end code line is read in the set order, the code lines between the start code line and the end code line are extracted to generate execution code corresponding to the identification information.

[0032] Specifically, when it is determined that the code line containing the first operator information is the start code line, and the code line containing the second operator information is the end code line, the code lines contained in the code file can be read in a set order based on the start code line. The set order can be an order of reading the initial position from the start code line and reading upward line by line. The identification information corresponding to the operator can be obtained according to the reading result; until the end code line is read according to the set order, the code line between the start code line and the end code line can be extracted to generate the execution code corresponding to the identification information, so as to be used for the creation of the data relationship table after the operator data is subsequently constructed.

[0033] In actual applications, the Spark application build can be triggered by executing the maven package command. At this time, all the Java code files involved can be scanned and the code content can be read line by line. If the code line read contains operators such as read or jdbc, the line of code can be marked as the start code line. After that, the method name and class name corresponding to the operator can be read line by line from the marked start code line, and stored in the memory as the identification information corresponding to the operator. When the code line read contains operators such as save or write, the code line is determined to be the end code line. Therefore, the code contained between the start code line and the end code line can be combined to form the execution code, and a data record (operator data) can be constructed by combining the method name and class name and stored in the memory until all the operator data associated with the data processing task is obtained, and then subsequent processing is performed to automatically complete the data processing task.

[0034] For example, when the business party needs to back up business data, it needs to submit the business data to the disaster recovery system. During this process, the Java code file associated with the data backup task in the Spark application can be scanned to determine the start code line, method name and class name containing the read operator, and the end code line containing the write operator. After that, the execution code can be extracted from the code file based on the start code line and the end code line, and an operator data can be generated and stored in the memory based on the method name and class name. After all the operator data corresponding to the data backup task is stored in the memory, the subsequent execution plan construction process can be performed.

[0035] In summary, by combining the execution code and identification information to construct the operator data, it can be ensured that when the data relationship table is generated later, the basic information required for the execution plan can be recorded, thereby ensuring that the execution plan is constructed faster, more accurately and quickly.

[0036] Step S204: parse the operator data to obtain data source information and data target information.

[0037] Specifically, after obtaining all the operator data associated with the data processing task as mentioned above, further, in order to ensure the subsequent automatic completion of the construction of the execution plan, the method of parsing the operator data can be adopted to obtain the data source information and data target information associated with the data processing task, so as to build a data relationship table based on this, which is used to record the data transfer relationship and the method name and class name of the related operator, so as to complete the automatic construction and execution of the execution plan.

[0038] The data source information specifically refers to the reading location information of the data to be transferred in the data processing task, and the data target information specifically refers to the storage location information of the data to be transferred in the data processing task. The data source information and the data target information are recorded in the execution code included in the operator data.

[0039] Furthermore, in the process of parsing operator data, considering that different data processing tasks involve different operators, different parsing methods can be set for different operators, so as to obtain data source information and data target information for subsequent use. In this embodiment, the specific implementation method is as follows: The operator data is parsed, and it is detected whether the operator data is associated with a target operator based on the parsing result; if so, a key field corresponding to the target operator is determined, and data source information and data target information are extracted from the operator data according to the key field; if not, an associated operator is determined in the operator data, and the data source information and data target information are obtained by parsing the associated operator.

[0040] Specifically, the target operator specifically refers to the SQL operator of the associated data processing task, and correspondingly, the key field specifically refers to the create field contained in the target statement corresponding to the target operator, and correspondingly, the associated operator specifically refers to the read operator or from operator of the associated data processing task.

[0041] Based on this, after obtaining the operator data, by parsing the operator data, it is possible to detect whether the operator data is associated with the target operator based on the parsing result; if so, it means that the operator data involves the SQL operator, so the data source information and data target information can be obtained by parsing the SQL code in the SQL operator; if not, it means that the operator data does not involve the SQL operator, then it is necessary to analyze the characteristics of the associated operator and extract the data source information and data target information in combination with the operator characteristics for subsequent use.

[0042] On the one hand, when the operator data is associated with the target operator, it means that the operator data contains the SQL operator, and then the key field can be used to complete the determination of the data source information and the data target information. In this embodiment, the specific implementation method is as follows: Determine the target statement included in the target operator, and detect whether the target statement contains the key field; if not, execute the steps of determining the associated operator in the operator data, and obtaining the data source information and the data target information by parsing the associated operator; if included, determine the associated field corresponding to the key field, extract the data source information in the operator data based on the key field, and extract the data target information in the operator data based on the associated field.

[0043] Specifically, the target statement refers to the SQL statement contained in the SQL operator, and correspondingly, the associated field refers to the from field.

[0044] Based on this, when the operator data is associated with the SQL operator, it is necessary to parse the SQL statement in the SQL operator to extract the data source information and the data target information. In order to ensure the accuracy of information extraction, it is necessary to first determine the target statement contained in the target operator. At this time, it can be detected whether the target statement contains the key field; if not, it means that the data source information and the data target information cannot be extracted from the SQL statement. Therefore, the steps of determining the associated operator in the operator data, parsing the associated operator, and obtaining the data source information and the data target information can be executed to complete the information extraction. If it is included, it means that the SQL statement contains the data source information and the data target information. Therefore, the associated field corresponding to the key field can be determined. After that, the data source information can be extracted from the operator data based on the key field, and the data target information can be extracted from the operator data based on the associated field. So that the data source information and the data target information can be used to construct the data relationship table later.

[0045] In actual applications, when detecting whether the operator data contains the SQL operator, it is possible to detect whether the SQL ("...") Spark API is called in the execution code contained in the operator data. If so, it can be determined that the operator data contains the SQL operator, otherwise it does not contain it.

[0046] When the operator data contains an SQL operator, you can parse the SQL statement in the SQL operator. Based on the parsing result, you can detect whether the SQL statement contains the create key field. If not, it means that the data source information and data target information cannot be extracted from the SQL statement. In this case, you can refer to the following description of the operator data not containing the SQL operator to complete the extraction of the data source information and data target information.

[0047] If it is included, it means that the SQL statement is a statement associated with creating a database object, such as some fields of the SQL statement are create table as select * from ...; at this time, it is determined that the statement contains the create key field, so the predicate of the create key field in the SQL statement can be extracted through a regular expression, and then used as the data source information, and then by extracting the from key field in the SQL statement, the predicate of the from key field can be extracted through a regular expression as the data target information, so as to facilitate the subsequent construction of the data relationship table.

[0048] In summary, when the operator data contains the target operator, the data source information and data target information can be extracted by parsing the target statement corresponding to the target operator, without the need for excessive operator analysis, thereby effectively improving the execution efficiency of data processing tasks.

[0049] On the other hand, when the operator data is not associated with the target operator, it means that the operator data does not contain the SQL operator, and then the data source information and the data target information can be determined based on the character string. In this embodiment, the specific implementation is as follows: The associated operator is parsed to obtain a source string and a target string; data source information is extracted from the operator data according to the source string, and data target information is extracted from the operator data according to the target string.

[0050] Specifically, the source string refers to a string that is associated with data source information in an associated operator, such as from, where, or join; the target string refers to a string that is associated with data target information in an associated operator, such as write.

[0051] Based on this, when the operator data does not contain the target operator, it is necessary to determine the associated operator in the operator data, and then determine the data source information and data target information by analyzing the characteristics of the associated operator. Specifically, the associated operator can be parsed to obtain the source string and the target string; then the data source information can be extracted from the operator data according to the source string, and the data target information can be extracted from the operator data according to the target string.

[0052] In actual applications, if the operator data does not contain SQL operators, you can reparse the read operator and from operator in the execution code, and extract the string from the operator as the data source information. For example, if the operator is a read operator, the content passed in is the data source, and spark.sql("select * from tableA,tableB,tableC") can be determined at this time. Therefore, the string after from and before key fields such as where or join (left join, inner join, right join) can be split by the symbol "," and the data source information can be determined based on the split result. In addition, if the operator is spark.read().option("jdbcurl", "").table("tableA"), table can be identified and the name of the passed in table can be used as the data source information. At the same time, you can also parse the save operator or write operator in the operator, and extract the string from the operator as the data target information. For example, if the operator is a write operator or a save operator, the content passed in is the target result, such as write("tableA"), save("tableB"). At this time, tableA and tableB can be extracted as data target information, so as to complete the construction of the data relationship table in combination with the data source information and the data target information.

[0053] Following the above example, after obtaining multiple operator data and storing them in memory, each operator data can be read from the memory in turn for parsing. After obtaining operator data 1, it can be detected whether operator data 1 contains an SQL operator. If it does, the SQL statement in the SQL operator can be parsed, and the predicate of the create key field in the SQL statement can be extracted as the data source name through a regular expression. If the SQL statement does not contain the create key field, the read operator or from operator in the execution code can be parsed, and the string can be extracted from the operator content as the data source name. At the same time, the from key field in the SQL statement can be extracted as the target result name through a regular expression.

[0054] In addition, when operator data 1 does not contain an SQL operator, the read or from operator in the execution code can be parsed and the string can be extracted from the operator content as the data source name, and the save or write operator in the execution code can be parsed and the string can be extracted from the operator content as the target result name.

[0055] Similar to the above processing, after processing operator data 2, operator data 3 and operator data respectively, we can get the data source name table1 and target result name table2 corresponding to operator data 1, the data source name table3 and target result name table2 corresponding to operator data 2, the data source name table1 and target result name table4 corresponding to operator data 3, the data source name table3 and target result name table4 corresponding to operator data 4. Subsequently, we can complete the construction of the execution plan based on the above information, thereby realizing the backup processing of business data.

[0056] In summary, by setting different strategies for extracting data source information and data target information for different types of operators, it can be ensured that when any type of data processing task is triggered, the data source information and data target information associated with the data processing task can be accurately extracted, thereby ensuring the effective construction of the execution plan.

[0057] Step S206: construct a data relationship table corresponding to the data processing task according to the data source information, the data target information and the identification information in the operator data.

[0058] Specifically, after obtaining the data source information and data target information associated with the operator data as described above, further, in order to automatically complete the construction of the execution plan in the future and support the execution of the data processing task, the identification information can be extracted from the operator data, and then the data relationship table corresponding to the data processing task is constructed by combining the identification information, the data source information and the data target information. The data relationship table records the association between the data source information, the data target information and the identification information, so that in the execution plan construction stage, the associated information can be read from the data relationship table to construct the execution plan. Among them, the data relationship table stores the data source information, the data target information and the identification information corresponding to each operator data as a record, so as to ensure the association and orderliness of the information.

[0059] Using the above example, after obtaining the data source name and target result name corresponding to each operator data, the method name and class name corresponding to the operator data can be read from the memory, and then the data source name and target result name can be combined to construct the data relationship table corresponding to the data backup task, so as to facilitate the subsequent execution of the data backup task. The constructed data relationship table is as follows: Table 1: Table 1

[0060] Step S208, generating an execution plan corresponding to the data processing task based on the data relationship table, and storing the data associated with the data processing task from the source end to the target end by executing the execution plan.

[0061] Specifically, after constructing the data relationship table for the data processing task, considering that each data record stored in the data relationship table can reflect the relationship between the operator, data source information and data target information, the execution plan corresponding to the data processing task can be directly generated based on the data relationship table, so that the data associated with the data processing task can be stored from the source end to the target end by executing the execution plan, and the execution of the data processing task can be completed. Among them, the execution plan specifically refers to the execution plan for completing the storage of data from the source end to the target end for the data processing task, which is used to extract data from the source end and load it to the target end for storage after processing.

[0062] Furthermore, in order to enable the execution plan constructed subsequently to complete the data processing task without being affected by the circular dependency, detection can be performed according to the following relationship. In this embodiment, the specific implementation method is as follows: The data target information contained in the data relationship table is used as the target key, and the data source information is used as the target value; it is detected whether there is a value in the data relationship table in which the target key is an associated key, and the value of the associated key is equal to the target key; if so, it is determined that there is a circular dependency relationship in the data processing task, and the status of the data processing task is updated to a failed status; if not, the step of generating an execution plan corresponding to the data processing task based on the data relationship table is executed.

[0063] Specifically, a circular dependency refers to a circular dependency between data source information and data target information. For example, data source tableA is associated with target result tableB, data source tableB is associated with target result tableC, tableC is associated with target result tableD, and tableD is associated with target result tableB. At this time, it is determined that a circular dependency exists between the data source and the target result. If an execution plan is automatically created at this time, the data processing task may not be completed, so the task needs to be updated to a failed state.

[0064] Based on this, before building an execution plan, in order to avoid the existence of circular dependencies affecting the execution results of the task, the data target information contained in the data relationship table can be used as the target key, and the data source information can be used as the target value; at this time, it can be detected whether there is a situation in the data relationship table where the target key is the value of the associated key, and the value of the associated key is equal to the target key; if so, it is determined that there is a circular dependency in the data processing task, and the status of the data processing task can be updated to a failed state at this time; if not, it means that there is no circular dependency in the data processing task, and then the steps of generating an execution plan corresponding to the data processing task based on the data relationship table can be continued.

[0065] In actual applications, when the Spark application submits the data processing task to the Spark cluster to allocate resources for execution in the Driver process, the data relationship table generated above can be loaded at its execution entry, and the execution plan can be constructed by reading the relationship record data in the data relationship table. At this time, each data target information can be used as a key and the corresponding data source information as a value to construct a HashMap. In the construction process, it is determined whether key1 is the value of key2, and whether the value of key2 is equal to key1, that is, whether any data target information is any other data source information. If so, it is considered that the data source information and data target information of the current Spark application have a circular dependency relationship. Therefore, a failure flag can be generated for the data processing task to indicate that the data processing task execution failed. If not, the execution plan can be constructed in combination with the data relationship table.

[0066] In summary, by reporting errors for situations where circular dependencies exist before building an execution plan, it is possible to avoid the problem of data processing tasks being unable to execute due to forced construction of an execution plan.

[0067] Furthermore, in order to ensure that the execution plan can be executed in an orderly manner and complete the data processing task, the execution plan can be determined by constructing a directed acyclic graph. In this embodiment, the specific implementation method is as follows: Constructing a target hash table according to the target key and the target value; determining the dependency relationship corresponding to the target hash table, and constructing a directed acyclic graph according to the target hash table and the dependency relationship; and using the directed acyclic graph as an execution plan corresponding to the data processing task.

[0068] Specifically, the target hash table refers to the HashMap constructed according to the target key and target value. Correspondingly, the dependency relationship refers to the dependency relationship between different keys in the target hash table, such as key1->key2, key2->key3, key4->key3, then it is determined that key2 depends on key1 and needs to wait for key1 to complete; key3 needs to wait for key2 and key4 to complete, and indirectly depends on key1, so key2 depends on key1. In specific implementation, a directed acyclic graph can be generated through DFS deep optimization search, which can be used as an execution plan to execute data processing tasks.

[0069] Based on this, when building an execution plan, a target hash table can be built according to the target key and target value; then the dependency relationship between each key and value in the target hash table can be determined. At this time, a directed acyclic graph can be built according to the target hash table and the dependency relationship; by using the directed acyclic graph as the execution plan corresponding to the data processing task, the effective execution of the data processing task can be guaranteed.

[0070] When executing the execution plan, it is necessary to select execution nodes with non-dependency relationships and add them to the thread pool, so as to complete the execution of tasks in sequence. In this embodiment, the specific implementation method is as follows: Extract non-dependent execution nodes in the directed acyclic graph, construct execution tasks corresponding to the execution nodes, and add the execution tasks to the thread pool; when the thread pool schedules the execution tasks, generate task instances according to target identification information corresponding to the execution nodes; and store data associated with the data processing task from the source end to the target end through the task instances.

[0071] Specifically, a task instance refers to an instance constructed for an execution task. The data processing task can be completed according to the execution instance corresponding to each execution task, thereby storing the data associated with the data processing task from the source end to the target end.

[0072] Based on this, after the execution plan is built, considering that the execution plan is determined based on a directed acyclic graph, non-dependent execution nodes can be extracted from all directed acyclic graphs. At this time, the corresponding execution task can be built for each execution node, and all execution tasks can be added to the thread pool; in the case of thread pool scheduling execution tasks, task instances can be generated according to the target identification information corresponding to the execution node; and then the data associated with the data processing task is stored from the source end to the target end through the task instance to complete the data processing task.

[0073] In actual applications, after constructing the target hash table, a directed acyclic graph can be constructed as an execution plan according to the dependencies corresponding to the target hash table. Then, the execution nodes without dependencies in the graph can be loaded, and the corresponding tasks can be generated and added to the thread pool. When the thread pool schedules the task, the corresponding instance can be generated based on the recorded method name and class name, and the corresponding method can be called to complete the task. After all tasks are completed, the data associated with the data processing task can be stored from the source end to the target end.

[0074] During this process, according to the dependencies of the nodes in the execution plan and their recorded execution status, all the completed subsequent nodes of the dependent nodes or the nodes without dependencies can be continuously added to the thread pool for parallel execution. Finally, the data transfer can be realized after the execution of all nodes is completed.

[0075] Using the above example, after obtaining the data relationship table shown in Table 1, the target result name in the data relationship table can be used as the key, and the data source name corresponding to the target result name can be used as the value to construct a HashMap. In the process of constructing the HashMap, it can be detected whether key1 has the value of key2, and whether the value of key2 is equal to key1. If so, it means that the current data backup task has a circular dependency. At this time, the result that the execution plan cannot be constructed can be fed back to the business party, so that the business party can manually complete the orchestration of the execution plan.

[0076] If it does not exist, a directed acyclic graph can be constructed as an execution plan based on the HashMap and its corresponding dependencies. After that, the directed acyclic graph can be scanned cyclically to determine the execution nodes that currently have no pre-dependencies or whose pre-dependencies have been completed. At this time, the table1 node and the table3 node are identified, so the corresponding Task tasks can be generated and added to the thread pool for scheduling and execution. After the Task task is completed, it can be marked as completed. After that, other execution nodes in the directed acyclic graph are identified, and the table2 node and the table4 node are determined at this time, and the above operations are performed cyclically. After all executions are completed, the backup operation of the business data can be completed.

[0077] The data processing method provided in this embodiment, in order to realize the automatic construction of the execution plan corresponding to the data processing task and quickly complete the storage of data from the source end to the target end, can obtain the code file associated with the data processing task in the Spark application construction stage, at this time, the operator data can be determined by traversing the code file; the relevant execution code and its corresponding identification information can be recorded through the operator data. Then, the data source information and data target information can be extracted from the operator data by parsing the operator data; and then the data relationship table corresponding to the data processing task can be constructed according to the data source information, the data target information and the identification information in the operator data; the data relationship table records the relationship between the data source, the data target and the identification information, and supports the task execution stage. The execution plan corresponding to the data processing task can be generated based on the data relationship table, and then the data associated with the data processing task can be stored from the source end to the target end by executing the execution plan. The construction of the execution plan can be automatically completed without the need for human arrangement, and the construction of the execution plan depends on the data relationship table, which can ensure that when the execution plan is executed, the data is quickly and accurately stored from the source end to the target end, so as to facilitate the use of downstream business.

[0078] See also Figure 3 , Figure 3 A flowchart of another data processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0079] Step S302: Obtain a code file associated with the data processing task, and determine operator data by traversing the code file.

[0080] Step S304: parse the operator data to obtain data source information and data target information.

[0081] Step S306, constructing a data relationship table corresponding to the data processing task according to the data source information, the data target information and the identification information in the operator data; wherein the data relationship table is used to store the data associated with the data processing task from the source end to the target end.

[0082] The above is a schematic scheme of another data processing method of this embodiment. It should be noted that the technical scheme of the other data processing method and the technical scheme of the above data processing method belong to the same concept, and the details of the technical scheme of the other data processing method that are not described in detail can all be referred to the description of the technical scheme of the above data processing method.

[0083] The following combination Figure 4 , taking the application of the data processing method provided in this specification in the data transfer scenario as an example, the data processing method is further described. Figure 4 A processing flow chart of a data processing method provided by an embodiment of the present specification is shown, which specifically includes the following steps.

[0084] Step S402, obtaining a code file associated with the data processing task, and traversing the code lines contained in the code file according to a preset traversal rule.

[0085] Step S404: select a code line containing the first operator information as a start code line according to the traversal result, and select a code line containing the second operator information as an end code line.

[0086] Step S406, determining identification information according to the start code line and the end code line, extracting execution code corresponding to the identification information from the code file, and constructing operator data based on the identification information and the execution code.

[0087] Step S408, parse the operator data, and detect whether the operator data is associated with the target operator according to the parsing result; if so, execute step S410; if not, execute step S412.

[0088] Step S410, determine the target statement included in the target operator, and when the target statement includes a key field, determine the associated field corresponding to the key field, extract data source information from the operator data based on the key field, and extract data target information from the operator data based on the associated field.

[0089] Step S412, parse the associated operator to obtain a source string and a target string, extract data source information from the operator data according to the source string, and extract data target information from the operator data according to the target string.

[0090] Step S414: construct a data relationship table corresponding to the data processing task according to the data source information, data target information and identification information in the operator data.

[0091] Step S416: Use the data target information contained in the data relationship table as the target key and the data source information as the target value, and construct a target hash table according to the target key and the target value.

[0092] Step S418, determining the dependency relationship corresponding to the target hash table, and constructing a directed acyclic graph according to the target hash table and the dependency relationship, and using the directed acyclic graph as the execution plan corresponding to the data processing task.

[0093] Step S420, extracting execution nodes with non-dependency relationships from all directed acyclic graphs, constructing execution tasks corresponding to the execution nodes, and adding the execution tasks to the thread pool.

[0094] Step S422, when the thread pool schedules the execution of the task, a task instance is generated according to the target identification information corresponding to the execution node, and the data associated with the data processing task is stored from the source end to the target end through the task instance.

[0095] In summary, in order to realize the automatic construction of the execution plan corresponding to the data processing task and quickly complete the storage of data from the source end to the target end, the code file associated with the data processing task can be obtained in the Spark application construction stage. At this time, the operator data can be determined by traversing the code file; the relevant execution code and its corresponding identification information can be recorded through the operator data. Then, the data source information and data target information can be extracted from the operator data by parsing the operator data; and then the data relationship table corresponding to the data processing task can be constructed according to the data source information, data target information and identification information in the operator data; the relationship between the data source, data target and identification information can be recorded through the data relationship table, and the execution plan corresponding to the data processing task can be generated based on the data relationship table in the task execution stage, and then the data associated with the data processing task can be stored from the source end to the target end by executing the execution plan. The construction of the execution plan can be automatically completed without the need for manual arrangement, and the construction of the execution plan depends on the data relationship table, which can ensure that when the execution plan is executed, the data is quickly and accurately stored from the source end to the target end, so as to facilitate the use of downstream businesses.

[0096] Corresponding to the above method embodiment, this specification also provides a data processing device embodiment, Figure 5 FIG. 1 is a schematic diagram showing the structure of a data processing device provided by an embodiment of the present specification. Figure 5 As shown, the device is applied to a data computing engine, comprising: An acquisition module 502 is configured to acquire a code file associated with a data processing task and determine operator data by traversing the code file; The parsing module 504 is configured to parse the operator data to obtain data source information and data target information; A construction module 506 is configured to construct a data relationship table corresponding to the data processing task according to the data source information, the data target information and the identification information in the operator data; The execution module 508 is configured to generate an execution plan corresponding to the data processing task based on the data relationship table, and store the data associated with the data processing task from the source end to the target end by executing the execution plan.

[0097] In an optional embodiment, the acquisition module 502 is further configured to: The code lines contained in the code file are traversed according to preset traversal rules; a code line containing first operator information is selected as a start code line according to the traversal result, and a code line containing second operator information is selected as an end code line; identification information is determined according to the start code line and the end code line, and an execution code corresponding to the identification information is extracted from the code file; operator data is constructed based on the identification information and the execution code.

[0098] In an optional embodiment, the acquisition module 502 is further configured to: Based on the start code line, the code lines contained in the code file are read in a set order to obtain identification information; when the end code line is read in the set order, the code lines between the start code line and the end code line are extracted to generate execution code corresponding to the identification information.

[0099] In an optional embodiment, the parsing module 504 is further configured to: The operator data is parsed, and it is detected whether the operator data is associated with a target operator based on the parsing result; if so, a key field corresponding to the target operator is determined, and data source information and data target information are extracted from the operator data according to the key field; if not, an associated operator is determined in the operator data, and the data source information and data target information are obtained by parsing the associated operator.

[0100] In an optional embodiment, the parsing module 504 is further configured to: Determine the target statement included in the target operator, and detect whether the target statement contains the key field; if not, execute the steps of determining the associated operator in the operator data, and obtaining the data source information and the data target information by parsing the associated operator; if included, determine the associated field corresponding to the key field, extract the data source information in the operator data based on the key field, and extract the data target information in the operator data based on the associated field.

[0101] In an optional embodiment, the parsing module 504 is further configured to: The associated operator is parsed to obtain a source string and a target string; data source information is extracted from the operator data according to the source string, and data target information is extracted from the operator data according to the target string.

[0102] In an optional embodiment, the device further comprises: The detection module is configured to use the data target information contained in the data relationship table as a target key, and the data source information as a target value; detect whether there is a value in the data relationship table where the target key is an associated key, and the value of the associated key is equal to the target key; if so, determine that there is a circular dependency relationship in the data processing task, and update the status of the data processing task to a failed status; if not, execute the step of generating an execution plan corresponding to the data processing task based on the data relationship table.

[0103] In an optional embodiment, the execution module 508 is further configured to: Constructing a target hash table according to the target key and the target value; determining the dependency relationship corresponding to the target hash table, and constructing a directed acyclic graph according to the target hash table and the dependency relationship; and using the directed acyclic graph as an execution plan corresponding to the data processing task.

[0104] In an optional embodiment, the execution module 508 is further configured to: Extract non-dependent execution nodes in the directed acyclic graph, construct execution tasks corresponding to the execution nodes, and add the execution tasks to the thread pool; when the thread pool schedules the execution tasks, generate task instances according to target identification information corresponding to the execution nodes; and store data associated with the data processing task from the source end to the target end through the task instances.

[0105] The data processing device provided in this embodiment, in order to realize the automatic construction of the execution plan corresponding to the data processing task and quickly complete the storage of data from the source end to the target end, can obtain the code file associated with the data processing task in the Spark application construction stage, at this time, the operator data can be determined by traversing the code file; the relevant execution code and its corresponding identification information can be recorded through the operator data. Then, the data source information and data target information can be extracted from the operator data by parsing the operator data; and then the data relationship table corresponding to the data processing task can be constructed according to the data source information, the data target information and the identification information in the operator data; the relationship between the data source, the data target and the identification information can be recorded through the data relationship table, and the execution plan corresponding to the data processing task can be generated based on the data relationship table in the task execution stage, and then the data associated with the data processing task can be stored from the source end to the target end by executing the execution plan. The construction of the execution plan can be automatically completed without the need for manual arrangement, and the construction of the execution plan depends on the data relationship table, which can ensure that when the execution plan is executed, the data is quickly and accurately stored from the source end to the target end, so as to facilitate the use of downstream business.

[0106] The above is a schematic scheme of a data processing device of this embodiment. It should be noted that the technical scheme of the data processing device and the technical scheme of the above data processing method belong to the same concept, and the details of the technical scheme of the data processing device that are not described in detail can be referred to the description of the technical scheme of the above data processing method.

[0107] Corresponding to the above method embodiment, this specification also provides another data processing device embodiment, Figure 6 FIG. 2 shows a schematic diagram of the structure of another data processing device provided by an embodiment of the present specification. Figure 6 As shown, the device comprises: The file acquisition module 602 is configured to acquire a code file associated with the data processing task and determine operator data by traversing the code file; The data parsing module 604 is configured to parse the operator data to obtain data source information and data target information; A relationship table building module 606 is configured to build a data relationship table corresponding to the data processing task according to the data source information, the data target information and the identification information in the operator data; The data relationship table is used to store the data associated with the data processing task from the source end to the target end.

[0108] The above is a schematic scheme of another data processing device of this embodiment. It should be noted that the technical scheme of the data processing device and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the data processing device can be referred to the description of the technical scheme of the above data processing method.

[0109] Figure 7 The block diagram of a computing device 700 according to an embodiment of the present specification is shown. The components of the computing device 700 include but are not limited to a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and the database 750 is used to store data.

[0110] The computing device 700 also includes an access device 740 that enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 740 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0111] In one embodiment of the present specification, the above components of the computing device 700 and Figure 7 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 7 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0112] The computing device 700 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 700 may also be a mobile or stationary server.

[0113] The processor 720 is used to execute the following computer executable instructions, which implement the steps of the above data processing method when executed by the processor.

[0114] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above data processing method.

[0115] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which can implement the steps of the above-mentioned data processing method when executed by a processor.

[0116] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above data processing method.

[0117] An embodiment of the present specification also provides a computer program product, including a computer program or instructions, which implement the steps of the above data processing method when executed by a processor.

[0118] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be referred to the description of the technical scheme of the above data processing method.

[0119] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0120] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0121] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0122] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0123] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can understand and use this specification well.

Claims

1. A data processing method, characterized in that: Applied to data computing engines, including: Obtaining a code file associated with the data processing task, and determining operator data by traversing the code file; Analyze the operator data to obtain data source information and data target information; Constructing a data relationship table corresponding to the data processing task according to the data source information, the data target information and the identification information in the operator data; An execution plan corresponding to the data processing task is generated based on the data relationship table, and data associated with the data processing task is stored from the source end to the target end by executing the execution plan.

2. The data processing method according to claim 1, characterized in that: The determining the operator data by traversing the code file includes: Traversing the code lines contained in the code file according to a preset traversal rule; According to the traversal result, a code line containing the first operator information is selected as a start code line, and a code line containing the second operator information is selected as an end code line; Determine identification information according to the start code line and the end code line, and extract execution code corresponding to the identification information from the code file; Operator data is constructed based on the identification information and the execution code.

3. The data processing method according to claim 2, characterized in that: The step of determining identification information according to the start code line and the end code line, and extracting execution code corresponding to the identification information from the code file, includes: Based on the start code line, code lines included in the code file are read in a set order to obtain identification information; When the end code line is read according to the set sequence, the code line between the start code line and the end code line is extracted to generate the execution code corresponding to the identification information.

4. The data processing method according to claim 1, characterized in that: The parsing of the operator data to obtain data source information and data target information includes: Analyze the operator data, and detect whether the operator data is associated with a target operator according to the analysis result; If yes, determine the key field corresponding to the target operator, and extract the data source information and the data target information from the operator data according to the key field; If not, a related operator is determined in the operator data, and the data source information and the data target information are obtained by parsing the related operator.

5. The data processing method according to claim 4, characterized in that: The extracting data source information and data target information from the operator data according to the key field includes: Determine a target statement included in the target operator, and detect whether the target statement includes the key field; If not included, executing the steps of determining the associated operator in the operator data, and obtaining the data source information and the data target information by parsing the associated operator; If included, determine the associated field corresponding to the key field, extract the data source information from the operator data based on the key field, and extract the data target information from the operator data based on the associated field.

6. The data processing method according to claim 4 or 5, characterized in that: The obtaining of data source information and data target information by parsing the association operator includes: Parsing the association operator to obtain a source string and a target string; The data source information is extracted from the operator data according to the source character string, and the data target information is extracted from the operator data according to the target character string.

7. The data processing method according to claim 1, characterized in that: Before the step of generating an execution plan corresponding to the data processing task based on the data relationship table is executed, the method further includes: Using the data target information contained in the data relationship table as a target key, and the data source information as a target value; Detect whether there is a target key in the data relationship table as a value of the associated key, and the value of the associated key is equal to the target key; If so, determining that a circular dependency exists between the data processing tasks, and updating the status of the data processing tasks to a failed status; If not, the step of generating an execution plan corresponding to the data processing task based on the data relationship table is executed.

8. The data processing method according to claim 7, characterized in that: The generating an execution plan corresponding to the data processing task based on the data relationship table includes: constructing a target hash table according to the target key and the target value; Determine the dependency relationship corresponding to the target hash table, and construct a directed acyclic graph according to the target hash table and the dependency relationship; The directed acyclic graph is used as an execution plan corresponding to the data processing task.

9. The data processing method according to claim 8, characterized in that: The step of storing the data associated with the data processing task from the source end to the target end by executing the execution plan includes: Extracting execution nodes with non-dependency relationships in the directed acyclic graph, constructing execution tasks corresponding to the execution nodes, and adding the execution tasks to the thread pool; In the case where the thread pool schedules the execution task, generating a task instance according to the target identification information corresponding to the execution node; The data associated with the data processing task is stored from the source end to the target end through the task instance.

10. A data processing method, characterized in that: include: Obtaining a code file associated with the data processing task, and determining operator data by traversing the code file; Analyze the operator data to obtain data source information and data target information; Constructing a data relationship table corresponding to the data processing task according to the data source information, the data target information and the identification information in the operator data; The data relationship table is used to store the data associated with the data processing task from the source end to the target end.

11. A computing device, characterized in that: include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method according to any one of claims 1 to 10 are implemented.

12. A computer-readable storage medium, characterized in that: It stores computer executable instructions, which, when executed by a processor, implement the steps of the method described in any one of claims 1 to 10.

13. A computer program product, characterized in that The method comprises a computer program or an instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 10.

Images