Data processing method and device

Abstract

The present application discloses a data processing method and apparatus, comprising: obtaining a target task and target data to be processed by the target task, the target data originating from a first network environment; and initiating the target task if it is determined that the target data meets the data quality requirements of a second network environment. By using this method, the problem of being unable to perform normal task scheduling due to the inability to determine the time when cross-network data from the first network environment arrives at the data processing platform of the second network environment and performing task scheduling according to an agreed time can be avoided. Furthermore, the problem of subsequent data generated by relying on the cross-network data from the first network environment and being unable to meet the data quality requirements of the second network environment and performing task scheduling according to an agreed time can be avoided, which can cause errors in the data generated, thereby affecting the entire data processing process.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more particularly to a data processing method, a data processing device, and an electronic device. The present application also relates to a data verification method, a data verification device, and an electronic device. Background Art

[0002] For cross-network data, since the data source and data processing are distributed in different network environments, for example, in hybrid cloud construction, data needs to be transferred from the public cloud network to the private cloud network for data processing. During the data processing of this type of cross-network data, task scheduling is required.

[0003] Existing scheduling methods for cross-network data usually schedule tasks according to agreed times. For example, data on the public cloud network is collected at an agreed time, and the data processing platform of the private cloud network schedules tasks based on the end time of data collection.

[0004] However, existing task scheduling methods for cross-network data have the following shortcomings:

[0005] If the data collection cycle is irregular or data cannot be collected within the agreed time due to abnormal circumstances, and the time when the cross-network data reaches the data processing platform cannot be determined, the data cannot be scheduled normally; and if the cross-network data cannot meet the data quality requirements, if the task is scheduled according to the agreed time, it may cause errors in the subsequent data generated by relying on the cross-network data, thereby affecting the entire data processing process. Summary of the Invention

[0006] The present application provides a data processing method to solve the problems of abnormal scheduling and data errors in existing cross-network data task scheduling. The present application also provides a data processing device and an electronic device.

[0007] This application provides a data processing method, including:

[0008] Obtaining a target task and target data required to be processed by the target task, wherein the target data is derived from a first network environment;

[0009] If it is determined that the target data meets the data quality requirement of the second network environment, the target task is started.

[0010] Optionally, obtaining the target task includes:

[0011] According to the predetermined offline data processing process, a data dependency set is obtained;

[0012] According to the data dependency set, a target task is obtained.

[0013] Optionally, obtaining a data dependency set according to a predetermined offline data processing process includes: parsing the task statement corresponding to the offline data processing process to obtain the data dependency set; or obtaining the data dependency set from task configuration information for the offline data processing process.

[0014] Optionally, the data dependency set includes a dependency set between data tables, a dependency set between data tables and data tasks, and a dependency set between data tasks. Obtaining the target task based on the data dependency set includes:

[0015] Obtaining a basic data table according to a set of dependency relationships between the data tables;

[0016] Obtaining data tasks that depend on the basic data table according to the basic data table and a set of dependency relationships between the data table and the data tasks;

[0017] The target task is obtained according to the data task that depends on the basic data table and the dependency relationship set between the data tasks.

[0018] Optionally, obtaining target data required for processing the target task includes:

[0019] Obtaining a first business time interval set corresponding to target data to be processed by the target task; the first business time interval set includes a business time interval corresponding to at least one data table;

[0020] Obtain a second business time interval set corresponding to the basic data in the basic data table; the second business time interval set includes a business time interval corresponding to at least one data table;

[0021] Determining whether the first service time interval set is the same as the second service time interval set;

[0022] If the first service time interval set is the same as the second service time interval set, the basic data in the basic data table is determined to be the target data.

[0023] Optionally, the dependency relationship set between the data table and the data task includes a business time parameter, where the business time parameter is used to represent a conversion relationship between a base time corresponding to the data task and a business time interval corresponding to the data to be processed by the data task. Obtaining the first business time interval set corresponding to the target data to be processed by the target task includes:

[0024] Obtaining a benchmark time corresponding to the target task;

[0025] A first business time interval set corresponding to the target data to be processed by the target task is obtained according to the reference time corresponding to the target task and the business time parameter.

[0026] Optionally, obtaining a reference time corresponding to the target task includes:

[0027] Obtaining basic data with the highest business data update frequency from the basic data in the basic data table from the first network environment;

[0028] The update time of the business data of the basic data with the highest business data update frequency that is consistent with the business data required at the current time is used as the reference time corresponding to the target task.

[0029] Optionally, also include:

[0030] Performing data verification on the cross-network data from the first network environment to obtain cross-network data without any transmission errors;

[0031] Performing data quality verification on the cross-network data without transmission errors according to the data quality verification method of the second network environment to obtain cross-network data that meets the usage requirements of the second network environment;

[0032] Correspondingly, determining whether the target data meets the data quality requirements of the second network environment includes:

[0033] If the cross-network data that meets the usage requirements of the second network environment includes the target data, it is determined that the target data meets the data quality requirements of the second network environment.

[0034] Optionally, the cross-network data originating from the first network environment includes data content and data verification information corresponding to the network environment passed by the data content during transmission, the data verification information including network environment identification information recording a transmission status of the data content, and performing data verification on the cross-network data originating from the first network environment to obtain cross-network data without transmission errors includes:

[0035] Determining whether the data content of the cross-network data changes during transmission;

[0036] If there is a change, determining the network environment in which the data content is located when the change occurs based on the network environment identification information recording the transmission status of the data content;

[0037] Collecting upstream cross-network data from an upstream network environment of the network environment where the data content changes, from the network environment where the data content changes;

[0038] When it is determined that the data content of the upstream cross-network data has not changed, the upstream cross-network data is regarded as cross-network data in which no transmission error has occurred.

[0039] Optionally, the determination of whether the data content of the cross-network data changes during transmission

[0040] Obtaining content identification information corresponding to the network environment and used to identify the data content according to the data content;

[0041] The content identification information is compared, and if the content identification information is different, it is determined that the data content of the cross-network data has changed during the transmission process.

[0042] Optionally, obtaining content identification information corresponding to the network environment and used to identify the data content according to the data content includes:

[0043] Perform a hash calculation on the data content to obtain a hash value corresponding to the network environment.

[0044] Optionally, the network environment identification information includes a transmission sequence number corresponding to the network environment, the transmission sequence number being incremented sequentially according to the network environment passed through during the cross-network data transmission process, and determining the network environment in which the data content is located when the data content changes based on the network environment identification information recording the transmission status of the data content includes:

[0045] The network environment in which the data content changes and in which the transmission sequence number is the smallest is determined as the network environment in which the data content changes.

[0046] Optionally, determining whether the target data meets the data quality requirement of the second network environment includes:

[0047] Performing data verification on the target data to determine that no transmission error occurs in the target data;

[0048] According to the data quality verification method of the second network environment, data quality verification is performed on the target data without any transmission error to determine that the target data meets the usage requirements of the second network environment.

[0049] Optionally, the first network environment is a remote network environment, and the second network environment is a local network environment.

[0050] Optionally, the first network environment is a public cloud network environment, and the second network environment is a private cloud network environment.

[0051] This application also provides a data verification method, including:

[0052] Obtaining cross-network data; the cross-network data includes data content and data verification information corresponding to the network environment passed by the data content during transmission, the data verification information including network environment identification information that can indicate the transmission status of the data content;

[0053] Determining whether the data content changes during transmission;

[0054] If it is determined that the data content has changed during the transmission process, the network environment in which the data content is located when the change occurs is determined according to the network environment identification information.

[0055] Optionally, determining whether the data content changes during transmission includes:

[0056] Obtaining content identification information corresponding to the network environment and used to identify the data content according to the data content;

[0057] The content identification information is compared, and if the content identification information is different, it is determined that the data content of the cross-network data has changed during the transmission process.

[0058] Optionally, the network environment identification information includes a transmission sequence number corresponding to the network environment, where the transmission sequence number increases sequentially according to the network environment passed through during the cross-network data transmission process. Determining the network environment in which the data content is located when the data content changes based on the network environment identification information includes:

[0059] The network environment in which the data content changes and in which the transmission sequence number is the smallest is determined as the network environment in which the data content changes.

[0060] Optionally, the method further includes: collecting cross-network data from an upstream network environment of the network environment where the data content changes, from the network environment where the data content changes.

[0061] The present application also provides a data processing device, comprising:

[0062] A target task and target data obtaining unit, configured to obtain a target task and target data required to be processed by the target task, wherein the target data is derived from the first network environment;

[0063] The target task starting unit is used to start the target task after determining that the target data meets the data quality requirements of the second network environment.

[0064] The present application also provides an electronic device, comprising:

[0065] processor;

[0066] The memory is used to store a data processing program, and when the program is read and executed by the processor, the program performs the following operations:

[0067] Obtaining a target task and target data required to be processed by the target task, wherein the target data is derived from a first network environment;

[0068] If it is determined that the target data meets the data quality requirement of the second network environment, the target task is started.

[0069] The present application also provides a data verification device, comprising:

[0070] a cross-network data obtaining unit, configured to obtain cross-network data; the cross-network data comprising data content and data verification information corresponding to the network environment through which the data content passes during transmission, the data verification information comprising network environment identification information recording a transmission status of the data content;

[0071] a change determination unit, configured to determine whether the data content has changed during transmission;

[0072] The network environment determining unit is configured to, after determining that the data content has changed during transmission, determine the network environment in which the data content is located when the data content changes according to the network environment identification information.

[0073] The present application also provides an electronic device, comprising:

[0074] processor;

[0075] The memory is used to store a data processing program, and when the program is read and executed by the processor, the program performs the following operations:

[0076] Obtaining cross-network data; the cross-network data includes data content and data verification information corresponding to the network environment passed by the data content during transmission, the data verification information including network environment identification information that can indicate the transmission status of the data content;

[0077] Determining whether the data content changes during transmission;

[0078] If it is determined that the data content has changed during the transmission process, the network environment in which the data content is located when the change occurs is determined according to the network environment identification information.

[0079] Compared with the prior art, this application has the following advantages:

[0080] The method provided by the present application starts the target task after obtaining the target data from the first network environment that the target task needs to process and determining that the target data meets the data quality requirements of the second network environment. By using this method, the problem of being unable to perform normal task scheduling due to the inability to determine the time when the cross-network data from the first network environment arrives at the data processing platform of the second network environment and the task is scheduled according to the agreed time can be avoided. It can also avoid the problem of errors in the subsequent data generated by relying on the cross-network data from the first network environment and the task being scheduled according to the agreed time when the cross-network data does not meet the data quality requirements of the second network environment, thereby affecting the entire data processing process. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1-A This is a schematic diagram of a scenario provided by an embodiment of the present application;

[0082] Figure 1 This is a flow chart of the method provided in the first embodiment of the present application;

[0083] Figure 2 This is a flow chart of the method provided in the second embodiment of the present application;

[0084] Figure 3 This is a block diagram of a device unit provided in the third embodiment of the present application;

[0085] Figure 4 is a schematic diagram of an electronic device provided in a fourth embodiment of the present application;

[0086] Figure 5 This is a block diagram of a device unit provided in the fifth embodiment of the present application;

[0087] Figure 6 This is a schematic diagram of an electronic device provided in the sixth embodiment of the present application. DETAILED DESCRIPTION

[0088] The following description sets forth many specific details to facilitate a thorough understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of the present application. Therefore, the present application is not limited to the specific implementations disclosed below.

[0089] In an offline data processing platform, the task scheduling process is the process of starting tasks and processing data, that is, mobilizing corresponding data tasks to process cross-network data collected on the data processing platform. The cross-network data may refer to data from other network environments that are different from the network environment in which the data processing platform is located. For example, in the field of cloud computing applications, common service models include public cloud, private cloud, and hybrid cloud. Public cloud refers to cloud computing applications that can provide shared resource services. Proprietary cloud, also known as private cloud, is a special model in cloud computing applications dedicated to a single organization, which only allows designated clients to run. Hybrid cloud is an integrated cloud service, such as connecting two sub-business departments of users across data centers. It combines public cloud and private cloud together. In the construction of hybrid cloud, it is necessary to connect the private cloud network and the public cloud network, such as Figure 1-A As shown, data is transmitted from the public cloud network to the private cloud network and processed in the data processing platform. In this process, the data source and data processing are distributed in different network environments, and cross-network data tasks need to be scheduled according to the predetermined data processing process.

[0090] For the above-mentioned task scheduling scenario of cross-network data, in order to avoid the problem of being unable to perform normal task scheduling due to the inability to determine the time when the cross-network data arrives at the data processing platform and performing task scheduling according to the agreed time, and in order to avoid the problem of subsequent errors in the data generated by the cross-network data due to the inability to meet the data quality requirements of the data processing platform and performing task scheduling according to the agreed time, thereby affecting the entire data processing process. The present application provides a data processing method, a data processing device corresponding to the method, and an electronic device. The present application also provides a data verification method, a data verification device corresponding to the method, and an electronic device. The following provides embodiments to describe the method, device, and electronic device in detail.

[0091] The first embodiment of the present application provides a data processing method, which can be implemented by a data processing platform that performs task scheduling for cross-network data. Figure 1 The data processing method flow chart provided in the first embodiment of this application is as follows: Figure 1 The method provided in this embodiment is described in detail. The embodiments described below are used to illustrate the principle of the method and are not intended to limit actual use.

[0092] like Figure 1 As shown, the data processing method provided in this embodiment includes the following steps:

[0093] S101: Obtain a target task and target data required to be processed by the target task.

[0094] This step is used to obtain the target task and the target data to be processed by the target task. The target data to be processed by the target task originates from the first network environment, and the target task is the data task to be started. The first network environment can refer to a remote network environment or a public cloud network environment. Conversely, the network environment in which the data processing platform is located can refer to a local network environment or a private cloud network environment. This is used here to indicate that the target data to be processed by the target task is cross-network data relative to the data processing platform. In this embodiment, a remote network environment is used as an example for illustration.

[0095] In this embodiment, the above target tasks can be obtained by:

[0096] A,according to the predetermined offline data processing process, a set of,data dependencies is obtained.

[0097] Offline data processing involves extracting, loading, and transforming data. For example, the ETL (Extract-Transform-Load) data processing process, primarily used in data warehouses, describes the process of extracting, transforming, and loading data from a source to a destination. Offline data processing involves dependencies between data, between data and tasks, and between tasks.

[0098] In this embodiment, the data dependency set can be obtained by parsing the task statement corresponding to the offline data processing process, or the data dependency set can be obtained from the task configuration information for the offline data processing process. In this case, it is necessary to ensure that the configuration of the data dependency has been completed when the task configuration of the offline data processing process is performed. For example, in the data production platform of the current network environment, the SQL statement corresponding to the predetermined offline data processing process is parsed by the data task parsing module to obtain all the data dependencies contained in the offline data processing process to form a data dependency set. For data tasks that are not SQL statements, the data dependencies need to be configured during the task configuration phase to obtain the data dependency set.

[0099] In this embodiment, the data dependency set may specifically include a dependency set between data tables, a dependency set between data tables and data tasks, and a dependency set between data tasks.

[0100] In this embodiment, the dependency relationship between data tables refers to the correspondence between a subtable generated in the data processing platform and the data table on which the subtable is dependent. The data table on which the subtable is dependent is also called the parent table of the subtable. That is, for a subtable and parent table with a dependency relationship, the data content of the subtable is generated by processing the data content of the parent table. As shown in Table 1, Data Table 1 is the parent table of Data Table 2, Data Table 0 is the parent table of Data Table 3, and Data Table 2 is the parent table of Data Table 3.

[0101] Subtable Parent table Data Table 0 Data Table 1 Data Table 2 Data Table 1 Data Table 3 Data Table 0 Data Table 3 Data Table 2 … …

[0102] Table 1

[0103] The dependency relationship between a data table and a data task refers to the correspondence between the data task used to generate the subtable and the parent table on which the subtable depends. Specifically, the parent table is the data task's dependency table, and the data task processes the parent table's data to generate the subtable's data. As shown in Table 2, Data Table 0 is the dependency table for Data Task 0, and Data Table 1 is the dependency table for Data Task 1.

[0104]

[0105]

[0106] Table 2

[0107] In this embodiment, the dependency set between the data table and the data task includes a business time parameter, which is used to indicate the business time interval corresponding to the data content when the data task processes the data content of the parent table. For example, after data task 0 is started, it needs to process the data content of data table 0 within a specific time period. The specific time period can be represented by the business time interval.

[0108] In this offline data processing process, after a data task is processed on one data node, the next data task will be processed. Data tasks have dependencies. The dependencies between data tasks can be obtained through the relationships between the data nodes corresponding to the data tasks. This refers to the correspondence between the data tasks of a downstream data node and the data tasks of the upstream data nodes on which it depends. If the current data task depends on the data task of an upstream data node, the data task of the upstream data node is the parent task of the current data task. As shown in Table 3, Data Task 1 is the parent task of Data Task 2, Data Task 0 is the parent task of Data Task 3, and Data Task 2 is the parent task of Data Task 4.

[0109] Data Tasks Parent Task Data Task 0 Data Task 1 Data Task 2 Data Task 1 Data Task 3 Data Task 0 Data Task 4 Data Task 2 ... ...

[0110] Table 3

[0111] B. Obtain the target task based on the above data dependency set.

[0112] In this embodiment, the process of obtaining the target task according to the data dependency set is specifically as follows:

[0113] B-1. Obtain the base data table based on the dependency relationships between data tables. For example, in Table 1 above, Data Table 0 and Data Table 1 do not have corresponding parent tables. This indicates that on the data processing platform in the local network environment, Data Table 0 and Data Table 1 do not have any dependent data tables. That is, Data Table 0 and Data Table 1 are base data tables, and the data content in Data Table 0 and Data Table 1 is base data.

[0114] B-2. Based on the base data table and the dependency relationships between the data table and the data tasks, obtain the data tasks that depend on the base data table. For example, by using the corresponding relationships between the data tasks and dependency tables in Table 2, you can query and obtain data tasks 0 and 3 that depend on base data table 0, and data tasks 1 and 2 that depend on base data table 1.

[0115] B-3. Based on the data tasks that depend on the base data table and the dependencies between these data tasks, a target task is obtained. This target task is the data task that depends on the base data table. For example, in Table 3, Data Task 0 and Data Task 1 do not have a parent task, indicating that Data Task 0 and Data Task 1 do not need to rely on the data tasks of their upstream data nodes when they start. Combined with the content in B-2, it can be seen that Data Task 0 and Data Task 1 only need to rely on the base data table. Therefore, Data Task 0 or Data Task 1 can be determined as the target task.

[0116] In this embodiment, the process of obtaining the target data to be processed by the target task is to determine whether the data processing platform has collected the target data from the target data's source network, or to determine whether the data collected by the data processing platform contains the target data. For example, let's assume that Data Task 0 is the target task, the underlying data table it relies on is Data Table 0, and the target data to be processed by Data Task 0 is the data content of Data Table 0. Therefore, obtaining the target data essentially involves determining that the data content of Data Table 0 is the same as the target data to be processed by Data Task 0.

[0117] It should be noted that a data task may rely on multiple basic data tables. Therefore, a target task may correspond to multiple basic data tables, and the target data to be processed by the target task may be a data set consisting of the data contents of multiple basic data tables.

[0118] In this embodiment, the target data required for processing the target task is obtained in the following manner:

[0119] First, a first set of business time intervals corresponding to the target data to be processed by the target task is obtained; the first set of business time intervals includes business time intervals corresponding to at least one data table. A business time interval is used to indicate the time interval within which the data in the data table falls. The dependency relationship set between the data table and the data task (Table 2) includes business time parameters, which are used to indicate the conversion relationship between the reference time corresponding to the data task and the business time interval corresponding to the data to be processed by the data task. For example, in the business time parameters in Table 2, "yyyy" represents the year, "mm" represents the month, "q" represents the quarter, "week" represents the week, and "dd" represents the day. $[yyyymm] is equivalent to $[yyyymm-0], where "yyyymm" is the reference time corresponding to data task 0. $[yyyymm] indicates that the data content to be processed by data task 0 is the data content of data table 0 in the month of the reference time; $[yyyymmdd-n] indicates that the data content to be processed by data task 1 is the data content of data table 1 before the n dates of the reference time.

[0120] In this embodiment, the first service time interval set is obtained as follows:

[0121] Obtain the reference time corresponding to the target task, and obtain the first business time interval set corresponding to the target data to be processed by the target task based on the reference time corresponding to the target task and the business time parameters in Table 2. The method for obtaining the reference time corresponding to the target task is: obtain the basic data with the highest business data update frequency from the basic data in the basic data table from the remote network environment; use the time in the business data update time of the basic data with the highest business data update frequency that is consistent with the business data update time required for the current time as the reference time corresponding to the target task. For example, obtain a data table set with the highest data update frequency from the network data collected locally, select a basic data table from the data table set, and use the update time in which the data content in the selected basic data table is updated the latest and is consistent with the business time required for the business as the reference time.

[0122] For example, through the above calculation method, it is known that the target data required to be processed by the current target task is the data content of February 2019 in Data Table 1 and the data content of March 7, 2019 in Data Table 2. Therefore, the first business time interval set corresponding to the target data required to be processed by the target task is [{Data Table 1, Monthly Table, Business Time Interval: 201902}, {Data Table 2, Day, Business Time Interval: 20190307}].

[0123] Next, obtain a second set of business time intervals corresponding to the basic data in the basic data table; this second set of business time intervals includes the business time intervals corresponding to at least one data table. For example, there are two basic data tables, Data Table 1 and Data Table 2. The currently updated data in Data Table 1 is the data for February 2019, and the currently updated data in Data Table 2 is the data for March 7, 2019. Then, the second set of business time intervals corresponding to the basic data in the basic data tables from the remote network environment is [{Data Table 1, monthly table, business time interval: 201902}, {Data Table 2, daily table, business time interval: 20190307}].

[0124] Finally, the first business time interval set is compared with the second business time interval set to determine whether the first business time interval set is the same as the second business time interval set. If the first business time interval set is the same as the second business time interval set, the basic data in the basic data table is determined to be the target data. For example, if the first business time interval set is [{data table 1, monthly table, business time interval: 201902}, {data table 2, day, business time interval: 20190307}] and the second business time interval set is [{data table 1, monthly table, business time interval: 201902}, {data table 2, day table, business time interval: 20190307}], it can be seen that the first business time interval set is the same as the second business time interval set, indicating that the local data processing platform has obtained the target data from the remote network environment required for processing the above-mentioned target task.

[0125] S102: If it is determined that the target data meets the data quality requirements of the second network environment, the target task is started.

[0126] After obtaining the target task and the target data to be processed by the target task in the above steps, this step is used to start the target task after determining that the target data meets the data quality requirements of the second network environment, that is, to process the target data through the target task.

[0127] The second network environment refers to the network environment in which the data processing platform is located. Compared with the first network environment in the above step S101, which is a remote network environment or a public cloud network, the second network environment can refer to a local network environment or a private cloud network.

[0128] In this embodiment, a local network environment is used as an example for illustration. The process of determining whether the target data meets the data quality requirements of the local network environment includes the following two methods:

[0129] Method 1: Determine whether the target data is included in cross-network data from a remote network environment that has passed data verification on the current data processing platform, meets local usage requirements, and originates from a remote network environment. Specifically, if the target data is included in the cross-network data that meets local usage requirements, then the target data is determined to meet the data quality requirements of the local network environment.

[0130] Method 2: Perform data verification on the target data obtained from the remote network environment to determine that no transmission errors have occurred in the target data, and perform data quality verification on the target data without transmission errors according to the data quality verification method of the local network environment to determine that the target data meets local usage requirements.

[0131] If the above method 1 is adopted, before this step, or before the above step S101, the cross-network data collected by the current data processing platform from the remote network environment needs to be pre-checked and verified for data quality. The process specifically includes the following:

[0132] 1. Perform data verification on cross-network data from a remote network environment to obtain cross-network data without transmission errors.

[0133] In this embodiment, the cross-network data originating from a remote network environment includes data content and data verification information corresponding to the network environment through which the data content passes during transmission. In the process of data being transmitted from the source system to the local data processing platform, each time the data passes through a network, a data verification information is recorded. The data verification information includes network environment identification information that can represent the transmission status of the data content and content identification information for identifying the data content. That is, each network environment generates corresponding network environment identification information and content identification information for the data content of the cross-network data. The network environment identification information may include information such as the network name, data transmission timestamp, and a transmission sequence number corresponding to the network environment passed through. The transmission sequence number increases sequentially according to the network environment passed through during the cross-network data transmission process. The above-mentioned content identification information can be a hash value obtained by hashing the data content in each network environment passed through by the cross-network data, or a value of the data volume corresponding to the data content in each network environment passed through.

[0134] In this embodiment, data verification is performed on cross-network data from a remote network environment to obtain cross-network data without transmission errors, including the following process:

[0135] (1) Determine whether the data content of the cross-network data has changed during transmission, that is, determine whether data distortion, data loss, or other transmission errors have occurred during transmission. The process is as follows: based on the data content of the cross-network data, obtain content identification information corresponding to the above-mentioned network environment, for example, perform hash calculation on the data content of the cross-network data in each network environment, and obtain hash values corresponding to the local network environment and other network environments that the cross-network data passes through; then, compare the content identification information corresponding to the local network environment with the content identification information corresponding to other network environments in the above-mentioned data verification information, for example, compare the hash value corresponding to the local network environment with the hash values corresponding to the data content of the cross-network data in other network environments, or compare the hash values corresponding to all network environments together. If any two hash values are different, it is determined that the data content of the cross-network data has changed during transmission.

[0136] (2) If the data content of the cross-network data changes during transmission, the network environment in which the data content changes is determined based on the network environment identification information that records the transmission status of the data content. In this embodiment, the network environment with the smallest transmission sequence number among the network environments in which the data content changes can be determined as the network environment in which the data content changes. For example, according to the comparison result of the above hash values, the transmission sequence number when the hash value begins to change is 3, that is, the same hash value corresponding to the network environments with transmission sequence numbers 1 and 2 is different from the hash values corresponding to the subsequent network environments with transmission sequence numbers 4, 5, 6, etc., so the network environment with transmission sequence number 3 is determined to be the network environment in which the data content changes.

[0137] (3) From the network environment determined above where the data content changes, collect upstream cross-network data from the upstream network environment of the network environment; and thereafter, determine whether the data content of the upstream cross-network data changes during transmission. For example, if the network environment with a transmission sequence number of 3 is determined above as the network environment where the data content changes, then re-collect cross-network data from the network environment with a transmission sequence number of 2 is collected, and after being collected to the local data processing platform, the re-collected cross-network data is judged according to the method (1) above.

[0138] (4) When it is determined that the data content of the upstream cross-network data has not changed, the upstream cross-network data is regarded as cross-network data without any transmission error.

[0139] It should be noted that if the data content of the cross-network data is determined not to have changed during the transmission process in (1) above, the cross-network data is directly regarded as cross-network data with no transmission error.

[0140] In this embodiment, the above-mentioned cross-network data also includes metadata information of the data content, which includes the update cycle of the data content and the business time interval of the data content. For example, the data update cycle of the data content is monthly, and the business time interval is a certain month of a certain year.

[0141] Through the above-mentioned data verification method provided in this embodiment, it is possible to quickly determine whether a transmission error occurs during the transmission of cross-network data from a source system to a data processing platform in a local network environment, and the network environment in which a transmission error occurs during the transmission of cross-network data can be accurately located, thereby achieving re-collection of cross-network data at the most reasonable and accurate transmission location, and ultimately obtaining cross-network data without transmission errors.

[0142] 2. According to the data quality verification method of the local network environment, the cross-network data without transmission errors is verified for data quality to obtain cross-network data that meets the usage requirements of the local network environment.

[0143] After the cross-network data originating from a remote network environment passes the above-mentioned data verification, it can be guaranteed that the data content is consistent with the data content of the source network. However, it cannot be guaranteed that the cross-network data meets the data production requirements of the local data processing platform. Therefore, data quality verification is also required for the cross-network data that has not experienced transmission errors. This process must first set data quality audit rules based on the data production requirements of the local data processing platform. Specifically, data quality audit rules are set from five aspects: data standardization, completeness, consistency, timeliness, and correctness. The above settings can be made at the table level or the field level, and data quality audit rules can be divided into strong rules and weak rules based on the requirements for data usage.

[0144] After setting the data quality audit rules, a data quality audit task is generated through the audit task program and executed. The data quality audit task includes information such as the task name, data table name, resource name, resource type, partition information, and task status (pending, started, failed, or successfully executed). The process of executing the data quality audit task for the cross-network data without transmission errors is the process of verifying its data quality. If the data quality verification for the cross-network data without transmission errors passes, it is determined that the cross-network data meets the usage requirements of the local network environment.

[0145] After processing the cross-network data collected by the local data processing platform in the above manner to obtain cross-network data that meets the usage requirements of the local network environment, when starting the target task, the cross-network data can be used as benchmark data to judge whether the target data required to be processed by the target task has transmission errors during the transmission process, and whether it meets the usage requirements of the local network environment. If the above-mentioned cross-network data that meets the usage requirements of the local network environment contains the above-mentioned target data, and it is determined that the target data meets the data quality requirements of the local network environment, then the target task is started, the target data is processed, and the task scheduling for the target data is completed.

[0146] The data processing method provided in this embodiment obtains the dependency relationship between data tables, the dependency relationship between data tables and data tasks, and the dependency relationship between data tasks according to the data processing flow preset by the data processing platform, obtains the target data to be processed layer by layer from the remote network environment and the target task to be started according to the above dependency relationship, and then after determining that the target data to be processed by the target task exists on the data processing platform and that the target data meets the data quality requirements of the local network environment, starts the target task and completes the task scheduling for the target data. By using this method, the problem of being unable to perform normal task scheduling caused by scheduling tasks according to the agreed time when the time when the cross-network data cannot be determined to arrive at the data processing platform can be avoided, and the problem of errors in the subsequent data generated by the cross-network data caused by scheduling tasks according to the agreed time when the cross-network data cannot meet the data quality requirements of the local network environment can be avoided, thereby affecting the entire data processing process.

[0147] The second embodiment of the present application provides a data verification method. Figure 2 The data verification method flow chart provided in the second embodiment of this application is as follows: Figure 2 The method provided in this embodiment is described in detail.

[0148] like Figure 2 As shown, the data processing method provided in this embodiment includes the following steps:

[0149] S201, obtaining cross-network data.

[0150] Cross-network data may refer to data from a network environment other than the network environment in which the data processing platform is located. In this embodiment, the cross-network data includes data content and data verification information corresponding to the network environment through which the data content passes during transmission. In the process of data transmission from the source network environment to the network environment in which the data processing platform is located, a data verification information is recorded for each network through which the data passes. The data verification information includes network environment identification information that can record the transmission status of the data content and content identification information for identifying the data content. That is, each network environment generates corresponding network environment identification information and content identification information for the data content of the cross-network data. The network environment identification information can be information such as the network name, data transmission timestamp, and transmission sequence number corresponding to the network environment passed through. The transmission sequence number increases sequentially according to the network environment passed through during the cross-network data transmission process. The above-mentioned content identification information can be a hash value obtained by hashing the data content in each network environment passed through by the cross-network data, or a value of the data volume corresponding to the data content in each network environment passed through.

[0151] S202: Determine whether the data content changes during the transmission process.

[0152] The process of determining whether the data content has changed during transmission is essentially to determine whether transmission errors such as data distortion and data loss have occurred during transmission of the data content. The above-mentioned determination process is specifically as follows: based on the data content of the cross-network data, content identification information corresponding to the above-mentioned network environment is obtained for identifying the data content, for example, hash calculation is performed on the data content of the cross-network data in each network environment to obtain hash values corresponding to the local network environment and other network environments passed by the cross-network data; then, the content identification information corresponding to the local network environment is compared with the content identification information corresponding to other network environments in the above-mentioned data verification information, for example, the hash value corresponding to the local network environment is compared with the hash values corresponding to the data content of the cross-network data in other network environments, or the hash values corresponding to all network environments are compared together. If any two hash values are different, it is determined that the data content of the cross-network data has changed during transmission.

[0153] S203: If it is determined that the data content has changed during the transmission process, the network environment in which the data content was located when the change occurred is determined according to the network environment identification information.

[0154] In this embodiment, the above-mentioned network environment identification information also includes a transmission sequence number corresponding to the network environment, and the transmission sequence number increases in sequence according to the network environment passed through during the cross-network data transmission process. The network environment in which the data content changes based on the network environment identification information refers to determining the network environment with the smallest transmission sequence number among the network environments in which the data content changes as the network environment in which the data content changes. For example, according to the comparison result of the above-mentioned hash values, it can be seen that the transmission sequence number when the hash value begins to change is 3, that is, the same hash value corresponding to the network environments of transmission sequence number 1 and transmission sequence number 2 is different from the hash values corresponding to the subsequent network environments of transmission sequence numbers 4, 5, 6, etc., so the network environment with transmission sequence number 3 is determined to be the network environment in which the data content changes.

[0155] S204: Collect cross-network data from the upstream network environment of the network environment where the data content changed. For example, if the network environment with transmission sequence number 3 is determined to be the network environment where the data content changed, then recollect cross-network data from the network environment with transmission sequence number 2. After the data is collected and sent to the local data processing platform, step S202 may be re-executed.

[0156] The method provided in this embodiment can quickly determine whether a transmission error occurs during the transmission of cross-network data from a source system to a local data processing platform, and can accurately locate the network environment in which the transmission error occurs during the transmission of cross-network data, thereby achieving re-collection of cross-network data at the most reasonable and accurate transmission location, and ultimately obtaining cross-network data without transmission errors.

[0157] The above-mentioned first embodiment provides a data processing method. Correspondingly, the third embodiment of the present application also provides a data processing device. Since the device embodiment is basically similar to the method embodiment, the description is relatively simple. For details of the relevant technical features, please refer to the corresponding description of the method embodiment provided above. The following description of the device embodiment is merely illustrative.

[0158] Please refer to Figure 3 To understand this embodiment, Figure 3 A unit block diagram of the device provided in this embodiment, such as Figure 3 As shown, the device provided in this embodiment includes:

[0159] A target task and target data obtaining unit 301 is configured to obtain a target task and target data required for processing the target task, wherein the target data is derived from a first network environment;

[0160] The target task starting unit 302 is configured to start the target task after determining that the target data meets the data quality requirement of the second network environment.

[0161] Optionally, obtain target tasks, including:

[0162] According to the predetermined offline data processing process, a data dependency set is obtained;

[0163] Obtain the target task based on the data dependency set.

[0164] Optionally, according to a predetermined offline data processing process, a data dependency set is obtained, including: parsing the task statements corresponding to the offline data processing process to obtain the data dependency set; or, obtaining the data dependency set from task configuration information for the offline data processing process.

[0165] Optionally, the data dependency set includes a dependency set between data tables, a dependency set between data tables and data tasks, and a dependency set between data tasks. Obtaining the target task based on the data dependency set includes:

[0166] Obtain the basic data table according to the dependency relationship set between the data tables;

[0167] According to the basic data table and the dependency relationship set between the data table and the data task, the data task that depends on the basic data table is obtained;

[0168] A target task is obtained according to the data tasks that depend on the basic data table and the dependency set between the data tasks.

[0169] Optionally, obtain the target data required for processing the target task, including:

[0170] Obtaining a first business time interval set corresponding to target data to be processed by a target task; the first business time interval set includes a business time interval corresponding to at least one data table;

[0171] Obtain a second business time interval set corresponding to basic data in a basic data table; the second business time interval set includes a business time interval corresponding to at least one data table;

[0172] Determining whether the first service time interval set and the second service time interval set are the same;

[0173] If the first business time interval set is the same as the second business time interval set, the basic data in the basic data table is determined to be the target data.

[0174] Optionally, the dependency relationship set between the data table and the data task includes a business time parameter, which is used to represent a conversion relationship between a base time corresponding to the data task and a business time interval corresponding to the data to be processed by the data task. Obtaining a first business time interval set corresponding to the target data to be processed by the target task includes:

[0175] Obtain the benchmark time corresponding to the target task;

[0176] According to the reference time and business time parameters corresponding to the target task, a first business time interval set corresponding to the target data to be processed by the target task is obtained.

[0177] Optionally, obtain the benchmark time corresponding to the target task, including:

[0178] Obtaining basic data with the highest business data update frequency from basic data in a basic data table of the first network environment;

[0179] The update time of the business data of the basic data with the highest business data update frequency that is consistent with the business data required at the current time is used as the reference time corresponding to the target task.

[0180] Optionally, the method further includes: performing data verification on the cross-network data from the first network environment to obtain cross-network data without transmission errors; performing data quality verification on the cross-network data without transmission errors according to a data quality verification method of the second network environment to obtain cross-network data that meets the usage requirements of the second network environment;

[0181] Correspondingly, determining whether the target data meets the data quality requirements of the second network environment includes:

[0182] If the cross-network data that meets the usage requirements of the second network environment includes the target data, it is determined that the target data meets the data quality requirements of the second network environment.

[0183] Optionally, the cross-network data originating from the first network environment includes data content and data verification information corresponding to the network environment through which the data content passes during transmission, the data verification information including network environment identification information recording a transmission status that can indicate the data content. Performing data verification on the cross-network data originating from the first network environment to obtain cross-network data without transmission errors includes:

[0184] Determine whether the data content of cross-network data changes during transmission;

[0185] If there is a change, the network environment in which the data content is located when the change occurs is determined based on the network environment identification information that records the transmission status of the data content;

[0186] Collect upstream cross-network data from the upstream network environment of the network environment where the data content changes;

[0187] When it is determined that the data content of the upstream cross-network data has not changed, the upstream cross-network data is regarded as cross-network data in which no transmission error has occurred.

[0188] Optionally, determine whether the data content of cross-network data has changed during transmission

[0189] Obtaining content identification information corresponding to the network environment and used to identify the data content according to the data content;

[0190] The content identification information is compared, and if the content identification information is different, it is determined that the data content of the cross-network data has changed during the transmission process.

[0191] Optionally, obtaining content identification information corresponding to the network environment and used to identify the data content according to the data content includes:

[0192] Perform hash calculation on the data content to obtain a hash value corresponding to the network environment.

[0193] Optionally, the network environment identification information includes a transmission sequence number corresponding to the network environment, where the transmission sequence number increases sequentially according to the network environment passed through during the cross-network data transmission process. Determining the network environment in which the data content is located when the data content changes based on the network environment identification information that records the transmission status of the data content includes:

[0194] The network environment in which the data content changes and in which the transmission sequence number is the smallest is determined as the network environment in which the data content changes.

[0195] Optionally, determining whether the target data meets the data quality requirements of the second network environment includes:

[0196] Perform data verification on the target data to ensure that no transmission error occurs to the target data;

[0197] According to the data quality verification method of the second network environment, data quality verification is performed on the target data without any transmission errors to determine whether the target data meets the usage requirements of the second network environment.

[0198] Optionally, the first network environment is a remote network environment, and the second network environment is a local network environment.

[0199] Optionally, the first network environment is a public cloud network, and the second network environment is a private cloud network.

[0200] In the above embodiments, a data processing method and a data processing device are provided. In addition, a fourth embodiment of the present application further provides an electronic device. The embodiment of the electronic device is as follows:

[0201] Please refer to Figure 4 To understand this embodiment, Figure 4 A schematic diagram of an electronic device provided in this embodiment.

[0202] like Figure 4 As shown, the electronic device includes: a processor 401; a memory 402;

[0203] The memory 402 is used to store a data processing program. When the program is read and executed by the processor, it performs the following operations:

[0204] Obtaining a target task and target data required to be processed by the target task, where the target data comes from a first network environment;

[0205] If it is determined that the target data meets the data quality requirements of the second network environment, the target task is started. Optionally, obtaining the target task includes:

[0206] According to the predetermined offline data processing process, a data dependency set is obtained;

[0207] Obtain the target task based on the data dependency set.

[0208] Optionally, according to a predetermined offline data processing process, a data dependency set is obtained, including: parsing the task statements corresponding to the offline data processing process to obtain the data dependency set; or, obtaining the data dependency set from task configuration information for the offline data processing process.

[0209] Optionally, the data dependency set includes a dependency set between data tables, a dependency set between data tables and data tasks, and a dependency set between data tasks. Obtaining the target task based on the data dependency set includes:

[0210] Obtain the basic data table according to the dependency relationship set between the data tables;

[0211] According to the basic data table and the dependency relationship set between the data table and the data task, the data task that depends on the basic data table is obtained;

[0212] A target task is obtained according to the data tasks that depend on the basic data table and the dependency set between the data tasks.

[0213] Optionally, obtain the target data required for processing the target task, including:

[0214] Obtaining a first business time interval set corresponding to target data to be processed by a target task; the first business time interval set includes a business time interval corresponding to at least one data table;

[0215] Obtain a second business time interval set corresponding to basic data in a basic data table; the second business time interval set includes a business time interval corresponding to at least one data table;

[0216] Determining whether the first service time interval set and the second service time interval set are the same;

[0217] If the first business time interval set is the same as the second business time interval set, the basic data in the basic data table is determined to be the target data.

[0218] Optionally, the dependency set between the data table and the data task includes a business time parameter, which is used to represent the conversion relationship between the base time corresponding to the data task and the business time interval corresponding to the data to be processed by the data task. Obtaining the first business time interval set corresponding to the target data to be processed by the target task includes: obtaining the base time corresponding to the target task; and obtaining the first business time interval set corresponding to the target data to be processed by the target task based on the base time and business time parameter corresponding to the target task.

[0219] Optionally, obtaining the benchmark time corresponding to the target task includes: obtaining basic data with the highest business data update frequency from basic data in a basic data table from the first network environment;

[0220] The update time of the business data of the basic data with the highest business data update frequency that is consistent with the business data required at the current time is used as the reference time corresponding to the target task.

[0221] Optionally, the method further includes: performing data verification on the cross-network data from the first network environment to obtain cross-network data without transmission errors; performing data quality verification on the cross-network data without transmission errors according to a data quality verification method of the second network environment to obtain cross-network data that meets the usage requirements of the second network environment;

[0222] Correspondingly, determining whether the target data meets the data quality requirements of the second network environment includes: if the cross-network data that meets the usage requirements of the second network environment includes the target data, then determining that the target data meets the data quality requirements of the second network environment.

[0223] Optionally, the cross-network data originating from the first network environment includes data content and data verification information corresponding to the network environment through which the data content passes during transmission, the data verification information including network environment identification information recording a transmission status that can indicate the data content. Performing data verification on the cross-network data originating from the first network environment to obtain cross-network data without transmission errors includes:

[0224] Determine whether the data content of cross-network data changes during transmission;

[0225] If there is a change, the network environment in which the data content is located when the change occurs is determined based on the network environment identification information that records the transmission status of the data content;

[0226] Collect upstream cross-network data from the upstream network environment of the network environment where the data content changes;

[0227] When it is determined that the data content of the upstream cross-network data has not changed, the upstream cross-network data is regarded as cross-network data in which no transmission error has occurred.

[0228] Optionally, determine whether the data content of cross-network data has changed during transmission

[0229] Obtaining content identification information corresponding to the network environment and used to identify the data content according to the data content;

[0230] The content identification information is compared, and if the content identification information is different, it is determined that the data content of the cross-network data has changed during the transmission process.

[0231] Optionally, obtaining content identification information corresponding to the network environment and used to identify the data content according to the data content includes:

[0232] Perform hash calculation on the data content to obtain a hash value corresponding to the network environment.

[0233] Optionally, the network environment identification information includes a transmission sequence number corresponding to the network environment, where the transmission sequence number increases sequentially according to the network environment passed through during the cross-network data transmission process. Determining the network environment in which the data content is located when the data content changes based on the network environment identification information that records the transmission status of the data content includes:

[0234] The network environment in which the data content changes and in which the transmission sequence number is the smallest is determined as the network environment in which the data content changes.

[0235] Optionally, determining whether the target data meets the data quality requirements of the second network environment includes:

[0236] Perform data verification on the target data to ensure that no transmission error occurs to the target data;

[0237] According to the data quality verification method of the second network environment, data quality verification is performed on the target data without any transmission errors to determine whether the target data meets the usage requirements of the second network environment.

[0238] Optionally, the first network environment is a remote network environment, and the second network environment is a local network environment.

[0239] Optionally, the first network environment is a public cloud network, and the second network environment is a private cloud network.

[0240] The above-mentioned second embodiment provides a data verification method. Correspondingly, the fifth embodiment of the present application also provides a data verification device. Since the device embodiment is basically similar to the method embodiment, the description is relatively simple. For details of the relevant technical features, please refer to the corresponding description of the method embodiment provided above. The following description of the device embodiment is merely illustrative.

[0241] Please refer to Figure 5 To understand this embodiment, Figure 5 A unit block diagram of the device provided in this embodiment, such as Figure 5 As shown, the device provided in this embodiment includes:

[0242] The cross-network data obtaining unit 501 is configured to obtain cross-network data; the cross-network data includes data content and data verification information corresponding to the network environment through which the data content passes during transmission, the data verification information including network environment identification information that can indicate the transmission status of the data content;

[0243] A change determination unit 502 is used to determine whether the data content has changed during the transmission process;

[0244] The network environment determining unit 503 is configured to, after determining that the data content has changed during transmission, determine the network environment in which the data content is located when the data content changes according to the network environment identification information.

[0245] Optionally, determine whether the data content has changed during transmission, including:

[0246] Obtaining content identification information corresponding to the network environment and used to identify the data content according to the data content;

[0247] The content identification information is compared, and if the content identification information is different, it is determined that the data content of the cross-network data has changed during the transmission process.

[0248] Optionally, the network environment identification information includes a transmission sequence number corresponding to the network environment. The transmission sequence number increases in sequence according to the network environment passed through during the cross-network data transmission process. Determining the network environment in which the data content changes based on the network environment identification information includes:

[0249] The network environment in which the data content changes and in which the transmission sequence number is the smallest is determined as the network environment in which the data content changes.

[0250] Optionally, the method further includes: collecting cross-network data from an upstream network environment of the network environment where the data content changes, from the network environment where the data content changes.

[0251] In the above embodiments, a data verification method and a data verification device are provided. In addition, the sixth embodiment of the present application further provides an electronic device. The embodiment of the electronic device is as follows:

[0252] Please refer to Figure 6 To understand this embodiment, Figure 6 This is a schematic diagram of an electronic device provided in this embodiment.

[0253] like Figure 6 As shown, the electronic device includes: a processor 601; a memory 602;

[0254] The memory 602 is used to store a data verification program. When the program is read and executed by the processor, it performs the following operations:

[0255] Obtaining cross-network data; the cross-network data includes data content and data verification information corresponding to the network environment through which the data content passes during transmission, the data verification information including network environment identification information that can indicate the transmission status of the data content;

[0256] Determine whether the data content has changed during transmission;

[0257] If it is determined that the data content has changed during the transmission process, the network environment in which the data content was located when the change occurred is determined based on the network environment identification information.

[0258] Optionally, determine whether the data content has changed during transmission, including:

[0259] Obtaining content identification information corresponding to the network environment and used to identify the data content according to the data content;

[0260] The content identification information is compared, and if the content identification information is different, it is determined that the data content of the cross-network data has changed during the transmission process.

[0261] Optionally, the network environment identification information includes a transmission sequence number corresponding to the network environment. The transmission sequence number increases in sequence according to the network environment passed through during the cross-network data transmission process. Determining the network environment in which the data content changes based on the network environment identification information includes:

[0262] The network environment in which the data content changes and in which the transmission sequence number is the smallest is determined as the network environment in which the data content changes.

[0263] Optionally, the method further includes: collecting cross-network data from an upstream network environment of the network environment where the data content changes, from the network environment where the data content changes.

[0264] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0265] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0266] 1. Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include non-transitory media such as modulated data signals and carrier waves.

[0267] 2. Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0268] Although the present application is disclosed as above with the preferred embodiments, it is not intended to limit the present application. Any person skilled in the art may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

Claims

1. A data processing method, characterized in that: include: Obtaining a target task and target data required to be processed by the target task, wherein the target data is derived from a first network environment; If it is determined that the target data meets the data quality requirements of the second network environment, starting the target task, the data quality requirements of the second network environment being passing data verification and complying with the data quality verification of the second network environment; The obtaining of target data required for processing the target task includes: Obtaining a first business time interval set corresponding to target data to be processed by the target task; the first business time interval set includes a business time interval corresponding to at least one data table; Obtaining a second business time interval set corresponding to basic data in a basic data table; the second business time interval set includes a business time interval corresponding to at least one data table, the basic data table being obtained based on a dependency relationship set between data tables in a data dependency relationship set, and the basic data table being a data table without any dependencies; Determining whether the first service time interval set is the same as the second service time interval set; If the first service time interval set is the same as the second service time interval set, the basic data in the basic data table is determined to be the target data.

2. The method according to claim 1, characterized in that The obtaining of the target task includes: According to the predetermined offline data processing process, a data dependency set is obtained; According to the data dependency set, a target task is obtained.

3. The method according to claim 2, characterized in that The step of obtaining a data dependency set according to a predetermined offline data processing process includes: Parsing the task statements corresponding to the offline data processing process to obtain a data dependency set; Alternatively, the data dependency set is obtained from task configuration information for the offline data processing flow.

4. The method according to claim 2, characterized in that The data dependency set includes a dependency set between data tables, a dependency set between data tables and data tasks, and a dependency set between data tasks. Obtaining a target task based on the data dependency set includes: Obtaining a basic data table according to a set of dependency relationships between the data tables; Obtaining data tasks that depend on the basic data table according to the basic data table and a set of dependency relationships between the data table and the data tasks; The target task is obtained according to the data task that depends on the basic data table and the dependency relationship set between the data tasks.

5. The method according to claim 1, wherein The dependency relationship set between the data table and the data task includes a business time parameter, which is used to represent the conversion relationship between the base time corresponding to the data task and the business time interval corresponding to the data to be processed by the data task. The first business time interval set corresponding to the target data to be processed by the target task is obtained, including: Obtaining a benchmark time corresponding to the target task; A first business time interval set corresponding to the target data to be processed by the target task is obtained according to the reference time corresponding to the target task and the business time parameter.

6. The method according to claim 5, characterized in that The obtaining of the reference time corresponding to the target task includes: Obtaining basic data with the highest business data update frequency from the basic data in the basic data table from the first network environment; The update time of the business data of the basic data with the highest business data update frequency that is consistent with the business data required at the current time is used as the reference time corresponding to the target task.

7. The method according to claim 1, characterized in that Also includes: Performing data verification on the cross-network data from the first network environment to obtain cross-network data without any transmission errors; Performing data quality verification on the cross-network data without transmission errors according to the data quality verification method of the second network environment to obtain cross-network data that meets the usage requirements of the second network environment; Correspondingly, determining whether the target data meets the data quality requirements of the second network environment includes: If the cross-network data that meets the usage requirements of the second network environment includes the target data, it is determined that the target data meets the data quality requirements of the second network environment.

8. The method according to claim 7, characterized in that The cross-network data originating from the first network environment includes data content and data verification information corresponding to the network environment through which the data content passes during transmission, the data verification information including network environment identification information recording a transmission status of the data content, and performing data verification on the cross-network data originating from the first network environment to obtain cross-network data without transmission errors includes: Determining whether the data content of the cross-network data changes during transmission; If there is a change, determining the network environment in which the data content is located when the change occurs based on the network environment identification information recording the transmission status of the data content; Collecting upstream cross-network data from an upstream network environment of the network environment where the data content changes, from the network environment where the data content changes; When it is determined that the data content of the upstream cross-network data has not changed, the upstream cross-network data is regarded as cross-network data in which no transmission error has occurred.

9. The method according to claim 8, characterized in that The determining whether the data content of the cross-network data changes during the transmission process includes: Obtaining content identification information corresponding to the network environment and used to identify the data content according to the data content; The content identification information is compared, and if the content identification information is different, it is determined that the data content of the cross-network data has changed during the transmission process.

10. The method according to claim 9, characterized in that The obtaining, according to the data content, content identification information corresponding to the network environment and used to identify the data content includes: Perform a hash calculation on the data content to obtain a hash value corresponding to the network environment.

11. The method according to claim 8, characterized in that The network environment identification information includes a transmission sequence number corresponding to the network environment, the transmission sequence number being incremented sequentially according to the network environment passed through during the cross-network data transmission process, and determining the network environment in which the data content is located when the data content changes based on the network environment identification information recording the transmission status of the data content, including: The network environment in which the data content changes and in which the transmission sequence number is the smallest is determined as the network environment in which the data content changes.

12. The method according to claim 1, characterized in that Determining that the target data meets the data quality requirements of the second network environment includes: Performing data verification on the target data to determine that no transmission error occurs in the target data; According to the data quality verification method of the second network environment, data quality verification is performed on the target data without any transmission error to determine that the target data meets the usage requirements of the second network environment.

13. The method according to any one of claims 1 to 12, characterized in that The first network environment is a remote network environment, and the second network environment is a local network environment.

14. The method according to any one of claims 1 to 12, characterized in that The first network environment is a public cloud network environment, and the second network environment is a private cloud network environment.

15. A data verification method, applied to the data processing method according to any one of claims 1 to 14, characterized in that: include: Obtaining cross-network data; the cross-network data includes data content and data verification information corresponding to the network environment passed by the data content during transmission, the data verification information including network environment identification information that can indicate the transmission status of the data content; Determining whether the data content changes during transmission; If it is determined that the data content changes during the transmission process, determining the network environment in which the data content is located when the change occurs based on the network environment identification information; The determining whether the data content changes during the transmission process includes: Obtaining content identification information corresponding to the network environment and used to identify the data content according to the data content; The content identification information is compared, and if the content identification information is different, it is determined that the data content of the cross-network data has changed during the transmission process.

16. The method according to claim 15, characterized in that The network environment identification information includes a transmission sequence number corresponding to the network environment, where the transmission sequence number increases sequentially according to the network environment passed through during the cross-network data transmission process. Determining the network environment in which the data content is located when the data content changes based on the network environment identification information includes: The network environment in which the data content changes and in which the transmission sequence number is the smallest is determined as the network environment in which the data content changes.

17. The method according to claim 15, characterized in that Also includes: The cross-network data from the upstream network environment of the network environment is collected from the network environment where the data content changes.

18. A data processing device, characterized in that: include: A target task and target data obtaining unit, configured to obtain a target task and target data required to be processed by the target task, wherein the target data is derived from the first network environment; a target task starting unit, configured to start the target task after determining that the target data meets the data quality requirement of the second network environment, wherein the data quality requirement of the second network environment is to pass data verification and comply with the data quality verification of the second network environment; The target task and target data obtaining unit is specifically used to: Obtaining a first business time interval set corresponding to target data to be processed by the target task; the first business time interval set includes a business time interval corresponding to at least one data table; Obtaining a second business time interval set corresponding to basic data in a basic data table; the second business time interval set includes a business time interval corresponding to at least one data table, the basic data table being obtained based on a dependency relationship set between data tables in a data dependency relationship set, and the basic data table being a data table without any dependencies; Determining whether the first service time interval set is the same as the second service time interval set; If the first service time interval set is the same as the second service time interval set, the basic data in the basic data table is determined to be the target data.

19. An electronic device, characterized in that: include: processor; The memory is used to store a data processing program, and when the program is read and executed by the processor, the program performs the following operations: Obtaining a target task and target data required to be processed by the target task, wherein the target data is derived from a first network environment; If it is determined that the target data meets the data quality requirements of the second network environment, starting the target task, the data quality requirements of the second network environment being passing data verification and complying with the data quality verification of the second network environment; The obtaining of target data required for processing the target task includes: Obtaining a first business time interval set corresponding to target data to be processed by the target task; the first business time interval set includes a business time interval corresponding to at least one data table; Obtaining a second business time interval set corresponding to basic data in a basic data table; the second business time interval set includes a business time interval corresponding to at least one data table, the basic data table being obtained based on a dependency relationship set between data tables in a data dependency relationship set, and the basic data table being a data table without any dependencies; Determining whether the first service time interval set is the same as the second service time interval set; If the first service time interval set is the same as the second service time interval set, the basic data in the basic data table is determined to be the target data.

20. A data verification device, wherein the execution method of the device is applied to the data processing method according to any one of claims 1 to 14, characterized in that: include: a cross-network data obtaining unit, configured to obtain cross-network data; the cross-network data comprising data content and data verification information corresponding to the network environment through which the data content passes during transmission, the data verification information comprising network environment identification information recording a transmission status of the data content; a change determination unit, configured to determine whether the data content has changed during transmission; a network environment determining unit, configured to, after determining that the data content changes during transmission, determine the network environment in which the data content is located when the data content changes according to the network environment identification information; The change judgment unit is specifically used to: Obtaining content identification information corresponding to the network environment and used to identify the data content according to the data content; The content identification information is compared, and if the content identification information is different, it is determined that the data content of the cross-network data has changed during the transmission process.

21. An electronic device, wherein the electronic device execution method is applied to the data processing method according to any one of claims 1 to 14, characterized in that: include: processor; The memory is used to store a data processing program, and when the program is read and executed by the processor, the program performs the following operations: Obtaining cross-network data; the cross-network data includes data content and data verification information corresponding to the network environment passed by the data content during transmission, the data verification information including network environment identification information that can indicate the transmission status of the data content; Determining whether the data content changes during transmission; If it is determined that the data content changes during the transmission process, determining the network environment in which the data content is located when the change occurs based on the network environment identification information; The determining whether the data content changes during the transmission process includes: Obtaining content identification information corresponding to the network environment and used to identify the data content according to the data content; The content identification information is compared, and if the content identification information is different, it is determined that the data content of the cross-network data has changed during the transmission process.

22. A computer storage medium, characterized in that The computer storage medium includes instructions, and when the instructions are read and executed by a processor, the method according to any one of claims 1 to 14 or any one of claims 15 to 17 is executed.

23. A computer program product, characterized in that The computer program product comprises instructions, and when the instructions are read and executed by a processor, the method according to any one of claims 1 to 14 or any one of claims 15 to 17 is executed.

Images