Custom data processing conversion flow construction method

Abstract

The invention relates to the technical field of big data, in particular to a self-defined data processing and conversion process construction method, which comprises the following steps of: creating a node template, creating an input node, a conversion node and an output node based on Apache Flink, creating an Flink Type dynamic table structure after a target data source is loaded in the input node, creating an Flink Type dynamic table structure after a target data source is loaded in the conversion node, and creating an Flink Type dynamic table structure after the target data source is loaded in the conversion node. In the aspect of classification and abstraction of conversion nodes, different field types of Flink Type are taken as the basis; the data processing flow construction comprises demand analysis, processing node instantiation, node flow combination and node flow execution and analysis, and the processing node instantiation process is based on the node template; according to the method, the universality and reusability of the conversion node are improved, the data processing flow meeting specific business requirements can be quickly constructed through standardized demand analysis and node instantiation, and meanwhile, the maintainability and expandability of the flow are improved through combination and execution analysis of the node flow.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a method for constructing a custom data processing and conversion process. Background Art

[0002] With the continuous development and upgrading of big data technology, data collection and data processing technologies have been able to meet the adaptation needs of various source data and heterogeneous data, and provide a rich change interface for data flow conversion and processing operations. In a data-driven business environment, quickly building a data processing task with a clear structure and complete functions has become the primary problem that data engineers need to face when handling actual tasks.

[0003] In custom task processes, DAG (directed acyclic graph) flows are widely used. DAG consists of a set of vertices and a set of directed edges. These edges do not form any loops, which means that starting from any vertex in the graph along the directed edges, you cannot return to the vertex. In various scheduling and process processing tasks, DAG graphs usually break up various abstract task goals onto various nodes in the graph. Since nodes can be standardized and reused, this model is efficient, fast, and highly malleable in creating complex task processes.

[0004] As a kind of custom task process, data processing flow is based on DAG. It studies how to standardize the construction of a set of data processing process nodes and provide a complete ETL (Extract-Transform-Load, which is the process of extracting, transforming and loading data from the source to the destination) processing process creation solution, which is of great significance to the realization of efficient data processing.

[0005] The existing technology builds customized functions for customized processing tasks based on data processing programming interfaces and specific processing tasks, including the following specific solutions: 1. Analyze the main structure of the specific data processing task, including three parts: data input source, data format conversion, and data output target. 2. For source data, data engineers manually or automatically write connection programs through various collection tools for manual access. 3. After obtaining the upstream metadata, start designing data structure conversion according to the specific business. Generally, the conversion flow is designed through various custom input and output functions (for example, using the Scala language). 4. After completing the data flow conversion program, the engineer selects the target address to be stored (generally a data warehouse), and completes the data persistence after comparing the specific structure after processing. 5. Finally, throughout the entire specific business process of processing data, the complete process of data extraction, conversion, and loading is completed to meet specific business needs.

[0006] Disadvantages of the solution:

[0007] 1. Strong business relevance and high coupling. Although the above solution implements a complete set of data processing (ETL) processes for specific businesses, the tasks in the design are too closely related to the specific business and the coupling is too high.

[0008] 2. Poor reusability. A large number of customized programs are designed according to specific business needs. After a processing task is completed, the program often needs to be modified at the hard-coded level before it can be applied to other processing tasks. It is difficult to write once and reuse multiple times between the various process components of the processing task.

[0009] 3. Inconvenient to expand. The granularity of splitting a specific data processing task is too large. Taking a specific data processing task as the basic unit, it is impossible to flexibly modify it according to changes in business needs, which is not convenient for the expansion of data processing tasks.

[0010] The existing data processing flow construction methods have limitations in flexibility, reusability and scalability, and are difficult to adapt to the rapidly changing business needs and the development of data processing technology. Therefore, there is an urgent need for a new data processing flow construction method that can provide higher flexibility, better reusability and stronger scalability to meet the needs of modern data processing. Summary of the invention

[0011] In order to solve the problems mentioned in the above background technology center, a method for building a custom data processing and conversion process is disclosed, which includes:

[0012] Node template creation is based on Apache Flink, including creating input nodes, conversion nodes, and output nodes. When creating the input node, after the target data source is loaded, a FlinkType dynamic table structure is generated. When creating the conversion node, the classification and abstraction of the conversion node are based on different field types of Flink Type. The beneficial effect of this technical solution is that by using the dynamic table structure of Apache Flink, various data sources can be flexibly processed. At the same time, the versatility and reusability of the conversion node are improved through classification and abstraction based on field types.

[0013] The data processing flow is constructed, including demand analysis, processing node instantiation, node flow combination, node flow execution and analysis, and the processing node instantiation process is based on the node template. The beneficial effect of this solution is that through standardized demand analysis and node instantiation, a data processing flow that meets specific business needs can be quickly constructed, and the combination and execution analysis of node flows improves the maintainability and scalability of the flow.

[0014] Preferably, in creating the input node, the dynamic table field conversion part is the inputFieldName, inputFieldType, outFieldName and outFieldType structure mode required for dynamic table flow. The beneficial effect of this solution is that by clarifying the structure mode of field conversion, the flow and conversion process of data can be more accurately controlled, thereby improving the accuracy and efficiency of data processing.

[0015] Preferably, when creating the conversion node, it is encapsulated as Concat Node and TrimNode according to TableAPI function. The beneficial effect of this solution is that by encapsulating TableAPI function, the creation process of the conversion node can be simplified, and a series of predefined conversion operations are provided, making data conversion more flexible and efficient.

[0016] Preferably, in creating the output node, after the processing engine calculation is completed, the calculation process is allowed to further process the output data according to the configured custom post-event information. The beneficial effect of this solution is that by introducing the custom post-event information, the output data can be processed more flexibly to meet the data output requirements in different scenarios.

[0017] Preferably, in the demand analysis, for input nodes, the starting data source information is extracted to determine the initial structure of the data processing task; for conversion nodes, the aggregation operation required for the task is extracted according to the target conversion data type and structure. The beneficial effect of this solution is that through detailed demand analysis, it can ensure that each link of the data processing process can accurately meet business needs, while improving the adaptability and flexibility of the process.

[0018] Preferably, in the demand analysis, for the output node, the target data source information is analyzed to determine whether the task needs to set a post-execution condition, and if the post-condition is required, how to fill in the post-task information. The beneficial effect of this solution is that by analyzing the target data source and the post-execution condition, it can be ensured that the output stage of the data processing flow can meet specific business rules and requirements.

[0019] Preferably, in the node process combination, DAG is used as a processing container for combining and connecting input nodes, conversion nodes, and output nodes. By splicing specific nodes in the form of a task flow graph, and saving the forward and reverse relationships between vertices and edges of specific nodes in the DAG context, the process creation of the entire custom processing task is completed. The beneficial effect of this solution is that through the DAG mode, the data processing process can be clearly defined and organized, while ensuring the acyclicity and efficiency of the process.

[0020] Preferably, during the execution and analysis of the node flow, after executing the data processing task flow, when an abnormality occurs in the process, the Yarn operation log is analyzed for reverse backtracking. The beneficial effect of this solution is that by analyzing the operation log, problems in the data processing process can be quickly located and solved, thereby improving the stability and reliability of the system.

[0021] A computer device is also disclosed, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method for constructing a custom data processing and conversion process when executing the computer program. By integrating the method into the computer device, the method for constructing a data processing process can be conveniently deployed and used in different computing environments.

[0022] A computer-readable storage medium is also disclosed, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the method for constructing a custom data processing and conversion process is implemented. Through the program on the storage medium, the method for constructing a data processing process can be easily distributed and installed, thereby improving its usability and popularity. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 It is a flow chart of embodiment 1 of the present invention. DETAILED DESCRIPTION

[0025] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0026] In order to solve the defects of the prior art mentioned in the background technology center, this embodiment provides a method for constructing a custom data processing and conversion process, which includes:

[0027] 1. Node template creation, based on Apache Flink, includes creating input nodes, conversion nodes, and output nodes. When creating the input node, after the target data source is loaded, a Flink Type dynamic table structure is generated. When creating the conversion node, the classification and abstraction of the conversion node are based on different field types of Flink Type. The beneficial effect of this technical solution is that by using the dynamic table structure of Apache Flink, various data sources can be flexibly processed. At the same time, the versatility and reusability of the conversion node are improved through classification and abstraction based on field types.

[0028] When creating an input node, the machine learning algorithm is used to automatically extract and analyze the features of the data source. The clustering algorithm is used to group the different features of the data source to optimize the data loading strategy. The formula of the clustering algorithm can be expressed as:

[0029]

[0030] Among them, C represents the cluster center, x i represents a data point, μ j represents the i-th cluster center, n is the number of data points, and k is the number of clusters; in this way, the configuration of the input node is automatically adjusted to adapt to different data source characteristics;

[0031] In the process of creating the transformation node, an automatic feature engineering method based on deep learning is introduced. An autoencoder is used to learn the low-dimensional representation of the data, thereby extracting the most useful features for the transformation operation. The objective function of the autoencoder can be expressed as:

[0032] L=XD(E(X))2

[0033] Among them, X is the input data, E is the encoder, and D is the decoder. By minimizing the reconstruction error L, the autoencoder can learn the effective representation of the data, thereby guiding the field conversion and type conversion of the conversion node;

[0034] In the design of output nodes, reinforcement learning is combined to optimize the data output strategy. For example, the reinforcement learning model is used to determine at what point in time and in what format to output data in order to maximize the overall performance of the system. The objective function of reinforcement learning can be expressed as:

[0035]

[0036] Among them, π is the policy, τ is the trajectory, and c t is the state at time t, a tis the action taken at time t, r is the reward function, and γ is the discount factor. The model is trained to find the optimal output policy by maximizing the expected return J(π).

[0037] 2. Data processing flow construction, including demand analysis, processing node instantiation, node flow combination, node flow execution and analysis, the processing node instantiation process is based on the node template. The beneficial effect of this solution is that through standardized demand analysis and node instantiation, a data processing flow that meets specific business needs can be quickly constructed, while the combination and execution analysis of node flows improves the maintainability and scalability of the flow.

[0038] In the demand analysis phase, data mining technology is used to identify potential patterns and associations in data sources. The association rules between different data fields are discovered through the association rule learning algorithm, the Apriori algorithm in this embodiment. The goal of the Apriori algorithm is to find all frequent item sets, and its formula can be expressed as:

[0039]

[0040] Among them, L k is the frequent itemset of the kth iteration, I is the set of all items, support(X) is the frequency of item set X in the dataset, and minSupport is the minimum support threshold defined by the user.

[0041] In the node process combination stage, the shortest path algorithm in graph theory, such as Dijkstra algorithm, is applied to optimize the connection order between nodes, thereby reducing the delay of data transmission. The formula of Dijkstra algorithm can be expressed as:

[0042] dist[v]=min u∈V (dist[u]+weight(u,v))

[0043] Among them, dist[v] is the shortest distance from the source point to the node v, V is the node set, and weight(u, v) is the weight of the edge from node u to node v.

[0044] In the node flow execution and analysis stage, real-time monitoring and anomaly detection mechanisms are introduced. For example, the stability of the process is monitored through the control chart in statistical process control (SPC). The calculation formula of the control chart can be expressed as:

[0045]

[0046] in, is the sample mean, R is the sample range, and A2 is a constant that depends on the sample size. By monitoring whether the data points are within the control limits, abnormal situations can be discovered and handled in a timely manner.

[0047] Preferably, in creating the input node, the dynamic table field conversion part is the inputFieldName (input field name), inputFieldType (input field type), outFieldName (output field name) and outFieldType (output field type) structure mode required for dynamic table flow. The beneficial effect of this solution is that by clarifying the structure mode of field conversion, the flow and conversion process of data can be more accurately controlled, thereby improving the accuracy and efficiency of data processing.

[0048] The information gain ratio is used to select the optimal field for conversion. The calculation formula of the information gain ratio is:

[0049]

[0050] Among them, Gain(D, a) is the information gain, IV(a) is the intrinsic value of attribute a, and D is the data set.

[0051] The calculation formula of information gain is:

[0052]

[0053] Among them, Ent(D) is the entropy of data set D, D v is a sample subset with value v on feature a, and V is the number of values ​​of feature a. In this way, the most meaningful fields for data conversion are automatically selected, thereby improving the efficiency and accuracy of data processing.

[0054] Preferably, when creating the conversion node, it is encapsulated as Concat Node (string concatenation node) and Trim Node (string trimming node) according to the TableAPI function. The beneficial effect of this solution is that by encapsulating the TableAPI function, the creation process of the conversion node is simplified, and a series of predefined conversion operations are provided, making data conversion more flexible and efficient. A neural network is used to automatically learn the rules of feature conversion. The forward propagation formula of the neural network can be expressed as:

[0055]

[0056] in, is the activation value of the ith node in layer l,

[0057] is the weight connecting the jth node in the l-1th layer and the ith node in the lth layer,

[0058] is the bias of the ith node in the lth layer, and f is the activation function. By training the neural network, complex transformation relationships in the data are automatically discovered.

[0059] Preferably, in creating the output node, after the processing engine calculation is completed, the calculation process is allowed to further process the output data according to the configured custom post-event information. The beneficial effect of this solution is that by introducing custom post-event information, the output data can be processed more flexibly to meet the data output requirements in different scenarios. The ChatAnthropic API is used to implement streaming transmission of model output, including synchronous and asynchronous streaming transmission, and the use of event streams.

[0060] Preferably, in the demand analysis, for input nodes, the starting data source information is extracted to determine the initial structure of the data processing task; for conversion nodes, the aggregation operation required for the task is extracted according to the target conversion data type and structure. The beneficial effect of this solution is that through detailed demand analysis, each link of the data processing process can accurately meet the business needs, while improving the adaptability and flexibility of the process.

[0061] Use machine learning algorithms to classify and extract features from data sources. Train a classifier to automatically identify the type of data source (such as structured databases, unstructured text, etc.) and extract key attributes to determine the initial structure of the data processing task.

[0062] In the demand analysis of the transformation node, data mining technology is used to identify potential patterns in the data and recommend appropriate aggregation operations. Cluster analysis is used to discover natural groupings in the data and recommend grouping, aggregation and other operations based on them.

[0063] Preferably, in the demand analysis, for the output node, it includes analyzing the target data source information, judging whether the task needs to set post-execution conditions, and how to fill in the post-task information when the post-conditions are required. The beneficial effect of this solution is that by analyzing the target data source and post-execution conditions, it is ensured that the output stage of the data processing flow can meet specific business rules and requirements. In the node process combination, DAG is used as a processing container that combines and connects input nodes, conversion nodes, and output nodes. By splicing specific nodes in the form of a task flow graph, and saving the forward and reverse relationships between vertices and vertices and edges of specific nodes in the DAG context, the process creation of the entire custom processing task is completed. The beneficial effect of this solution is that through the DAG model, the data processing flow can be clearly defined and organized, while ensuring the acyclicity and efficiency of the flow.

[0064] In the technical expansion of node process combination, this embodiment takes the following measures to further refine the specific implementation: implement a dynamic DAG construction system that can automatically adjust the DAG structure according to real-time data flow and task dependencies. This is achieved by introducing an event-driven architecture so that the DAG can respond to changes in data flow and reconfigure the dependencies between nodes. Use machine learning technology to analyze the dependencies between tasks and predict the order and resource requirements of task execution. This is achieved by training a graph neural network (GNN), which can learn the complex relationships between nodes and provide optimization suggestions for task scheduling. Develop an intelligent task scheduler that can optimize the arrangement of tasks in the DAG based on task priority, resource requirements, and expected execution time. This is achieved by implementing an optimizer based on a genetic algorithm that can find the most efficient task arrangement on the premise of satisfying all dependencies. Integrate fault recovery and fault tolerance mechanisms in the DAG to ensure that tasks can be quickly recovered and rescheduled when a node fails. This is achieved by implementing a monitoring system that can monitor node status in real time and automatically trigger a recovery process when a fault is detected. Create a visual DAG editor that allows users to intuitively design and adjust data processing processes through a graphical interface. This editor provides drag-and-drop functionality, as well as real-time preview of DAG structure and task dependencies, making process design more intuitive and easy to understand. Implement a performance monitoring system to collect and analyze key performance indicators during DAG execution. This is achieved by integrating a time series database and visualization tools, allowing users to monitor process performance and quickly identify bottlenecks.

[0065] Preferably, in the execution and analysis of the node flow, after executing the data processing task flow, when an abnormality occurs in the process, the Yarn operation log is analyzed for reverse tracing. The beneficial effect of this solution is that by analyzing the operation log, the problems in the data processing process can be quickly located and solved, and the stability and reliability of the system can be improved. Anomaly detection algorithm: an anomaly detection algorithm based on machine learning is introduced to predict the overdue anomalies and process behavior anomalies in the business process in real time by mining the log records of process execution and the activity execution time information. This method can effectively find out the processes with potential overdue anomalies and the processes with abnormal behavior. Using the log aggregation function of Yarn, the logs of each container are centrally managed for easy analysis. By setting yarn.log-aggregation-enable to true, the logs can be collected and aggregated to the file system after the execution is completed, such as HDFS, so that users can view the logs. In this embodiment, natural language processing (NLP) technology is used to conduct in-depth analysis of the Yarn operation log to automatically identify error modes and performance bottlenecks. And an automated fault recovery mechanism is implemented. When a process abnormality is detected, the system automatically tries predefined recovery strategies, such as retrying failed tasks, reallocating resources, etc.

[0066] A computer device is also disclosed, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method for constructing a custom data processing and conversion process when executing the computer program. By integrating the method into the computer device, the method for constructing a data processing process can be conveniently deployed and used in different computing environments.

[0067] And a corresponding computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the method for constructing a custom data processing and conversion process is implemented. Through the program on the storage medium, the method for constructing a data processing process can be easily distributed and installed, thereby improving its usability and popularity.

[0068] In some embodiments, the computer readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or various devices including one or any combination of the above memories. The computer may be various computing devices including intelligent terminals and servers.

[0069] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0070] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0071] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0072] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0073] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0074] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a multimedia terminal device (which can be a mobile phone, a computer, a television receiver, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0075] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for constructing a custom data processing and conversion process, characterized in that: include: Node template creation, based on Apache Flink, includes creating input nodes, conversion nodes, and output nodes. When creating the input node, after the target data source is loaded, a FlinkType dynamic table structure is generated. When creating the conversion node, the classification and abstraction of the conversion node are based on different field types of Flink Type. The data processing flow is constructed, including demand analysis, processing node instantiation, node flow combination, node flow execution and analysis, and the processing node instantiation process is based on the node template.

2. The method for constructing a custom data processing and conversion process according to claim 1, characterized in that: In creating the input node, the dynamic table field conversion part is the inputFieldName, inputFieldType, outFieldName and outFieldType structure mode required for dynamic table flow.

3. The method for constructing a custom data processing and conversion process according to claim 1, characterized in that: When creating the conversion node, it is encapsulated into ConcatNode and Trim Node according to the TableAPI function.

4. The method for constructing a custom data processing and conversion process according to claim 1, characterized in that: In creating the output node, after the processing engine calculation is completed, the calculation process is allowed to further process the output data according to the configured custom post-event information.

5. The method for constructing a custom data processing and conversion process according to any one of claims 1 to 4, characterized in that: In the demand analysis, for input nodes, the starting data source information is extracted to determine the initial structure of the data processing task; for conversion nodes, the aggregation operations required for the task are extracted according to the target conversion data type and structure.

6. The method for constructing a custom data processing and conversion process according to claim 5, characterized in that: In the demand analysis, for the output node, it includes analyzing the target data source information, determining whether the task needs to set post-execution conditions, and how to fill in the post-task information when post-conditions are required.

7. The method for constructing a custom data processing and conversion process according to any one of claims 1 to 4, characterized in that: In the node process combination, DAG is used as a processing container that combines and connects input nodes, conversion nodes, and output nodes. By splicing specific nodes in the form of a task flow graph and saving the forward and reverse relationships between vertices and edges of specific nodes in the DAG context, the process creation of the entire custom processing task is completed.

8. The method for constructing a custom data processing and conversion process according to any one of claims 1 to 4, characterized in that: During the node flow execution and analysis, after executing the data processing task flow, if an exception occurs in the process, the Yarn operation log is analyzed for reverse tracing.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for constructing a custom data processing and conversion process described in any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for constructing a custom data processing and conversion process according to any one of claims 1 to 8 is implemented.

Images