Real-time warehouse counting assembly line intelligent arrangement method, device and equipment, storage medium and program product

Abstract

The invention provides a real-time warehouse counting assembly line intelligent arrangement method, device and equipment, a storage medium and a program product, and relates to the field of warehouse counting intelligent arrangement. The arrangement method comprises the following steps: constructing an ETL platform; collecting a knowledge descriptive corpus and an assembly line arrangement translation case corpus, and combining the two corpora according to a predetermined proportion to form a training corpus; taking chatglm-6B as a pre-training model, training the pre-training model by using the training corpus, and outputting an intelligent arrangement model; an interface is integrated on an ETL platform to call an intelligent arrangement model, a user inputs natural language description and / or SQL statements and / or PromQL statements, and a system automatically generates a pipeline arrangement instruction, completes arrangement and displays the pipeline arrangement instruction on a page. According to the method, under high-quality training data, the assembly line arrangement instruction can be accurately generated according to the descriptive natural language, the SQL language or the PromQL language provided by the user, and the assembly line arrangement work is automatically completed.

Description

Technical Field

[0001] The present invention relates to the field of intelligent orchestration of data warehouses, and in particular to a method, device, equipment, storage medium and program product for intelligent orchestration of real-time data warehouse pipelines. Background Art

[0002] In the process of big data processing, in order to achieve real-time data processing, streaming computing is often adopted to quickly calculate and transform the data after it is generated and then write it to the database. At the same time, the data processing requirements in different scenarios are very different. Therefore, big data engineers are required to use ETL platforms to arrange the flow chart of data flow to meet the service object's requirements for data processing. In the prior art, the ETL platform provides componentized function operators, each of which is a simple data calculation function, and the construction of the data calculation process mainly relies on the manual drag and drop arrangement on the system interface, connecting different operators in various ways, and then outputting the interface configuration in the form of model description language to construct a directed acyclic graph. On this basis, "ETL metadata orchestration", "ETL task hierarchical organization", "combination of stream-based ETL and entity-relationship-based ETL", "distributed ETL tasks", "event-driven ETL tasks" and so on have been derived.

[0003] The existing solutions have the following problems: 1. Although the manual drag-and-drop orchestration method makes project implementation easier, it requires manpower to decompose the calculation steps and learn the platform configuration methods. In a system with a rich operator library, the learning cost is high, and the quality of the orchestrated data processing flow is also closely related to the level of the engineer.

[0004] 2. In order to meet customer needs, engineers need to configure a large number of processes. During the configuration process, there will be a lot of repetitive work, which is a heavy workload. Summary of the invention

[0005] Purpose of the invention: To propose a real-time data warehouse pipeline intelligent orchestration method, device, equipment, storage medium and program product to solve the above-mentioned problems existing in the prior art.

[0006] First, a real-time data warehouse pipeline intelligent orchestration method is proposed. The steps are as follows: Build an ETL platform; the underlying layer of the ETL platform is based on the Flink runtime architecture, which implements several available operators through Java code and Flink libraries; Collect knowledge description corpus and pipeline arrangement translation case corpus, and combine the two corpora in a predetermined ratio to form training corpus; Using chatglm-6B as a pre-trained model, training the pre-trained model using the training corpus, and outputting an intelligent arrangement model; The intelligent orchestration model is called by an integrated interface on the ETL platform. The user inputs a natural language description and / or an SQL statement and / or a PromQL statement. The system automatically generates pipeline layout instructions and completes the layout, which is displayed on the page.

[0007] In a further embodiment of the first aspect, the process of building the ETL platform specifically includes: The Flink runtime architecture is used as the underlying layer of the ETL platform, integrating input / output operators, time aggregation operators, dimension aggregation operators, function calculation operators, type conversion operators, stream merging operators, and stream splitting operators. Define instructions for each operator, which are divided into input, operation, and output; Develop the backend interface and generate executable Flink tasks through the instructions.

[0008] In a further embodiment of the first aspect, the knowledge description corpus is used to describe the functions of all available operators in the ETL platform. The pipeline arrangement translation case corpus is used to translate natural language and / or SQL statements and / or PromQL statements into pipeline arrangement instructions. The pipeline arrangement translation case corpus needs to be assembled into a standard prompt format of the language model.

[0009] In a further embodiment of the first aspect, the pipeline arrangement of the translation case corpus includes: The corpus of Chinese natural language translation into pipeline arrangement instructions is used to translate the Chinese description of the pipeline into the pipeline arrangement instructions.

[0010] The corpus of standard SQL translated into pipeline arrangement instructions is used to translate the SQL code for building the pipeline into pipeline arrangement instructions.

[0011] The corpus of PromQL translated into pipeline arrangement instructions is used to translate the PromQL code for building pipelines into pipeline arrangement instructions.

[0012] In a further embodiment of the first aspect, after combining the knowledge descriptive corpus and the pipeline arrangement translation case corpus to form a training corpus, it also includes: token encoding the training corpus; and training the pre-training model using the token-encoded training corpus.

[0013] In a further embodiment of the first aspect, the process of training the pre-trained model using the token-encoded training corpus specifically includes: using chatglm-6B as a pre-trained model and training it using the LoRA method: freezing the parameters of chatglm-6B, adding a bypass network, only training the dimensionality increase / reduction matrix of the bypass network, and adding the pre-trained model output and the bypass output results together as the layer output.

[0014] The second aspect of the present invention proposes a real-time data warehouse pipeline intelligent orchestration device, which is built on an ETL platform. The bottom layer of the ETL platform is based on the Flink runtime architecture, which implements several available operators through Java code and Flink library; in addition, the real-time data warehouse pipeline intelligent orchestration device also includes: The first execution module is used to collect the knowledge description corpus and the pipeline arrangement translation case corpus, and combine the two corpora in a predetermined ratio to form a training corpus; A second execution module is used to train the chatglm-6B model using the training corpus and output an intelligent arrangement model; The third execution module is used to integrate the interface on the ETL platform to call the intelligent orchestration model and provide a human-computer interaction interface; The user enters natural language descriptions and / or SQL statements and / or PromQL statements in the human-computer interaction interface, and the real-time data warehouse pipeline intelligent orchestration device automatically generates pipeline layout instructions and completes the layout, which is displayed on the page.

[0015] In a third aspect of the present invention, an electronic device is provided, the electronic device comprising a processor and a memory storing computer program instructions. When the processor executes the computer program instructions, the real-time data warehouse pipeline intelligent orchestration method as described in the first aspect is implemented.

[0016] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein at least one executable instruction is stored in the storage medium. When the executable instruction is executed on an electronic device, the electronic device executes the real-time data warehouse pipeline intelligent orchestration method as described in the first aspect.

[0017] According to a fifth aspect of the present invention, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the real-time data warehouse pipeline intelligent orchestration method according to the first aspect.

[0018] Compared with the traditional technology, the technical solution of the present invention has the following beneficial effects: (1) The self-developed ETL platform is used to stabilize the platform's instruction set, provide users with a visual interface, and can be adapted to other platforms in the future.

[0019] (2) The process of collecting corpus provides a basis for the stability of the model.

[0020] (3) The model training process uses appropriate parameters to ensure that the model can produce the expected output.

[0021] (4) Model usage: The platform provides users with a visual usage method to assist ETL engineers in arranging calculation pipelines, thereby improving engineers' work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a flow chart of the real-time data warehouse pipeline intelligent orchestration method in an embodiment.

[0023] Figure 2 This is the ETL platform operation architecture diagram in the embodiment.

[0024] Figure 3 It is a structural diagram of a real-time data warehouse pipeline intelligent orchestration device in an embodiment.

[0025] Figure 4 is a structural diagram of an electronic device in an embodiment. DETAILED DESCRIPTION

[0026] In the following description, a large number of specific details are provided to provide a more thorough understanding of the present invention. However, it is apparent to those skilled in the art that the present invention can be implemented without one or more of these details. In other examples, in order to avoid confusion with the present invention, some technical features known in the art are not described.

[0027] Before describing the embodiments, some definitions of terms that may appear below are first given.

[0028] ETL (Extract-Transform-Load): the process of extracting, transforming, and loading a large amount of raw data into the target storage data warehouse.

[0029] LLM (Large Language Model): is an AI model designed to understand and generate human language. They are trained on large amounts of text data and can perform a wide range of tasks, including text summarization, translation, sentiment analysis, etc. LLMs are characterized by their large size and contain billions of parameters, which help them learn complex patterns in language data. These models are usually based on deep learning architectures such as transformers, which helps them achieve impressive performance on various NLP tasks.

[0030] DSL (Domain-Specific Language): A language designed for a specific context. It is designed only for a certain applicable domain, but it is sufficient to represent problems in this domain and build corresponding solutions.

[0031] Flink: Apache Flink is a framework and distributed processing engine for stateful computations on unbounded and bounded data streams. Flink is designed to run on all common cluster environments, performing computations at memory speed and at any scale.

[0032] SQL (Standard Query Language): is a standard computer language used to access and process databases.

[0033] PromQL (Prometheus Query Language): a standard query language implemented by the Prometheus time series database middleware.

[0034] LoRA (Low-Rank Adaptation of Large Language Models): A parameter-efficient fine-tuning method that freezes the pre-trained model weights and then trains only the additional layers by adding them while freezing the original model parameters.

[0035] This embodiment discloses a real-time data warehouse pipeline intelligent arrangement method, which relies on the flink runtime library and uses LLM to assist engineers in pre-arranging the pipeline. Engineers only need to communicate with the system through natural language and commonly used DSL language, and the system can automatically generate several feasible pipeline arrangement plans for them, and engineers can make fine adjustments. The system can also check the engineer's arrangement results and point out possible errors.

[0036] The main steps of the real-time data warehouse pipeline intelligent orchestration method are: self-developed ETL platform, collection and cleaning of operators and instruction corpus, training LLM, and using LLM output system to orchestrate instructions.

[0037] 1. Self-developed ETL platform: Based on the Flink runtime, it encapsulates component operators and provides a visualization page. ETL engineers can use drag-and-drop to configure computing tasks. The specific solution is as follows: A. Encapsulated input / output operators: This package implements the file-based input and socket network port-based input / output operators provided by the Flink runtime, the JDBC, Elasticsearch, and Kafka input / output operators provided by third parties, and the self-developed HTTP protocol-based input / output operators, which are provided in the ETL operator library in the form of class objects.

[0038] B. Encapsulated calculation operators: including time aggregation operators (such as averaging time periods), dimension aggregation operators (such as averaging dimensions), function calculation operators (such as exponentiation), and type conversion operators (such as extracting field values ​​from JSON strings). Currently, there are more than 100 operators, which are provided in the form of class objects in the ETL operator library.

[0039] C. Encapsulated stream processing operators: including stream merging operators and stream splitting operators, which are functions that merge multiple independent data streams into one data stream according to certain rules, or split one data stream into multiple data streams. They are provided in the form of class objects in the ETL operator library.

[0040] D. Implementation of computing tasks: The ETL platform interface provides visualization capabilities. Each operator is abstracted into a front-end UI component. Users can build a complete pipeline by dragging and dropping operators. The page layout result of the pipeline will be converted into a string in a special format, that is, the pipeline layout instruction string, and saved to the back-end database. Each pipeline will start an ETL back-end task, which parses the pipeline layout instruction, obtains the required operators from the operator library, dynamically assembles them into executable flink tasks, and finally starts to monitor external data streams and process data.

[0041] 2. Collect and clean the operator and instruction corpus: including detailed description corpus of operators, arrangement instruction corpus provided by the ETL platform, process arrangement cases described in natural language, process arrangement cases described in SQL, and process arrangement cases described in PromQL.

[0042] 3. Fine-tune the base model based on chatglm6B.

[0043] 4. ETL platform integrated language model: convert the process arrangement instructions output by the model into process arrangement results visible on the interface. This conversion process is implemented on the front-end page, that is, parsing the pipeline arrangement instruction string and drawing the corresponding UI components on the page.

[0044] Based on the above embodiments, this embodiment discloses some operational details of a real-time data warehouse pipeline intelligent orchestration method. It should be noted that the details disclosed below do not represent a limitation of the present invention, but are only a preferred embodiment of many feasible solutions.

[0045] Step 1: Develop your own ETL platform.

[0046] (1): The bottom layer of the self-developed ETL platform is based on the Flink runtime architecture, and a large number of available operators need to be implemented through Java code and the Flink library. Operators are divided into input / output operators, time aggregation operators, dimension aggregation operators, function calculation operators, type conversion operators, stream merging operators, stream splitting operators, etc.

[0047] (2): Define instructions for each operator. Instructions are divided into three types: input, operation, and output.

[0048] (3): Develop the backend interface and generate executable Flink tasks through instructions.

[0049] (4): Develop front-end UI interface and provide human-computer interaction capabilities.

[0050] Step 2: Collect corpus.

[0051] A total of two corpora were collected, and the message description is as follows: The first type is the knowledge description corpus, which contains the functional description of all operators in the operator library of the ETL platform. The operator description is in English, which is consistent with the description in the pipeline arrangement instructions; the operator function is described in Chinese. For example: "The meaning of the AVG (dim) operator is to find the average of the dim dimension."

[0052] The second type is the case corpus for pipeline translation, which can be understood as the corpus for machine translation. The second type of corpus needs to be assembled into the standard prompt format of the language model. The format is as follows: { “context”: “Instruction:<task instruction>\nInput:<input>\nAnswer: ”, “target”: “<expected output>” } The cases are divided into three categories: The first category is the corpus of Chinese natural language translated into pipeline arrangement instructions, which translates the Chinese description of the pipeline into the pipeline arrangement instructions; for example: "Indicator cpu_usage, first average the coreid dimension, then multiply by 100" is converted into the pipeline arrangement instruction "[INPUT] [CHOOSE(cpu_usage)][AVG(coreid)] [DOT(100)][OUTPUT]".

[0053] The second category is the corpus that translates standard SQL into pipeline arrangement instructions. The SQL code for building the pipeline is translated into pipeline arrangement instructions. For example, "select avg(cpu_usage)*100 from cpu_usage group bycoreid" is converted into the pipeline arrangement instruction "[INPUT] [CHOOSE(cpu_usage)][AVG(coreid)] [DOT(100)][OUTPUT]".

[0054] The third category is the corpus translated from PromQL to pipeline layout instructions, which translates the PromQL code for building the pipeline into pipeline layout instructions. For example, "avg(cpu_usage) by (coreid) *100" is converted into the pipeline layout instruction "[INPUT] [CHOOSE(cpu_usage)][AVG(coreid)] [DOT(100)][OUTPUT]".

[0055] Step 3: Train the model.

[0056] (1): Download the chatglm-6B pre-trained model.

[0057] (2): Tokenize the training corpus, load the model on the V100 graphics card, and train using the LoRA method. That is, freeze the parameters of the pre-trained model, add a bypass network, only train the dimension increase / reduction matrix of the bypass network, and add the pre-trained model output and the bypass output results together as the layer output. Main parameters: lora_rank=4, learning_rate=1e-4, lora_dropout=0.05.

[0058] (3): Save LoRA parameters and verify the validation set.

[0059] Step 4: Integrated model.

[0060] (1): Encapsulate the large language model into a microservice and provide an interface for the front-end to call.

[0061] (2): The integrated interface on the ETL platform calls the trained model. The user enters a natural language description, SQL statement, or PromQL statement. The system automatically generates pipeline layout instructions and completes the layout, which is then displayed on the page.

[0062] In an exemplary embodiment, a real-time data warehouse pipeline intelligent orchestration device 500 is disclosed, and its structural diagram is shown in Figure 3 The device is built based on the ETL platform 501. The operation architecture of the ETL platform 501 is shown in Figure 2The bottom layer of the ETL platform 501 is based on the Flink runtime architecture, which implements several available operators through Java code and the Flink library.

[0063] The real-time data warehouse pipeline intelligent orchestration device 500 also includes a first execution module 502, a second execution module 503, and a third execution module 504. The first execution module 502 is used to collect knowledge descriptive corpus and pipeline arrangement translation case corpus, and combine the two corpora in a predetermined proportion to form a training corpus. The second execution module 503 is used to train the chatglm-6B model using the training corpus and output the intelligent orchestration model. The third execution module 504 is used to integrate the interface on the ETL platform 501 to call the intelligent orchestration model and provide a human-computer interaction interface 505. The user enters a natural language description and / or SQL statement and / or PromQL statement under the human-computer interaction interface 505, and the real-time data warehouse pipeline intelligent orchestration device 500 automatically generates a pipeline arrangement instruction and completes the arrangement, which is displayed on the page.

[0064] In an exemplary embodiment, an electronic device is provided. The electronic device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 4 As shown. The electronic device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the electronic device is used to exchange information between the processor and an external device. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a position adjustment method is implemented. The display unit of the electronic device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the electronic device casing, or an external keyboard, touchpad or mouse.

[0065] The real-time data warehouse pipeline intelligent orchestration method disclosed in the above embodiment can, in actual application, compile its operation logic into a set of executable instructions, which are written into a storage medium and run on the above electronic device.

[0066] More specific examples of computer-readable storage media mentioned in this embodiment may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), an erasable programmable read-only memory (EPROM or flash memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. Computer-readable storage media may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device.

[0067] In an exemplary embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the process of the real-time data warehouse pipeline intelligent orchestration method disclosed in the above embodiment is implemented, which is not described in detail here.

[0068] To summarize the above embodiments, the present invention combines large language model training technology with traditional ETL tools. Under high-quality training data, it can accurately generate pipeline layout instructions based on the descriptive natural language (Chinese), SQL language or PromQL language provided by the user, and automatically complete the pipeline layout work.

[0069] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. Although the present invention has been represented and described with reference to specific preferred embodiments, it shall not be interpreted as a limitation of the present invention itself. Various changes can be made to it in form and detail without departing from the spirit and scope of the present invention defined in the attached claims.

Claims

1. A real-time data warehouse pipeline intelligent orchestration method, characterized in that: The steps include: Build an ETL platform; the underlying layer of the ETL platform is based on the Flink runtime architecture, which implements several available operators through Java code and Flink libraries; Collect knowledge description corpus and pipeline arrangement translation case corpus, and combine the two corpora in a predetermined ratio to form training corpus; Using chatglm-6B as a pre-trained model, training the pre-trained model using the training corpus, and outputting an intelligent arrangement model; The intelligent orchestration model is called by an integrated interface on the ETL platform. The user inputs a natural language description and / or an SQL statement and / or a PromQL statement. The system automatically generates pipeline layout instructions and completes the layout, which is displayed on the page.

2. The real-time data warehouse pipeline intelligent orchestration method according to claim 1 is characterized in that: The construction of the ETL platform specifically includes: The Flink runtime architecture is used as the underlying layer of the ETL platform, integrating input / output operators, time aggregation operators, dimension aggregation operators, function calculation operators, type conversion operators, stream merging operators, and stream splitting operators. Define instructions for each operator, which are divided into input, operation, and output; Develop the backend interface and generate executable Flink tasks through the instructions.

3. The real-time data warehouse pipeline intelligent orchestration method according to claim 1 is characterized by: The knowledge description corpus is used to describe the functions of all available operators in the ETL platform; The pipeline arrangement translation case corpus is used to translate natural language and / or SQL statements and / or PromQL statements into pipeline arrangement instructions; The pipeline arrangement translation case corpus needs to be assembled into the standard prompt format of the language model.

4. The real-time data warehouse pipeline intelligent orchestration method according to claim 3 is characterized in that: The pipeline arrangement translation case corpus includes: The corpus of Chinese natural language translation into pipeline arrangement instructions is used to translate the Chinese description of the pipeline into the pipeline arrangement instructions; The corpus of standard SQL translated into pipeline arrangement instructions is used to translate the SQL code for building the pipeline into pipeline arrangement instructions; The corpus of PromQL translated into pipeline arrangement instructions is used to translate the PromQL code for building pipelines into pipeline arrangement instructions.

5. The real-time data warehouse pipeline intelligent orchestration method according to claim 1 is characterized in that: After combining the knowledge description corpus and the pipeline arrangement translation case corpus to form the training corpus, it also includes: The training corpus is tokenized; and the pre-training model is trained using the tokenized training corpus.

6. The real-time data warehouse pipeline intelligent orchestration method according to claim 5 is characterized in that: The training of the pre-training model using the token-encoded training corpus specifically includes: Use chatglm-6B as the pre-trained model and train it using the LoRA method: freeze the parameters of chatglm-6B, add a bypass network, only train the dimension increase / reduction matrix of the bypass network, and add the pre-trained model output and the bypass output results together as the layer output.

7. A real-time data warehouse pipeline intelligent orchestration device, which is built on an ETL platform and is characterized by: The underlying layer of the ETL platform is based on the Flink runtime architecture, which implements several available operators through Java code and Flink libraries; Also includes: The first execution module is used to collect the knowledge description corpus and the pipeline arrangement translation case corpus, and combine the two corpora in a predetermined ratio to form a training corpus; A second execution module is used to train the chatglm-6B model using the training corpus and output an intelligent arrangement model; The third execution module is used to integrate the interface on the ETL platform to call the intelligent orchestration model and provide a human-computer interaction interface; The user enters natural language descriptions and / or SQL statements and / or PromQL statements in the human-computer interaction interface, and the real-time data warehouse pipeline intelligent orchestration device automatically generates pipeline layout instructions and completes the layout, which is displayed on the page.

8. An electronic device, characterized in that: The device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the real-time data warehouse pipeline intelligent orchestration method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The storage medium stores at least one executable instruction. When the executable instruction is executed on the electronic device, the electronic device executes the real-time data warehouse pipeline intelligent orchestration method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the real-time data warehouse pipeline intelligent orchestration method described in any one of claims 1 to 6 is implemented.

Images