Generation method of Flink stream processing task, electronic equipment, storage medium and computer program product

Abstract

The invention provides a Flink stream processing task generation method, electronic equipment, a storage medium and a computer program product, and the method comprises the steps: taking task description information for target power distribution equipment as the input of a trained power distribution field large model, and obtaining a configuration file of a stream processing task outputted by the power distribution field large model; wherein the task description information indicates to process data of a plurality of measuring points on the target power distribution equipment; verifying the configuration file to determine whether the configuration file is correct or not; and if yes, generating an Flink stream processing task based on a preset operator and a preset data source indicated by the configuration file. According to the scheme, the problem that a correct Flink stream processing task cannot be quickly and accurately generated due to the fact that the writing ability of a user on the Flink stream processing task is insufficient is solved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method for generating a Flink stream processing task, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Apache Flink is an open source stream processing framework that is mainly used for stateful computing of unbounded and bounded data streams. In the field of power distribution, the operating status of power equipment needs to be monitored continuously for a long time. Therefore, various parameters in the operation of the power system can be calculated in the form of Flink stream processing tasks to detect in real time whether the power system is operating normally.

[0003] However, the writing process of Flink stream processing tasks requires certain technical skills of the user, and requires the user to have a preliminary understanding of Flink, including the basic concepts of Flink, the use of API (Application Programming Interface) and the logic of task scheduling. This is a high learning cost for non-technical personnel or beginners, and it is difficult to quickly and accurately write correct Flink stream processing tasks. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a method for generating a Flink stream processing task, an electronic device, a computer-readable storage medium, and a computer program product, so as to solve the problem that users are unable to quickly and accurately generate correct Flink stream processing tasks due to insufficient ability to write Flink stream processing tasks.

[0005] On the one hand, the present application provides a method for generating a Flink stream processing task, including: Taking the task description information for the target power distribution equipment as the input of the trained power distribution field large model, obtaining the configuration file of the stream processing task output by the power distribution field large model; wherein the task description information indicates processing the data of several measuring points on the target power distribution equipment; Verifying the configuration file to determine whether the configuration file is correct; If yes, a Flink stream processing task is generated based on the preset operator and the preset data source indicated by the configuration file.

[0006] In one embodiment, the training step of the large model in the power distribution field includes: Inputting the training corpus in the field of power distribution into the initial large model to pre-train the initial large model; Configure fine-tuning parameters for the initial large pre-trained model; The first sample data of the first sample data set is input into the pre-trained initial large model, so that the pre-trained large model is fine-tuned based on the fine-tuning parameters to obtain a large model in the power distribution field; wherein, the first sample data includes user input content and assistant reply content, the user input content is task description information in the power distribution field, and the assistant reply content is configuration information of the flow processing task in the power distribution field.

[0007] In one embodiment, the method further comprises: The second sample data of the second sample data set is input into the power distribution field large model, so that the power distribution field large model continues to be fine-tuned based on the fine-tuning parameters to obtain an updated power distribution field large model; wherein, the second sample data includes system instruction content, user input content and assistant reply content, and the system instruction content is the context information of the user input content.

[0008] In one embodiment, before inputting the second sample data of the second sample data set into the power distribution domain macro model, the method further includes: Inputting the test sample data of the test sample data set into the power distribution domain big model, and obtaining the output result generated by the power distribution domain big model based on the user input content in the test sample data; Determine whether the output result is consistent with the assistant reply content in the test sample data; Obtaining the accuracy of the large model in the power distribution field according to the judgment results corresponding to the multiple test sample data in the test sample data set; Determine whether the accuracy reaches the accuracy threshold; if not, execute the step of inputting the second sample data of the second sample data set into the power distribution field large model.

[0009] In one embodiment, the power distribution field training corpus includes multiple types of sub-training corpora, and the multiple types of sub-training corpora are pre-configured in input order; The step of inputting the power distribution field training corpus into the initial large model to pre-train the initial large model includes: Input each type of sub-training corpus into the initial large model in sequence according to the input order to implement the pre-training process of the initial large model; wherein each sub-training corpus includes prompt information and completion information.

[0010] In one embodiment, the multiple types of sub-training corpora include equipment measurement point information, equipment historical operation data, distribution room attribute information, distribution business term information, operator information, and comprehensive scenario information; wherein the comprehensive scenario information is integrated information of the equipment measurement point information, the equipment historical operation data, the distribution room attribute information, the distribution business term information, and the operator information; The step of inputting each type of sub-training corpus into the initial large model in sequence according to the input order to implement the pre-training process of the initial large model includes: Inputting the equipment measurement point information into the initial large model; Inputting the equipment historical operation data into the initial large model; Inputting the power distribution room attribute information into the initial large model; Inputting the power distribution business terminology information into the initial large model; Inputting the operator information into the initial large model; The comprehensive scene information is input into the initial large model.

[0011] In one embodiment, the method further comprises: In response to the operator configuration instruction, the operator in the operator configuration instruction is loaded.

[0012] In another aspect, the present application provides an electronic device, the electronic device comprising: processor; a memory for storing processor-executable instructions; The processor is configured to execute the above-mentioned Flink stream processing task generation method.

[0013] Furthermore, the present application provides a computer-readable storage medium, which stores a computer program, and the computer program can be executed by a processor to complete the above-mentioned Flink stream processing task generation method.

[0014] In addition, the present application provides a computer program product, including a computer program / instruction, which implements the above-mentioned Flink stream processing task generation method when executed by a processor.

[0015] This application solution can use the trained large model of the power distribution field to perform intelligent analysis on the task description information, thereby outputting a configuration file, and after the configuration file is verified, the Flink stream processing task can be generated through the configuration file. This process can be executed automatically, and only the task description information in the form of natural language is required, which gets rid of explicit commands, so that users do not need to write programs or enter display commands, and can build stream processing tasks through natural language, reducing the user's learning cost and avoiding the problem that the user cannot quickly and accurately generate the correct Flink stream processing task due to insufficient writing ability of Flink stream processing tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solution of the embodiments of the present application, the drawings required for use in the embodiments of the present application are briefly introduced below.

[0017] Figure 1 A schematic diagram of an application scenario of a method for generating a Flink stream processing task provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application; Figure 3 A flowchart of a method for generating a Flink stream processing task provided in an embodiment of the present application; Figure 4 A flowchart of a method for training a large model in the power distribution field provided in one embodiment of the present application; Figure 5 A schematic flow chart of a method for fine-tuning a large model in the power distribution field provided in one embodiment of the present application; Figure 6 A flowchart of a method for pre-training a large model in the power distribution field provided in one embodiment of the present application; Figure 7 A block diagram of a device for generating a Flink stream processing task according to an embodiment of the present application. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0019] Similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0020] Figure 1 A schematic diagram of an application scenario of the method for generating a Flink stream processing task provided in an embodiment of the present application. Figure 1 As shown, the application scenario includes a client 20 and a server 30; the client 20 can be a user terminal such as a host, a mobile phone, a tablet computer, etc., which is used to send task description information to the server 30; the server 30 can be a server, a server cluster or a cloud computing center, which can use the task description information to generate corresponding Flink stream processing tasks.

[0021] like Figure 2 As shown, this embodiment provides an electronic device 1, including: at least one processor 11 and a memory 12, Figure 2A processor 11 is taken as an example. The processor 11 and the memory 12 are connected via a bus 10. The memory 12 stores instructions that can be executed by the processor 11. The instructions are executed by the processor 11 so that the electronic device 1 can execute all or part of the process of the method in the following embodiment. In one embodiment, the electronic device 1 can be the above-mentioned server 30, which is used to execute the method for generating Flink stream processing tasks.

[0022] The memory 12 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable red-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0023] The present application also provides a computer-readable storage medium, which stores a computer program. The computer program can be executed by the processor 11 to complete the method for generating a Flink stream processing task provided in the present application.

[0024] The present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the method for generating a Flink stream processing task provided in the present application.

[0025] See also Figure 3 , which is a flow chart of a method for generating a Flink stream processing task provided in an embodiment of the present application, such as Figure 3 As shown, the method may include the following steps 310 to 330.

[0026] Step 310: Use the task description information for the target power distribution equipment as the input of the power distribution field big model to obtain the configuration file of the stream processing task output by the power distribution field big model; wherein the task description information indicates processing the data of several measuring points on the target power distribution equipment.

[0027] Here, the target power distribution equipment is the power distribution equipment for which the Flink stream processing task needs to be generated. The power distribution equipment can be a combination of one or more of the high-voltage distribution cabinet, generator, transformer, power line, circuit breaker, low-voltage switch cabinet, distribution board, switch box, control box, etc. in the power distribution system.

[0028] The task description information is information describing the task of processing the data of several measuring points on the target power distribution equipment to obtain the corresponding results. The task description information is input by the user in the form of natural language.

[0029] Exemplarily, the task description information is "calculate the average of the data of measuring point 23 on the transformer over the past week"; or, the task description information is "determine whether the high-voltage distribution cabinet is operating normally based on the data of measuring point 45 on the high-voltage distribution cabinet."

[0030] When a Flink stream processing task needs to be generated for a power distribution system, the user can enter task description information in natural language through the user interface and submit it to the server that executes the Flink stream processing task generation method.

[0031] After the server obtains the task description information, it can input the task description information into the trained power distribution domain model, process the task description information through the power distribution domain model, and generate a configuration file for the stream processing task. Here, the configuration file includes operators, operator arrangement order, and parameter settings for processing tasks described in the task description information (such as temperature alarms, load anomalies, etc.). The format of the configuration file can be predefined. For example, the configuration file can be a JSON (JavaScript Object Notation) file.

[0032] For example, the content of the configuration file generated for the task description information "Task flow for determining whether the temperature of power distribution No. 3 is abnormal" is as follows: { "task_id": "room-3-temperature-alert", "description": "Temperature alert for room 3", "operators": [ { "operator": "aggregate", "aggregate_function": "avg", "window_size": "5m" }, { "operator": "alert", "alert_conditions": { "temperature": ">75", "alert_type": "warning" } } ] } From the content of this configuration file, we can see that this task involves two operators, namely aggregation and alert. The aggregation operator is used to calculate the average of temperature data in a 5-minute time window; the alert operator is used to output an alarm message when the average in any time window is greater than 75.

[0033] Step 320: Verify the configuration file to determine whether the configuration file is correct.

[0034] After obtaining the configuration file, the configuration file can be parsed to verify the correctness and completeness of its syntax and semantics. Here, the verification method can be a rule engine or regular expression, etc. Through verification, it can be ensured that the definition of the operator in the configuration file and its connection relationship meet the expected standards, and the consistency of the data format is maintained, effectively identifying potential errors and inconsistencies.

[0035] If the configuration file is incorrect, the configuration file can be ignored and a prompt message indicating that the configuration file is incorrect is generated. If the configuration file is correct, step 330 can be continued.

[0036] Step 330: If yes, generate a Flink stream processing task based on the preset operators and preset data sources indicated by the configuration file.

[0037] If the configuration file is correct, based on the operator ID, operator name and other information indicated in the configuration file, you can select the corresponding operator from the operators configured on the Flink component management platform, and arrange the selected operators according to the operator arrangement order indicated in the configuration file. Use the data source in the configuration file as the input of the operator to generate a Flink stream processing task.

[0038] Through this measure, the trained large model of the power distribution field can be used to perform intelligent analysis on the task description information, thereby outputting the configuration file, and after the configuration file is verified, the Flink stream processing task can be generated through the configuration file. This process can be executed automatically, and only the task description information in the form of natural language is required, getting rid of explicit commands, so that users do not need to write programs or enter display commands, and can build stream processing tasks through natural language, reducing the user's learning cost and avoiding the problem that users cannot quickly and accurately generate correct Flink stream processing tasks due to insufficient writing ability of Flink stream processing tasks.

[0039] In one embodiment, see Figure 4 , is a flow chart of a training method for a large model in the power distribution field provided in an embodiment of the present application, such as Figure 4 As shown, the method may include the following steps 410 to 430.

[0040] Step 410: Input the power distribution field training corpus into the initial large model to pre-train the initial large model.

[0041] Among them, the training corpus in the power distribution field is the corpus data related to power distribution equipment, operators, and power distribution business in the power distribution field.

[0042] Collect corpus data in the field of power distribution, and perform data cleaning and data standardization.

[0043] Here, data cleaning can include noise removal, missing value processing, and outlier detection.

[0044] To remove the noise, Levenshtein distance is used to remove duplicate records. To deal with data noise, a simple linear model can be defined and residual analysis can be used to identify excessive noise values.

[0045] Missing value processing: For missing values, use reasonable interpolation or filling methods, such as using the mean or median to fill, or determine the missing value based on the context.

[0046] Outlier detection: Identifying and handling outliers in data, such as using statistical methods (such as the standard deviation method) or machine learning algorithms (such as isolation forests) for detection.

[0047] For numerical data, data standardization can be achieved through $z$-score standardization, mapping the data to a standard normal distribution. For example, the standardization process is represented by the following formula (1): (1) in, Represents the numerical value of any feature; is the mean value of the feature; is the standard deviation of the feature; The result of normalization for this feature.

[0048] For categorical data, you can convert the categorical data into a numerical format to achieve data standardization. Common methods include One-Hot Encoding and Label Encoding.

[0049] After data cleaning and data standardization, the distribution field training data can be obtained, and the distribution field training data can be input into the initial large model to pre-train it. Here, the initial large model can be GPT-3.5-Turbo, GPT-3.5 or other large models. Preferably, the initial large model is GPT-3.5-Turbo.

[0050] Step 420: Configure fine-tuning parameters for the pre-trained initial large model.

[0051] After pre-training the initial large model, the initial large model can learn the basic knowledge of the power distribution field. In this case, the initial large model can be fine-tuned to train a large model in the power distribution field.

[0052] Before fine-tuning begins, you can configure the fine-tuning parameters, which may include batch size, initial learning rate and learning rate decay coefficient, loss function, etc.

[0053] The initial learning rate and the learning rate decay coefficient can be used to calculate the learning rate of each round. For example, the calculation method can be expressed by the following formula (2): (2) Among them, LR represents the learning rate; ILR represents the initial learning rate; e is a natural constant; t is the round of training; is the learning rate decay coefficient.

[0054] The learning rate is set relatively high in the initial stage to ensure that the large model can quickly adjust to the new task generation requirements. Towards the end of training, the learning rate is gradually reduced to ensure that the fine-tuned large model can stably generate accurate tasks.

[0055] Step 430: Input the first sample data of the first sample data set into the pre-trained initial large model, so that the pre-trained large model is fine-tuned on the basis of fine-tuning parameters to obtain a large model in the power distribution field; wherein, the first sample data includes user input content and assistant reply content, the user input content is the task description information in the power distribution field, and the assistant reply content is the configuration information of the flow processing task in the power distribution field.

[0056] A plurality of first sample data are input into the initial large model with a batch size in the fine-tuning parameter.

[0057] The user input content in the first sample data is used to instruct the initial large model to predict the configuration information. The difference between the predicted configuration information and the assistant's reply content in the first sample data can be evaluated by the loss function in the fine-tuning parameter, and the initial large model can be adjusted based on the difference. The adjustment process uses the learning rate of the current round in the fine-tuning parameter.

[0058] After multiple rounds of fine-tuning, a large model for the power distribution domain can be obtained. The large model for the power distribution domain can generate configuration files for Flink stream processing tasks based on the task description information in the power distribution domain.

[0059] The following is an example of the first sample data: {"messages": [{"role": "user", "content": "Calculate the real-time data of measurement point 401"},{"role": "assistant", "content": "{ "operators": { "name": "Real-time data" }, { "name": "Real-time data calculation" }, { "name": "Real-time output" } } "}]} Among them, user refers to user input, content is content, and the user input content is "calculate the real-time data of measuring point 401"; assistant refers to the assistant reply, content is content, and the assistant reply content is three operators named "real-time data", "real-time data calculation" and "real-time output" and the sorting method of the three operators.

[0060] Through the above measures, the pre-trained initial large model can be fine-tuned using the first sample data set to obtain a large model in the power distribution field.

[0061] In one embodiment, after the large model of the power distribution field is fine-tuned using the first sample data set, it can be fine-tuned to obtain a large model of the power distribution field with better performance.

[0062] The second sample data of the second sample data set is input into the large model of the power distribution field, so that the large model of the power distribution field continues to be fine-tuned on the basis of the fine-tuning parameters to obtain an updated large model of the power distribution field.

[0063] The second sample data includes system instruction content, user input content, and assistant reply content. The system instruction content is the context information of the user input content. In this case, the system instruction content is used to define the background, rules, or specific task requirements of the dialogue between the user input content and the assistant reply content, which can help the large model in the power distribution field understand the background and expected behavior of the dialogue.

[0064] The following is an example of the second sample data: {"messages": [{"role": "system", "content": "Convert natural language to a JSON operator set, which must include input and output. Operator set: [{ "name" : "Real-time data", "id" : 1, "type" : input },{ "name" : "Real-time data calculation", "id" : 2, "type" : calculation }{ "name" : "Real-time output", "id" : 1, "type" : output }. Measurement point set: [{ "name" : "Transformer measurement point 401", "base" : "Province B / City H / Company J / District A / Equipment 3", } ]. "}, {"role": "user", "content": "Calculate the real-time data of measuring point 401"}, {"role":"assistant", "content": "{ "operators": { "name": "Real-time data" }, { "name": "Real-time data calculation" }, { "name": "Real-time output" } } "}]} Among them, user refers to user input, content is content, and the user input content is "calculate the real-time data of measuring point 401"; assistant refers to the assistant reply, content is content, and the assistant reply content is three operators named "real-time data", "real-time data calculation" and "real-time output" and the sorting method of the three operators.

[0065] System refers to system instructions, and content refers to content. System instruction content includes operator set and measurement point set. The operator set includes the operator identifier, operator name, and operator type of the three operators. The measurement point set records the measurement point information corresponding to the user input content.

[0066] In addition, in order to optimize the fine-tuning results of the model, for tasks that are not detailed enough in the first sample data set, the data set can be re-labeled to obtain a second sample data set. In this case, the tasks in the second sample data set are more detailed, which can optimize the ability of the large model in the power distribution field to understand specific tasks after fine-tuning.

[0067] For complex business scenarios, we can expand the diversity of task description information in the second sample data (using different task description information to describe the same task) and select a variety of operator combinations to enhance the generalization capability of large models in the power distribution field in different business scenarios.

[0068] Through the above measures, the large model of the power distribution field can be updated using the expanded second sample data set, thereby obtaining a large model of the power distribution field with better performance.

[0069] In one embodiment, before fine-tuning the large model in the power distribution field using the second sample data set, it can be determined whether the large model in the power distribution field needs to be further fine-tuned. Figure 5 , is a flow chart of a fine-tuning method for a large model in the power distribution field provided in an embodiment of the present application, such as Figure 5 As shown, the fine-tuning process may be performed through the following steps 510 to 540.

[0070] Step 510: Input the test sample data of the test sample data set into the power distribution domain big model, and obtain the output result generated by the power distribution domain big model based on the user input content in the test sample data.

[0071] Among them, the test sample data includes user input content and assistant reply content. The user input content is the task description information in the power distribution field, and the assistant reply content is the configuration information of the flow processing task in the power distribution field.

[0072] The test sample data set may select sample data different from the first sample data set as the test sample data.

[0073] The test sample data is input into the power distribution domain big model, and the user input content in the test sample data can be processed by the power distribution domain big model to obtain the output result.

[0074] Step 520: Determine whether the output result is consistent with the assistant's reply content in the test sample data.

[0075] If the large model in the power distribution field can correctly process user input, the output results should be consistent with the assistant's replies in the test sample data.

[0076] Compare the output results corresponding to each test sample data with the assistant's reply content to determine whether the two are consistent.

[0077] Step 530: Obtain the accuracy of the large model in the power distribution field according to the judgment results corresponding to the multiple test sample data in the test sample data set.

[0078] By counting the number of consistent judgment results corresponding to multiple test sample data and dividing this number by the total number of judgment results, the accuracy of the large model in the power distribution field can be obtained.

[0079] Step 540: Determine whether the accuracy rate reaches the accuracy rate threshold, and if not, execute the step of inputting the second sample data of the second sample data set into the power distribution field large model.

[0080] The accuracy threshold can be configured as needed. For example, the accuracy threshold can be any value such as 95%, 96%, or 97%.

[0081] After the accuracy rate is obtained, it can be determined whether the accuracy rate reaches an accuracy rate threshold.

[0082] On the one hand, if yes, it means that the large model in the power distribution field has reached the application standard. In this case, the large model in the power distribution field can be directly used for the subsequent configuration file generation task. On the other hand, if no, it means that the large model in the power distribution field has not yet reached the application standard. In this case, the large model in the power distribution field can be further fine-tuned through the second sample data set to update the large model in the power distribution field.

[0083] Through the above measures, the large model in the power distribution field can be fine-tuned when it does not meet the application standards. If it meets the application standards, there is no need to continue fine-tuning, thus saving the computing resources occupied by fine-tuning.

[0084] In one embodiment, the power distribution field training corpus includes multiple types of sub-training corpora, and the multiple types of sub-training corpora are pre-configured in input order.

[0085] When the initial large model is pre-trained through the training corpus in the power distribution field, each type of sub-training data can be input into the initial large model in sequence according to the input order to implement the pre-training process of the initial large model. Among them, each sub-training data includes prompt information (Prompt) and completion information (Completion). Each sub-training data instructs the initial large model to output completion information according to the prompt information.

[0086] By configuring the input order for different types of sub-training corpora and inputting different types of sub-training data in sequence, the initial large model can gradually learn different types of power distribution domain knowledge and achieve better learning results.

[0087] In one embodiment, the various types of sub-training data include equipment measurement point information, equipment historical operation data, distribution room attribute information, distribution business terminology information, operator information, and comprehensive scenario information.

[0088] See also Figure 6 , is a flow chart of a method for pre-training a large model in the power distribution field provided in an embodiment of the present application, such as Figure 6 As shown, when each type of sub-training corpus is input into the initial large model in sequence according to the input order, the following pre-training process of step 610 to step 660 can be executed.

[0089] Step 610: Input the equipment measurement point information into the initial large model.

[0090] The equipment measurement point information is information related to the measurement point on the specific power distribution equipment. Exemplarily, the equipment measurement point information may include the measurement point identification, measurement point type (telemetry / telesignaling), measurement point name, measurement point abbreviation, measurement point code, equipment identification (the identification of the equipment where the measurement point is located), distribution room identification (the identification of the distribution room where the equipment is located), measurement unit, coefficient, upper limit, lower limit, etc.

[0091] For example, a piece of equipment measurement point information as a sub-training corpus can be expressed as: { "prompt": "Point ID: 001, Point type: Telemetry, Point name: Voltage sensor, Point abbreviation: Voltage, Point code: 12345, Distribution room ID: 001A, Equipment ID: 56789, Unit: V, Coefficient: 1.0, Upper limit: 500, Lower limit: 0", "completion": "This voltage sensor is used to monitor the voltage of the power system. The operating range is between 0V and 500V, with a coefficient of 1.0. It is suitable for distribution room 001A." } During the pre-training process, the device measurement point information may be first input into the initial large model so that the initial large model can learn the relevant knowledge of the device measurement points.

[0092] Step 620: Input the historical operation data of the equipment into the initial large model.

[0093] The historical operation data of the equipment may include historical data and historical alarm information of the measuring point. For example, the historical operation data of the equipment may include the measuring point identification, historical average value, historical alarm type, historical alarm level, etc.

[0094] For example, several pieces of historical operation data of devices are given below as sub-training corpus: { "prompt": "Measurement point id": "PT1001", "Historical Mean": 23.5, "Historical alarm type": "Current is too high", "Historical Alarm Level": "Severe" "completion": "The historical average value of the measurement point PT1001 is 23.5, the historical alarm type is overcurrent, and the historical alarm level is severe." } { "prompt":"Measurement point id": "PT1002", "Historical Mean": 18.0, "Historical alarm type": "Voltage is too low", "Historical Alarm Level": "Medium" "completion":"The historical average value of the measuring point PT1002 is 18.0, the historical alarm type is low voltage, and the historical alarm level is medium." } { "prompt":"Measurement point id": "PT1003", "Historical Mean": 25.0, "Historical alarm type": "Current is too high", "Historical Alarm Level": "Minor" "completion":"The historical average value of the measuring point PT1003 is 25.0, the historical alarm type is overcurrent, and the historical alarm level is minor." } After the initial large model is pre-trained with the equipment measurement point information, the equipment historical operation data can be input into the initial large model so that the initial large model can learn the relevant knowledge of the measurement point alarm under the historical average.

[0095] Step 630: Input the property information of the power distribution room into the initial large model.

[0096] The power distribution room attribute information is the relevant information of the location of the power distribution equipment. Exemplarily, the power distribution room attribute information may include the name of the power distribution room, project representation, status, load level, power supply capacity, trombone, etc.

[0097] For example, several pieces of attribute information of a power distribution room are given below as sub-training corpus: { "prompt": "{Distribution room name: Distribution room A, Project id: Project 001, Status: Normal, Load level: High load, Power supply capacity: 5000kVA, Factory number: Factory number 001}", "completion": "The distribution room is named 'Distribution Room A' and belongs to the project 'Project 001'. The current status is 'Normal' and the load level is 'High Load'. Its power supply capacity is 5000kVA and it belongs to 'Factory No. 001'. The distribution room is responsible for the power supply and load monitoring of the corresponding area to ensure the stable operation of the system." } { "prompt": "{Distribution room name: Distribution room B, Project id: Project 002, Status: Fault, Load level: Medium load, Power supply capacity: 3000kVA, Factory number: Factory number 002}", "completion": "The distribution room is named 'Distribution Room B' and belongs to the project 'Project 002'. The current status is 'Fault' and the load level is 'Medium Load'. Its power supply capacity is 3000kVA and it belongs to 'Factory No. 002'. The distribution room currently has a fault and needs to be checked and repaired immediately to restore normal power supply." } { "prompt": "{Distribution room name: Distribution room C, Project id: Project 003, Status: Normal, Load level: Low load, Power supply capacity: 1000kVA, Factory number: Factory number 003}", "completion": "The distribution room is named 'Distribution Room C' and belongs to project 'Project 003'. The current status is 'Normal' and the load level is 'Low Load'. Its power supply capacity is 1000kVA and it belongs to 'Factory No. 003'. The load of this distribution room is low and it is suitable for low-load operation and monitoring." } After the initial large model is pre-trained by inputting the historical operation data of the equipment, the attribute information of the distribution room can be input into the initial large model so that the initial large model can learn the relevant knowledge of the location where the distribution equipment is located.

[0098] Step 640: Input power distribution business terminology information into the initial large model.

[0099] The power distribution business terminology information is information that explains professional terms in the power distribution business. The power distribution business terminology information can be obtained from the operation manual.

[0100] For example, several pieces of power distribution business terminology information are given below as sub-training corpus: { "prompt": "Three-phase current alarm", "completion": "The current flows through the three phases (phase A, phase B and phase C) and passes through the load. The three-phase current alarm detects whether the current size and balance exceed the set limit." } { "prompt": "Measurement point ID", "completion": "A measurement point ID is a number used to uniquely identify a measurement point, usually related to the physical location or function of the device." } After pre-training the initial large model by inputting the distribution room attribute information, the distribution business terminology information can be input into the initial large model, so that the initial large model can learn the specific connotations of professional names in the distribution business, thereby more fully understanding the corpus information in the distribution field.

[0101] Step 650: Input the operator information into the initial large model.

[0102] The operator information may include operator functions, input and output requirements, configuration parameters, etc.

[0103] For example, an operator information as a sub-training corpus can be expressed as: { "prompt": "Function: Collect current measurement point value Parameters: Measurement point ID, Name: current_harvesting, Type: source operator", "completion": "Flink operator, used to collect current measurement points, belongs to the source operator" } After pre-training the initial large model by inputting the power distribution business terminology information, the operator information can be input into the initial large model so that the initial large model can analyze the operator properties and learn the usage scenarios of different operators, thereby selecting the appropriate operator combination for the business when generating the configuration file later.

[0104] Step 660: Input the comprehensive scene information into the initial large model.

[0105] Among them, the comprehensive scenario information is the integration of equipment measurement point information, equipment historical operation data, distribution room attribute information, distribution business terminology information and operator information.

[0106] For example, a piece of comprehensive scene information as a sub-training corpus can be expressed as: { "prompt": "Please understand the following transformer measurement point information, obtain measurement point attributes, historical data range, possible anomalies and related operators: {Transformer measurement point 401, the distribution room is B Province / H City / J Company / A District / Equipment 3, the associated operator is current measurement point A1, historical data 100~300, possible anomalies are data interruption, high load rate, and the operators related to it are measurement point real-time data Source operator, transformer imbalance calculation, transformer load calculation, data interruption alarm calculation, alarm output operator, and real-time data operator.}", "completion": { "Measurement point number": "401", "Location: Distribution Room": "Province B / City H / Company J / Area A / Equipment 3", "Associated operator": "Current measurement point A1", "Historical data range": "100~300", "Possible exception": [ "Data interruption", "High load factor" ], "Related Operators": [ "Measurement point real-time data Source operator", "Transformer Unbalance Calculation", "Transformer Load Calculation", "Data interruption alarm calculation", "Alarm output operator", "Real-time Data Operator" ] } } During the pre-training process, after pre-training the initial large model using the aforementioned sub-training corpora, the comprehensive scenario information is input into the initial large model, so that the initial large model can learn the semantic logic after integrating various types of knowledge in the power distribution field.

[0107] Through the above measures, several types of sub-training corpora are input into the initial large model training in sequence, so that the initial large model can complete the learning process of distribution field knowledge in stages and obtain better pre-training effect.

[0108] In one embodiment, before generating a Flink stream processing task using the operator indicated by the configuration file, multiple operators may be preconfigured on a management platform of the Flink component.

[0109] Users can integrate operators on the management platform by issuing operator configuration instructions to the management platform.

[0110] Among them, the operator configuration instruction can be defined by the following fields of the operator: input data type, output data type, state variable, calculation range, configuration parameters, calculation logic and output process.

[0111] Operator types are mainly divided into three types: data source operators (source operators), output operators (sink operators) and calculation operators.

[0112] The main function of the data source operator is to obtain data from external data sources (such as databases, message queues, file systems, external APIs, etc.), convert the data into a data format suitable for the stream processing framework (such as POJO, Tuple, Row, etc.), and then pass it to the downstream operator.

[0113] The main function of the output operator is to output the results of stream processing to an external storage location or system (such as a database, message queue, file system, communication interface, etc.). The output operator can write data to the target system on demand, either in batches (for example, once every 100 data items) or in real time.

[0114] The main function of a computing operator is to generate a new data stream after some transformation operations on the input data. A computing operator can usually perform the following operations: First, the conversion operation: Map: Convert each piece of input data into a new form or structure.

[0115] FlatMap: Similar to mapping, but one input data can generate multiple output data.

[0116] Filter: Filter input data based on conditions and output data that meets the conditions.

[0117] Aggregate: Perform aggregation operations such as summarizing, counting, and so on on input data.

[0118] Union: Merge multiple data streams into one data stream.

[0119] Second, window operation: Window: Divide the input stream into time or count windows and perform operations within the windows. For example, sliding windows, tumbling windows, session windows, etc.

[0120] Third, state operation: Stateful: The computing operator can maintain state and is used for calculations across multiple events or time periods, such as cumulative sum, average, maximum value, etc.

[0121] According to the application requirements of power distribution scenarios, the operators that can be configured on the management platform include: Data source operators: data source operators for real-time data of measuring points, data source operators for historical data of measuring points, data source operators for log data, etc.

[0122] Output operators: alarm output operator, real-time data operator, historical data operator, log recording operator, etc.

[0123] Calculation operators: real-time data calculation operator, telemetry alarm calculation operator, telesignal alarm calculation operator, electricity calculation operator, demand calculation operator, virtual measuring point calculation operator, capacitor cabinet unbalance calculation operator, transformer unbalance calculation operator, transformer load calculation operator, data interruption alarm calculation operator, temperature alarm calculation operator, humidity alarm calculation operator, log analysis calculation operator, etc.

[0124] In response to the operator configuration instruction, the operator in the operator configuration instruction is loaded. In this case, the loaded operator can be selected based on the configuration file to generate a Flink stream processing task.

[0125] In addition, this solution can provide a version control mechanism for Flink stream processing tasks, allowing users to track the historical change records of tasks and monitor the task execution status, so that users can view and manage the created Flink stream processing tasks. Flink stream processing tasks can be quickly rolled back and version restored as needed to ensure that problems can be quickly dealt with when they occur during task execution.

[0126] It can manage Flink clusters, track the usage of CPU (Central Processing Unit), memory, and I / O (Input / Output) in real time, and provide optimization suggestions based on performance indicators to improve the overall performance and resource utilization of the Flink cluster.

[0127] Figure 7 is a block diagram of a device for generating a Flink stream processing task according to an embodiment of the present invention. Figure 7 As shown, the device may include: An input module 710 is used to use the task description information for the target power distribution equipment as the input of the trained power distribution field large model to obtain a configuration file of the stream processing task output by the power distribution field large model; wherein the task description information indicates processing the data of several measuring points on the target power distribution equipment; A verification module 720 is used to verify the configuration file to determine whether the configuration file is correct; The generation module 730 is used to generate a Flink stream processing task based on the preset operator and the preset data source indicated by the configuration file.

[0128] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned Flink stream processing task generation method, which will not be repeated here.

[0129] In several embodiments provided in the present application, the disclosed devices and methods may also be implemented in other ways. The device embodiments described above are merely schematic, for example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram may represent a module, a program segment or a part of a code, and a module, a program segment or a part of a code contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they may sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0130] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0131] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

Claims

1. A method for generating a Flink stream processing task, characterized in that: include: Taking the task description information for the target power distribution equipment as the input of the trained power distribution field large model, obtaining the configuration file of the stream processing task output by the power distribution field large model; wherein the task description information indicates processing the data of several measuring points on the target power distribution equipment; Verifying the configuration file to determine whether the configuration file is correct; If yes, a Flink stream processing task is generated based on the preset operator and the preset data source indicated by the configuration file.

2. The method according to claim 1, characterized in that The training steps of the large model in the power distribution field include: Inputting the training corpus in the field of power distribution into the initial large model to pre-train the initial large model; Configure fine-tuning parameters for the initial large pre-trained model; The first sample data of the first sample data set is input into the pre-trained initial large model, so that the pre-trained large model is fine-tuned based on the fine-tuning parameters to obtain a large model in the power distribution field; wherein, the first sample data includes user input content and assistant reply content, the user input content is task description information in the power distribution field, and the assistant reply content is configuration information of the flow processing task in the power distribution field.

3. The method according to claim 2, characterized in that The method further comprises: The second sample data of the second sample data set is input into the power distribution field large model, so that the power distribution field large model continues to be fine-tuned based on the fine-tuning parameters to obtain an updated power distribution field large model; wherein, the second sample data includes system instruction content, user input content and assistant reply content, and the system instruction content is the context information of the user input content.

4. The method according to claim 3, characterized in that Before inputting the second sample data of the second sample data set into the power distribution domain macro model, the method further includes: Inputting the test sample data of the test sample data set into the power distribution domain big model, and obtaining the output result generated by the power distribution domain big model based on the user input content in the test sample data; Determine whether the output result is consistent with the assistant reply content in the test sample data; Obtaining the accuracy of the large model in the power distribution field according to the judgment results corresponding to the multiple test sample data in the test sample data set; Determine whether the accuracy reaches the accuracy threshold; if not, execute the step of inputting the second sample data of the second sample data set into the power distribution field large model.

5. The method according to claim 2, characterized in that: The power distribution field training corpus includes multiple types of sub-training corpora, and the multiple types of sub-training corpora are pre-configured in input order; The step of inputting the power distribution field training corpus into the initial large model to pre-train the initial large model includes: Input each type of sub-training corpus into the initial large model in sequence according to the input order to implement the pre-training process of the initial large model; wherein each sub-training corpus includes prompt information and completion information.

6. The method according to claim 5, characterized in that The multiple types of sub-training corpora include equipment measurement point information, equipment historical operation data, distribution room attribute information, distribution business term information, operator information, and comprehensive scenario information; wherein the comprehensive scenario information is the integrated information of the equipment measurement point information, the equipment historical operation data, the distribution room attribute information, the distribution business term information, and the operator information; The step of inputting each type of sub-training corpus into the initial large model in sequence according to the input order to implement the pre-training process of the initial large model includes: Inputting the equipment measurement point information into the initial large model; Inputting the equipment historical operation data into the initial large model; Inputting the power distribution room attribute information into the initial large model; Inputting the power distribution business terminology information into the initial large model; Inputting the operator information into the initial large model; The comprehensive scene information is input into the initial large model.

7. The method according to claim 1, characterized in that The method further comprises: In response to the operator configuration instruction, the operator in the operator configuration instruction is loaded.

8. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing processor-executable instructions; The processor is configured to execute the method for generating a Flink stream processing task according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program can be executed by a processor to complete the method for generating a Flink stream processing task according to any one of claims 1 to 7.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the method for generating a Flink stream processing task according to any one of claims 1 to 7 is implemented.

Images