Data processing method and device, equipment and storage medium

Abstract

The invention discloses a data processing method and device, equipment and a storage medium, and relates to the technical field of data processing. The method comprises the steps of determining an original calculation task according to business requirements of a power plant; generating a processing logic diagram of the original calculation task based on the operation on the execution sequence of each sub-calculation task of the original calculation task; obtaining target monitoring stream data of the power plant, and extracting various data from the target monitoring stream data according to the data type required by the original calculation task and the extraction strategy of the various data; and configuring the extracted data to each sub-calculation task of the original calculation task according to the processing logic diagram to obtain a target calculation task. According to the technical scheme, the risk of single-point failure is avoided, the data processing speed is increased, and delay is reduced.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of data processing technology, and in particular, to a data processing method, apparatus, device and storage medium. Background Art

[0002] With the rapid development of information technology, the amount of data in large industrial facilities such as power plants has increased dramatically. These data include sensor monitoring data, real-time load data of the power grid, equipment status information, etc., covering all aspects from power generation to transmission. Faced with such a huge amount of data, how to scientifically and effectively process these data to ensure the safe and efficient operation of power plants has become a key issue that the power industry needs to solve urgently.

[0003] At present, traditional data processing methods usually adopt a centralized processing method, that is, all data is aggregated to a central server for unified processing. However, this method not only has the risk of single point failure, but also often has problems of slow processing speed and high latency when processing massive data due to the limitations of computing power and storage capacity.

[0004] Therefore, it is urgent to propose a new method to solve the above problems. Summary of the invention

[0005] The present invention provides a data processing method, device, equipment and storage medium, which not only avoid the risk of single point failure, but also improve the data processing speed and reduce delay.

[0006] In a first aspect, an embodiment of the present invention provides a data processing method, which is applied to any server in a server cluster, and the method includes:

[0007] Determine the original computing tasks according to the business needs of the power plant;

[0008] Generate a processing logic diagram of the original computing task based on the operation of the execution order of each sub-computing task of the original computing task;

[0009] Obtain the target monitoring stream data of the power plant, and extract various types of data from the target monitoring stream data according to the data types required by the original computing task and the extraction strategies of various types of data;

[0010] The various types of extracted data are allocated to the various sub-computing tasks of the original computing task according to the processing logic diagram to obtain the target computing task.

[0011] The technical solution of the present invention is as follows: First, determine the original calculation task according to the business requirements of the power plant; generate a processing logic diagram of the original calculation task based on the operation of the execution order of each sub-calculation task of the original calculation task; obtain the target monitoring flow data of the power plant, and extract various types of data from the target monitoring flow data according to the data types required by the original calculation task and the extraction strategies of various types of data; configure the extracted various types of data to each sub-calculation task of the original calculation task according to the processing logic diagram to obtain the target calculation task. In the above technical solution, by first determining the original calculation task according to the business requirements on any one of the servers in the server cluster, not only can the risk of single-point failure be avoided, but also a data basis is provided for generating the processing logic diagram of the original calculation task. Then, generate the processing logic diagram of the original calculation task based on the operation of the execution order of each sub-calculation task of the original calculation task, which can more clearly discover possible problems (such as unreasonable execution order) in the task process, so that timely adjustment can be made, thereby avoiding errors and delays in the task execution process and improving the execution efficiency and reliability of the task. In addition, through graphical representation, the relationship between each sub-task and its execution order can be more intuitively displayed, making the complex calculation process clear at a glance. This helps the staff to better understand and maintain the system. After that, obtain the target monitoring flow data of the power plant, and extract various types of data from the target monitoring flow data according to the data types required by the original calculation task and the extraction strategies of various types of data, which can avoid blindly searching and processing irrelevant data in a large amount of target monitoring flow data, and only extract the data related to the original calculation task. This not only reduces the interference caused by irrelevant data, but also reduces the workload and complexity of data processing, saves computing resources and time costs, thereby improving the data processing efficiency and enhancing the data accuracy and quality. Finally, configure the extracted various types of data to each sub-calculation task of the original calculation task according to the processing logic diagram to obtain the target calculation task, which ensures that each sub-calculation task receives the correct data, thereby improving the data processing efficiency and ensuring the accuracy of the subsequent execution of the target calculation task. In addition, configuring data according to the processing logic diagram can not only avoid the wrong transmission of data, but also reduce unnecessary processing steps, so that each sub-calculation task can quickly obtain the required data and start processing immediately, reducing the waiting time and redundancy of data transmission, thereby reducing the delay and improving the execution efficiency of the subsequent target calculation task, and solving the problems of single-point failure risk, slow processing speed and high delay existing in the prior art.

[0012] In a second aspect, an embodiment of the present invention further provides a data processing device, which is applied to any one of the servers in the server cluster. The device includes:

[0013] A determination module, configured to determine an original calculation task according to the business requirements of the power plant;

[0014] A generation module, configured to generate a processing logic diagram of the original computing task based on operations of the execution order of each sub-computing task of the original computing task;

[0015] An extraction module, configured to obtain target monitoring flow data of a power plant, and extract various types of data from the target monitoring flow data according to the data types required by the original computing task and the extraction strategies of various types of data;

[0016] A configuration module, configured to configure the extracted various types of data to each sub-computing task of the original computing task according to the processing logic diagram, to obtain a target computing task.

[0017] In a third aspect, an embodiment of the present invention further provides an electronic device, which includes:

[0018] At least one processor; and a memory communicatively connected to the at least one processor;

[0019] Wherein, the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor, so that the at least one processor can execute any one of the data processing methods in the first aspect.

[0020] In a fourth aspect, an embodiment of the present invention further provides a storage medium containing computer-executable instructions, and the computer-executable instructions implement any one of the data processing methods in the first aspect when executed by a computer processor.

[0021] It should be noted that the above computer instructions may be stored in whole or in part on a computer-readable storage medium. Wherein, the computer-readable storage medium may be packaged together with the processor of the data processing device, or may be separately packaged from the processor of the data processing device, and the present application does not make a limitation thereto.

[0022] For the descriptions of the second aspect, the third aspect, and the fourth aspect in this application, reference may be made to the detailed description of the first aspect; and for the beneficial effects of the descriptions of the second aspect, the third aspect, and the fourth aspect, reference may be made to the analysis of the beneficial effects of the first aspect, and details are not described herein again.

[0023] In this application, the names of the above data processing devices do not constitute a limitation to the devices or functional modules themselves. In actual implementation, these devices or functional modules may appear under other names. As long as the functions of each device or functional module are similar to those of this application, they belong to the scope of the claims of this application and their equivalent technologies.

[0024] These aspects or other aspects of this application will be more clearly understood in the following description. Description of the Drawings

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0026] Figure 1 It is a flowchart of a data processing method provided by an embodiment of the present invention;

[0027] Figure 2 It is a flowchart of another data processing method provided by an embodiment of the present invention;

[0028] Figure 3 It is a schematic structural diagram of a data processing device provided by an embodiment of the present invention;

[0029] Figure 4 It is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. Specific embodiments

[0030] The following will further elaborate on the present invention in conjunction with the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present invention, rather than limiting the present invention. Additionally, it should be noted that for the sake of description, only parts related to the present invention rather than all structures are shown in the drawings.

[0031] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone.

[0032] The terms "first" and "second" in the specification and drawings of this application are used to distinguish different objects or different processes for the same object, rather than to describe the specific order of the objects.

[0033] In addition, the terms "including" and "having" and any variations thereof mentioned in the description of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally further include other unlisted steps or units, or may optionally further include other steps or units inherent to these processes, methods, products, or devices.

[0034] It should be mentioned before discussing exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe various operations (or steps) as sequential processes, many operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of various operations can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to methods, functions, procedures, subroutines, subprograms, etc. In addition, the embodiments in the present invention and the features in the embodiments can be combined with each other without conflict.

[0035] It should be noted that, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0036] In the description of the present application, unless otherwise specified, “plurality” means two or more.

[0037] Figure 1 This is a flow chart of a data processing method provided by an embodiment of the present invention. This embodiment is applicable to the situation where a large amount of power plant time series data needs to be processed. The method can be executed by a data processing device, which can be implemented in software and / or hardware. Exemplarily, the device can be a computer or a server. Figure 1 The data processing method of this embodiment specifically includes the following steps:

[0038] Step 110: Determine the original computing task according to the business requirements of the power plant.

[0039] Specifically, a power plant refers to a factory that converts various primary energy sources (such as coal, wind, water, etc.) in nature into electrical energy (secondary energy). For example, power plants include thermal power plants, hydroelectric power plants, wind power plants, etc. Business requirements refer to requirements set in advance based on the operation, management, maintenance or development of the power plant. Original computing tasks refer to data processing tasks determined based on the business requirements of the power plant. Original computing tasks include sub-computing tasks.

[0040] In a specific implementation, any server in the server cluster of the power plant can determine the original computing task of the business demand in the correspondence table between the business demand and the computing task according to the business demand of the power plant.

[0041] It should be noted that the correspondence table between business requirements and computing tasks is established by staff in advance according to actual situations or requirements.

[0042] In this embodiment, by determining the original computing task on any one of the servers in the server cluster according to business requirements, not only can the risk of single-point failure be avoided, but also a data basis is provided for generating the processing logic diagram of the original computing task later.

[0043] Step 120: Generate a processing logic diagram of the original computing task based on the execution order of each sub-computing task of the original computing task.

[0044] Specifically, a sub-computing task refers to a component of the original computing task, and each sub-task is responsible for performing a specific data processing operation (such as summation, average calculation, data screening, data conversion, etc.). The execution order refers to the order in which each sub-computing task is executed in the entire original computing task. The processing logic diagram refers to a diagram used to show the execution order and logical relationship between the original computing task and its each sub-computing task.

[0045] In specific implementation, first, it can be determined whether there is a preset execution order of each sub-computing task of the original computing task. If there is a preset execution order, then determine the execution order of each sub-computing task of the original computing task as the preset execution order. If there is no preset execution order, then determine the execution order of each sub-computing task of the original computing task as the real-time execution order. After determining the execution order of each sub-computing task of the original computing task, identify the execution order of each sub-task in the original computing task and their dependency relationship from the determined execution order, and then generate a processing logic flow diagram of the original computing task based on the identified information using a graphical tool (such as Graphviz, Mermaid, etc.). Among them, the real-time execution order refers to the execution order of each sub-computing task of the original computing task determined according to the system running state, resource availability, and data flow dynamics.

[0046] In this embodiment, generating a processing logic diagram of the original computing task based on the execution order of each sub-computing task of the original computing task can more clearly discover possible problems (such as unreasonable execution order) in the task process, so that timely adjustment can be made, thereby avoiding errors and delays in the task execution process and improving the execution efficiency and reliability of the task. In addition, through graphical representation, the relationship between each sub-task and its execution order can be more intuitively shown, making the complex computing process clear at a glance. It helps staff better understand and maintain the system.

[0047] Step 130: Obtain the target monitoring flow data of the power plant, and extract various types of data from the target monitoring flow data according to the data types required by the original calculation task and the extraction strategies for various types of data.

[0048] Specifically, the target monitoring flow data refers to the continuous data flow collected in real time by various monitoring devices (such as sensors) during the operation of the power plant. The data types refer to the data types required by the original calculation task set in advance according to the actual situation or requirements (such as numerical type, temperature type, pressure type, etc.). The extraction strategies refer to the methods and rules for extracting the required data from the target monitoring flow data required by the original calculation task set in advance according to the actual situation or requirements. For example: The extraction strategy can be a forwarding strategy, a broadcast strategy, a key-value-based strategy, or a random strategy, etc.

[0049] In specific implementation, first, the target monitoring flow data of the power plant (such as sensor readings and device status) can be obtained from the time series database, then the data types required by the original calculation task and the extraction strategy are determined, and then various types of data are extracted from the target monitoring flow data according to the data types required by the original calculation task and the extraction strategies for various types of data. Specifically, the data types required by the original calculation task can be determined first according to the original calculation task in the correspondence table between the original calculation task and the data type, and at the same time, the extraction strategy of the original calculation task, that is, the extraction strategies for various types of data, can be determined according to the original calculation task in the correspondence table between the original calculation task and the extraction strategy, and then the required various types of data are extracted from the target monitoring flow data according to the data types required by the original calculation task and the extraction strategies for various types of data. For example: If the data type required by the original calculation task is temperature and the extraction strategy is the broadcast strategy, then all the data containing the temperature field are extracted from the target monitoring flow data.

[0050] It should be noted that the correspondence table between the original calculation task and the data type and the correspondence table between the original calculation task and the extraction strategy are determined in advance according to the actual situation or requirements.

[0051] In this embodiment, through the above steps, it is possible to avoid blindly searching and processing irrelevant data in a large amount of target monitoring flow data, and only extract the data related to the original calculation task, which not only reduces the interference caused by irrelevant data, but also reduces the workload and complexity of data processing, saves computing resources and time costs, thereby improving the data processing efficiency and enhancing the data accuracy and quality.

[0052] Step 140: Configure the extracted various types of data to each sub-calculation task of the original calculation task according to the processing logic diagram to obtain the target calculation task.

[0053] Specifically, the target calculation task refers to the calculation task formed after data configuration, which contains specific data and execution logic.

[0054] In a specific implementation, after extracting various types of data from the target monitoring stream data according to the data types required by the original computing task and the extraction strategies for various types of data, the processing logic diagram of the original computing task is parsed to obtain the execution order, mutual dependency relationships, and data flow directions of each sub-computing task in the original computing task. Then, according to the parsed content, it is determined to which sub-computing tasks the extracted various types of data should be respectively configured, and it is checked whether the format of the data to be allocated matches the input requirements of the corresponding sub-computing tasks. If not, corresponding data format conversion and adaptation operations are required. Finally, according to the determined corresponding relationships described above, the data that has undergone format conversion or adaptation is passed to the corresponding sub-computing tasks to obtain the target computing task.

[0055] In this embodiment, by configuring the extracted various types of data to each sub-computing task of the original computing task according to the processing logic diagram, the target computing task is obtained, ensuring that each sub-computing task receives the correct data, thereby improving the data processing efficiency and ensuring the accuracy of the subsequent execution of the target computing task. In addition, configuring data according to the processing logic diagram can not only avoid the incorrect transmission of data, but also reduce unnecessary processing steps, enabling each sub-computing task to quickly obtain the required data and immediately start processing, reducing the waiting time and redundancy of data transmission, thereby reducing the latency and improving the execution efficiency of the subsequent target computing task.

[0056] In the embodiment of the present invention, first, an original computing task is determined according to business requirements on any server in the server cluster, which can not only avoid the risk of single-point failure, but also provide a data basis for generating the processing logic diagram of the original computing task. Then, based on the operations of the execution order of each sub-computing task of the original computing task, the processing logic diagram of the original computing task is generated, which can more clearly discover possible problems (such as unreasonable execution order) in the task process, so that timely adjustment can be made, thereby avoiding errors and delays in the task execution process and improving the execution efficiency and reliability of the task. In addition, through graphical representation, the relationships between each sub-task and their execution order can be more intuitively displayed, making the complex computing process clear at a glance. This helps the staff to better understand and maintain the system. After that, the target monitoring flow data of the power plant is obtained, and various types of data are extracted from the target monitoring flow data according to the data types required by the original computing task and the extraction strategies of various types of data, which can avoid blindly searching and processing irrelevant data in a large amount of target monitoring flow data and only extract the data related to the original computing task. This not only reduces the interference caused by irrelevant data, but also reduces the workload and complexity of data processing, saves computing resources and time costs, thereby improving the data processing efficiency and enhancing the data accuracy and quality. Finally, the extracted various types of data are configured to each sub-computing task of the original computing task according to the processing logic diagram to obtain the target computing task, which ensures that each sub-computing task receives the correct data, thereby improving the data processing efficiency and ensuring the accuracy of the subsequent execution of the target computing task. In addition, configuring data according to the processing logic diagram can not only avoid the incorrect transmission of data, but also reduce unnecessary processing steps, enabling each sub-computing task to quickly obtain the required data and start processing immediately, reducing the waiting time and redundancy of data transmission, thereby reducing the delay and improving the execution efficiency of the subsequent target computing task, and solving the problems of single-point failure risk, slow processing speed and high delay existing in the prior art.

[0057] Figure 2 FIG. is a flowchart of another data processing method provided by an embodiment of the present invention. This embodiment is a specific implementation based on the above embodiment. In this embodiment, the method may further include:

[0058] Step 210: Determine an original computing task according to the business requirements of the power plant.

[0059] Step 211: Generate a processing logic diagram of the original computing task based on the operations of the execution order of each sub-computing task of the original computing task.

[0060] Further, step 211 may specifically include: when the real-time execution order of each sub-computation task of the original computation task is different from the preset execution order, updating the real-time execution order to the preset execution order, and generating a processing logic diagram of the original computation task based on the preset execution order; when the real-time execution order of each sub-computation task of the original computation task is the same as the preset execution order, generating a processing logic diagram of the original computation task based on the real-time execution order.

[0061] Specifically, the real-time execution order refers to the execution order of each sub-computation task of the original computation task determined according to the system running state, resource availability, and data flow dynamics. The preset execution order refers to the execution order of the sub-computation tasks preset in advance according to the actual situation or requirements.

[0062] In specific implementation, first, obtain the real-time execution order of each sub-computation task of the original computation task. Then, compare the real-time execution order of each sub-computation task of the original computation task with the preset execution order to obtain a comparison result. Then, take different measures according to the comparison result: if the real-time execution order is different from the preset execution order, update the real-time execution order to the preset execution order, and generate a processing logic diagram of the original computation task based on the preset execution order; if the real-time execution order is the same as the preset execution order, directly generate a processing logic diagram of the original computation task based on the current real-time execution order.

[0063] In this embodiment, through the above steps, the accuracy and stability of the computation task execution are ensured, and the overall efficiency and reliability are improved.

[0064] Further, when the number of original computation tasks is at least two, after step 211, it further includes: generating a comprehensive processing logic diagram according to the priorities of the at least two original computation tasks and the processing logic diagrams of the at least two original computation tasks; correspondingly, configuring the extracted various types of data to each sub-computation task of the original computation task according to the processing logic diagram to obtain a target computation task, including: configuring the extracted various types of data to each sub-computation task of the at least two original computation tasks according to the comprehensive processing logic diagram to obtain at least two target computation tasks.

[0065] Specifically, the comprehensive processing logic diagram refers to a graphical representation including the execution order of all original computation tasks and their sub-computation tasks generated according to the priorities of these original computation tasks and their corresponding processing logic diagrams when there are at least two original computation tasks.

[0066] In specific implementation, first, determine the priorities of each original computing task according to the actual situation or requirements. Then, after obtaining the processing logic diagrams of at least two original computing tasks, generate a comprehensive processing logic diagram according to the priorities of each original computing task and its corresponding processing logic diagram. Specifically, the processing logic diagram of the original computing task with the highest priority can be used as the basic framework, and then, in the order of decreasing priority, gradually integrate the processing logic diagrams of other original computing tasks into the basic framework to obtain the comprehensive processing logic diagram. Finally, from the various types of data extracted previously, according to the data flow direction and dependency relationships specified in the comprehensive processing logic diagram, accurately configure the corresponding data to each sub-computing task to obtain at least two target computing tasks.

[0067] In this embodiment, through the above steps, the execution processes of at least two original computing tasks are integrated, making the overall management and scheduling work more intuitive and efficient, enhancing task coordination, and at the same time clearly showing the dependency relationships between tasks, which helps to identify and resolve potential dependency conflicts, ensures that tasks are executed in the correct order, and thus improves the accuracy of data processing.

[0068] Step 212: Obtain the first monitored flow data within the target period.

[0069] Specifically, the target period refers to a specific time period determined according to the actual situation or requirements. The first monitored flow data refers to the monitored flow data within the target period.

[0070] In specific implementation, any one of the servers in the server cluster of the power plant can obtain the first monitored flow data from the database storing the power plant monitored flow data based on the target period.

[0071] In this embodiment, by obtaining the first monitored flow data within the target period, a data basis is provided for determining the target monitored flow data later.

[0072] Furthermore, step 212 may specifically include: obtaining the original monitored flow data within the target period from the time series database of the power plant; determining the original monitored flow data that meets the device measurement range of the data acquisition device corresponding to the original monitored flow data as the first monitored flow data.

[0073] Specifically, the time series database refers to a database specifically used for storing and managing data that changes over time. For example, the time series database can be Apache IoTDB. The original monitored flow data refers to the original data that is collected in real time by various data acquisition devices in the power plant without any processing or screening. In this embodiment, the first monitored flow data may also refer to the original monitored flow data within the target period obtained from the time series database of the power plant that meets the device measurement range of the corresponding data acquisition device.

[0074] In a specific implementation, any one of the servers in the server cluster of the power plant can first obtain the original monitoring stream data within the target period from the time series database of the power plant. Then, through the interface (such as the DataStream API) of the distributed computing framework (such as Apache Flink) configured on the server, the original monitoring stream data that meets the device measurement range of the data acquisition device corresponding to the original monitoring stream data is filtered out, and the data that meets the device measurement range is determined as the first monitoring stream data.

[0075] In this embodiment, through the above steps, the unstable working condition data in the original monitoring stream data is removed, thereby ensuring that the data for subsequent processing is valid.

[0076] Step 213: Compare the timestamps of the first monitoring stream data to obtain the maximum timestamp.

[0077] In a specific implementation, after obtaining the first monitoring stream data, the timestamps of the first monitoring stream data can be compared to obtain the maximum timestamp.

[0078] In this embodiment, by determining the maximum timestamp, a data basis is provided for obtaining the target timestamp later.

[0079] Step 214: Calculate the sum of the maximum timestamp and the maximum delay time corresponding to the original calculation task to obtain the target timestamp.

[0080] Specifically, the maximum delay time refers to a time threshold set for the original calculation task according to the business requirements of the power plant and the data processing requirements. The target timestamp refers to the timestamp obtained by calculating the sum of the maximum timestamp and the maximum delay time corresponding to the original calculation task, and is used to determine the time boundary for screening the data of the next period.

[0081] In a specific implementation, in practical applications, it can be determined that the target timestamp = the maximum timestamp + the maximum delay time corresponding to the original calculation task.

[0082] In this embodiment, by determining the target timestamp, a data basis is provided for obtaining the supplementary monitoring stream data later.

[0083] Step 215: Obtain the second monitoring stream data in the next period of the target period, and filter out the data with timestamps less than the target timestamp from the second monitoring stream data to obtain the supplementary monitoring stream data.

[0084] Specifically, the second monitoring stream data refers to the monitoring stream data obtained from the database storing the power plant monitoring stream data in the next period of the target period. The supplementary monitoring stream data refers to the data with timestamps less than the target timestamp filtered out from the second monitoring stream data, and is used to supplement the first monitoring stream data to ensure the integrity and accuracy of the calculation task.

[0085] In a specific implementation, the second monitoring flow data can be obtained from the database storing the power plant monitoring flow data based on the next cycle of the target cycle, and then the data with a timestamp less than the target timestamp can be filtered out from the obtained second monitoring flow data to obtain the supplementary monitoring flow data.

[0086] In this embodiment, through the above steps, a data basis is provided for determining the target monitoring flow data later.

[0087] Step 216: Determine the first monitoring flow data and the supplementary monitoring flow data as the target monitoring flow data.

[0088] Specifically, in this embodiment, the target monitoring flow data refers to the data set jointly composed of the first monitoring flow data and the supplementary monitoring flow data.

[0089] In a specific implementation, after obtaining and filtering out the first monitoring flow data and the supplementary monitoring flow data, the first monitoring flow data and the supplementary monitoring flow data can be merged to obtain the target monitoring flow data, ensuring that the data that arrives late due to delay is not missed, thereby improving the integrity and reliability of the data.

[0090] Step 217: Extract various types of data from the target monitoring flow data according to the data types required by the original calculation task and the extraction strategies of various types of data.

[0091] Step 218: Configure the extracted various types of data to each sub-calculation task of the original calculation task according to the processing logic diagram to obtain the target calculation task.

[0092] Further, after step 218, it further includes: executing the target calculation task to obtain the target value of the target calculation task; storing the target value.

[0093] Specifically, the target value refers to the final calculation result obtained after executing the target calculation task.

[0094] In a specific implementation, after obtaining the target calculation task, according to the execution order and dependency relationship defined in the processing logic diagram, each sub-calculation task in the target calculation task can be executed in sequence to obtain the complete output of the target calculation task, that is, the target value. Subsequently, this target value can be stored in the corresponding database (such as the time series database of the power plant) for subsequent use and analysis.

[0095] In this embodiment, by executing the target calculation task to obtain the target value of the target calculation task, a large amount of data can be processed in a short time and the result can be obtained quickly, which helps to achieve faster decision support and improve the business response speed. Then storing the target value is convenient for its long-term preservation and subsequent query and use.

[0096] In one implementation, after obtaining the target computing task, the computing resources of each server in the server cluster can also be obtained, and then the target server for executing the target computing task is determined according to the computing resources of each server. After that, the target computing task is sent to the target server so that the target server executes the target computing task to obtain the target value of the target computing task and stores the target value.

[0097] Optionally, the target computing task includes an execution function and a window operation.

[0098] Further, executing the target computing task to obtain the target value of the target computing task includes: dividing the data of the target computing task based on the window operation to obtain each sub-data of the target computing task; and respectively substituting each sub-data into the execution function to obtain the target value.

[0099] Specifically, the execution function refers to a function that performs specific calculations and processing on data in the target computing task, which defines the processing logic and algorithm of the data. For example, the execution function can be a specific function for calculating the average value, summing, finding the minimum value, finding the maximum value, etc. The window operation refers to a data processing technique used to divide the data of the target computing task into smaller and more manageable data subsets (i.e., windows). For example, the window operation can be a rolling window based on quantity, a rolling window based on time, a sliding window, or a session window, etc.

[0100] In a specific implementation, first, the parameters corresponding to the window operation of the target computing task (such as the time length of the window and the number of data items in the window) are obtained. Then, the data of the target computing task is divided by using the obtained window operation parameters, so as to obtain each sub-data of the target computing task. For example, if the data of the target computing task is the temperature data from 15:00 to 15:30 on February 24, 2025, the window operation is a rolling window based on time, and the window operation parameter is 10 minutes, then the sub-data of the target computing task includes: the temperature data from 15:00 to 15:10 on February 24, 2025, the temperature data from 15:10 to 15:20 on February 24, 2025, and the temperature data from 15:20 to 15:30 on February 24, 2025. Finally, the execution function is applied to each sub-data to obtain the target value.

[0101] In this embodiment, through the above steps, the data can be processed in batches, reducing the memory pressure, improving the computing efficiency, and reducing the overall computing time.

[0102] The data processing method provided by the embodiments of the present invention first determines the original calculation task on any server in the server cluster according to business requirements, which can not only avoid the risk of single-point failure, but also provide a data basis for generating the processing logic diagram of the original calculation task later. Then, based on the operation of the execution order of each sub-calculation task of the original calculation task, the processing logic diagram of the original calculation task is generated, which can more clearly discover possible problems in the task process (such as unreasonable execution order), so that adjustments can be made in time, thereby avoiding errors and delays in the task execution process, and improving the execution efficiency and reliability of the task. In addition, through graphical representation, the relationship between each sub-task and its execution order can be more intuitively displayed, making the complex calculation process clear at a glance. This helps the staff to better understand and maintain the system. After that, the first monitoring flow data in the target period is obtained, the timestamps of the first monitoring flow data are compared to obtain the maximum timestamp, the sum of the maximum timestamp and the maximum delay time corresponding to the original calculation task is calculated to obtain the target timestamp, the second monitoring flow data in the next period of the target period is obtained, and the data with timestamps less than the target timestamp is screened out from the second monitoring flow data to obtain the supplementary monitoring flow data. The first monitoring flow data and the supplementary monitoring flow data are determined as the target monitoring flow data, ensuring that the data that arrives late due to delay is not missed, thereby improving the integrity and reliability of the data. Then, according to the data types required by the original calculation task and the extraction strategies of various types of data, various types of data are extracted from the target monitoring flow data, which can avoid blindly searching and processing irrelevant data in a large amount of target monitoring flow data, and only extract the data related to the original calculation task. This not only reduces the interference caused by irrelevant data, but also reduces the workload and complexity of data processing, saves computing resources and time costs, thereby improving the data processing efficiency and enhancing the data accuracy and quality. Finally, the extracted various types of data are configured to each sub-calculation task of the original calculation task according to the processing logic diagram to obtain the target calculation task, ensuring that each sub-calculation task receives the correct data, thereby improving the data processing efficiency and ensuring the accuracy of the subsequent execution of the target calculation task. In addition, configuring data according to the processing logic diagram can not only avoid the wrong transmission of data, but also reduce unnecessary processing steps, enabling each sub-calculation task to quickly obtain the required data and start processing immediately, reducing the waiting time and redundancy of data transmission, thereby reducing the delay and improving the execution efficiency of the subsequent target calculation task, and solving the problems of single-point failure risk, slow processing speed and high delay existing in the prior art.

[0103] Figure 3 FIG. is a schematic structural diagram of a data processing device provided by an embodiment of the present invention. This device and the data processing methods of the above embodiments belong to the same inventive concept. For the details not described in detail in the embodiments of the data processing device, reference can be made to the embodiments of the above data processing methods.

[0104] As Figure 3 shown, the device includes:

[0105] A determination module 310, configured to determine an original calculation task according to the service requirements of the power plant;

[0106] A generation module 320, configured to generate a processing logic diagram of the original calculation task based on an operation of the execution order of each sub-calculation task of the original calculation task;

[0107] An extraction module 330, configured to obtain target monitoring flow data of the power plant, and extract various types of data from the target monitoring flow data according to the data types required by the original calculation task and the extraction strategies of various types of data;

[0108] A configuration module 340, configured to configure the extracted various types of data to each sub-calculation task of the original calculation task according to the processing logic diagram to obtain a target calculation task.

[0109] Based on the above embodiments, the extraction module 330 obtains the target monitoring flow data of the power plant, including:

[0110] Obtain the first monitoring flow data within a target period;

[0111] Compare the timestamps of the first monitoring flow data to obtain the maximum timestamp;

[0112] Calculate the sum of the maximum timestamp and the maximum delay time corresponding to the original calculation task to obtain a target timestamp;

[0113] Obtain the second monitoring flow data within the next period of the target period, and filter out the data with timestamps less than the target timestamp from the second monitoring flow data to obtain supplementary monitoring flow data;

[0114] Determine the first monitoring flow data and the supplementary monitoring flow data as the target monitoring flow data.

[0115] Based on the above embodiments, the extraction module 330 obtains the first monitoring flow data within a target period, including:

[0116] Obtain the original monitoring flow data within the target period from the time series database of the power plant;

[0117] Determine the original monitoring flow data that meets the device measurement range of the data acquisition device corresponding to the original monitoring flow data as the first monitoring flow data.

[0118] Based on the above embodiments, the generation module 320 is specifically configured to:

[0119] In the case where the real-time execution order of each sub-computation task of the original computation task is different from the preset execution order, update the real-time execution order to the preset execution order, and generate a processing logic diagram of the original computation task based on the preset execution order;

[0120] In the case where the real-time execution order of each sub-computation task of the original computation task is the same as the preset execution order, generate a processing logic diagram of the original computation task based on the real-time execution order.

[0121] Based on the above embodiments, when the number of the original computation tasks is at least two, the apparatus further includes:

[0122] An integrated generation module, configured to generate an integrated processing logic diagram according to the priorities of at least two original computation tasks and the processing logic diagrams of the at least two original computation tasks after generating the processing logic diagram of the original computation task based on the operations on the execution orders of each sub-computation task of the original computation task;

[0123] Correspondingly, the configuration module 340 is specifically configured to:

[0124] Configure the extracted various types of data to each sub-computation task of the at least two original computation tasks according to the integrated processing logic diagram, to obtain at least two target computation tasks.

[0125] Based on the above embodiments, the apparatus further includes:

[0126] An execution module, configured to execute the target computation task after obtaining the target computation task, to obtain a target value of the target computation task; and store the target value.

[0127] Based on the above embodiments, the target computation task includes an execution function and a window operation. When the execution module executes the target computation task to obtain the target value of the target computation task, it includes:

[0128] Dividing the data of the target computation task based on the window operation, to obtain each sub-data of the target computation task;

[0129] Substituting each sub-data into the execution function respectively, to obtain the target value.

[0130] The data processing apparatus provided by the embodiments of the present invention can execute the data processing method provided by any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method.

[0131] It should be noted that in the embodiments of the above data processing device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of mutual distinction and do not limit the protection scope of the present invention.

[0132] Figure 4 FIG. is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. Figure 4 FIG. shows a block diagram of an exemplary electronic device 4 suitable for implementing the embodiments of the present invention. Figure 4 The shown electronic device 4 is only an example and should not impose any limitation on the functions and usage scope of the embodiments of the present invention.

[0133] As Figure 4 shown, the electronic device 4 is presented in the form of a general-purpose computing electronic device. The components of the electronic device 4 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 connecting different system components (including the system memory 28 and the processing unit 16).

[0134] The bus 18 represents one or more of several types of bus structures, including a memory bus or a memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus structures. By way of example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0135] The electronic device 4 typically includes a variety of computer system readable media. These media can be any available media accessible by the electronic device 4, including volatile and non-volatile media, removable and non-removable media.

[0136] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The electronic device 4 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 can be used to read and write non-removable, non-volatile magnetic media ( Figure 4 not shown, commonly referred to as a "hard disk drive"). Although Figure 4not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 through one or more data medium interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0137] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods in the embodiments described in the present invention.

[0138] Electronic device 4 may also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and may also communicate with one or more devices that enable a user to interact with the electronic device 4, and / or communicate with any device that enables the electronic device 4 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be through an input / output (I / O) interface 22. Also, electronic device 4 may communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public network, such as the Internet) through network adapter 20. As Figure 4 shown, network adapter 20 communicates with other modules of electronic device 4 through bus 18. It should be understood that although Figure 4 not shown, other hardware and / or software modules may be used in conjunction with electronic device 4, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0139] Processing unit 16 executes various functional applications and page displays by running programs stored in system memory 28. For example, it implements the data processing method provided by the embodiments of the present invention, which is applied to any one of the servers in the server cluster. The method includes:

[0140] Determine the original computing task according to the business requirements of the power plant;

[0141] Generate a processing logic diagram of the original computing task based on operations of the execution order of each sub-computing task of the original computing task;

[0142] Obtain the target monitoring flow data of the power plant, and extract various types of data from the target monitoring flow data according to the data types required by the original calculation task and the extraction strategies for various types of data;

[0143] Configure the extracted various types of data to each sub-calculation task of the original calculation task according to the processing logic diagram to obtain a target calculation task.

[0144] Of course, those skilled in the art can understand that the processor can also implement the technical solutions of the data processing methods provided in any embodiment of the present invention.

[0145] An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements, for example, the data processing method provided in the embodiment of the present invention, and is applied to any one of the servers in the server cluster. The method includes:

[0146] Determine the original calculation task according to the business requirements of the power plant;

[0147] Generate a processing logic diagram of the original calculation task based on the operations of the execution order of each sub-calculation task of the original calculation task;

[0148] Obtain the target monitoring flow data of the power plant, and extract various types of data from the target monitoring flow data according to the data types required by the original calculation task and the extraction strategies for various types of data;

[0149] Configure the extracted various types of data to each sub-calculation task of the original calculation task according to the processing logic diagram to obtain a target calculation task.

[0150] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the computer-readable storage medium include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer-readable storage medium can be any tangible medium that contains or stores a program, and the program can be used by or in combination with an instruction execution system, apparatus, or device.

[0151] A computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which computer-readable program code is carried. Such a propagated data signal may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device.

[0152] The program code contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0153] The computer program code for performing the operations of the present invention may be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0154] Those of ordinary skill in the art should understand that the various modules or steps of the present invention described above may be implemented using a general-purpose computing device. They may be concentrated on a single computing device or distributed over a network composed of multiple computing devices. Optionally, they may be implemented using program code executable by a computer device, so that they can be stored in a storage device and executed by the computing device, or they may be separately fabricated into individual integrated circuit modules, or multiple modules or steps among them may be fabricated into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0155] In addition, the acquisition, storage, use, processing, etc. of data in the technical solution of the present invention all comply with the relevant provisions of national laws and regulations.

[0156] Note that the above is only a preferred embodiment of the present invention and the technical principles applied. Those skilled in the art will understand that the present invention is not limited to the specific embodiments here, and various obvious changes, re-adjustments and substitutions can be made by those skilled in the art without departing from the protection scope of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments only. Without departing from the concept of the present invention, more other equivalent embodiments can be included, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A data processing method, characterized in that: Applied to any server in a server cluster, the method comprises: Determine the original computing tasks according to the business needs of the power plant; Generate a processing logic diagram of the original computing task based on the operation of the execution order of each sub-computing task of the original computing task; Obtain the target monitoring stream data of the power plant, and extract various types of data from the target monitoring stream data according to the data types required by the original computing task and the extraction strategies of various types of data; The various types of extracted data are allocated to the various sub-computing tasks of the original computing task according to the processing logic diagram to obtain the target computing task.

2. The data processing method according to claim 1, characterized in that: Obtain target monitoring stream data of the power plant, including: Obtaining the first monitoring flow data within the target period; Comparing the timestamps of the first monitoring flow data to obtain a maximum timestamp; Calculate the sum of the maximum timestamp and the maximum delay time corresponding to the original computing task to obtain a target timestamp; Acquire second monitoring flow data in a next cycle of the target cycle, and filter out data with a timestamp smaller than the target timestamp from the second monitoring flow data to obtain supplementary monitoring flow data; The first monitoring flow data and the supplementary monitoring flow data are determined as the target monitoring flow data.

3. The data processing method according to claim 2, characterized in that: Obtain the first monitoring flow data within the target period, including: Acquire the original monitoring flow data within the target period from the time series database of the power plant; The original monitoring stream data that meets the device measurement range of the data acquisition device corresponding to the original monitoring stream data is determined as the first monitoring stream data.

4. The data processing method according to claim 1, characterized in that: Generating a processing logic diagram of the original computing task based on the operation of the execution order of each sub-computing task of the original computing task includes: When the real-time execution order of each sub-computing task of the original computing task is different from the preset execution order, the real-time execution order is updated to the preset execution order, and a processing logic diagram of the original computing task is generated based on the preset execution order; In a case where the real-time execution order of each sub-computing task of the original computing task is the same as the preset execution order, a processing logic diagram of the original computing task is generated based on the real-time execution order.

5. The data processing method according to claim 1, characterized in that: When the number of the original computing tasks is at least two, after generating a processing logic diagram of the original computing tasks based on an operation on the execution order of each sub-computing task of the original computing tasks, the method further includes: generating a comprehensive processing logic diagram according to the priorities of at least two original computing tasks and the processing logic diagrams of the at least two original computing tasks; Accordingly, the extracted data are assigned to the sub-computing tasks of the original computing task according to the processing logic diagram to obtain the target computing task, including: The various types of extracted data are allocated to the respective sub-computing tasks of the at least two original computing tasks according to the comprehensive processing logic diagram to obtain at least two target computing tasks.

6. The data processing method according to claim 1, characterized in that: After obtaining the target computing task, it also includes: Execute the target computing task to obtain the target value of the target computing task; The target value is stored.

7. The data processing method according to claim 6, characterized in that: The target calculation task includes executing a function and a window operation, executing the target calculation task, and obtaining a target value of the target calculation task, including: Dividing the data of the target computing task based on the window operation to obtain sub-data of the target computing task; Substitute each of the sub-data into the execution function to obtain the target value.

8. A data processing device, characterized in that: Applied to any server in a server cluster, the device comprises: A determination module, used to determine the original computing tasks according to the business needs of the power plant; A generating module, configured to generate a processing logic diagram of the original computing task based on an operation on the execution order of each sub-computing task of the original computing task; An extraction module is used to obtain the target monitoring stream data of the power plant and extract various types of data from the target monitoring stream data according to the data types required by the original computing task and the extraction strategies of various types of data; The configuration module is used to configure the extracted various types of data to each sub-computing task of the original computing task according to the processing logic diagram to obtain the target computing task.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data processing method described in any one of claims 1 to 7.

10. A storage medium containing computer executable instructions, characterized in that: The computer executable instructions are used to execute the data processing method according to any one of claims 1 to 7 when executed by a computer processor.

Images