Production data missing filling method and device, equipment and medium

Abstract

The invention provides a production data missing filling method and device, equipment and a medium, and relates to the technical field of data processing, and the method comprises the steps: obtaining a plurality of original data sets of a power plant in a target time period; according to each original data set, generating original DataSet data; according to a time difference between adjacent acquisition timestamps in the original DataSet data, determining whether the original DataSet data lacks an acquisition timestamp or not; if it is determined that the acquisition timestamp is missing in the original DataSet data, performing timestamp filling on the original DataSet data to obtain first updated DataSet data, and performing numerical value filling on the first updated DataSet data according to a preset data filling rule to obtain target DataSet data; if it is determined that the collection timestamp is not missing in the original DataSet data, numerical value filling is conducted on the original DataSet data according to a preset data filling rule, and target DataSet data is obtained. The problems that an existing method for filling missing production data is low in filling efficiency, and the integrity of the data is difficult to guarantee can be effectively solved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method for filling missing production data, a device for filling missing production data, an electronic device, and a machine-readable storage medium. Background Art

[0002] With the rapid development of the power industry, the production data of power plants (such as thermal power plants) has shown an exponential growth. However, due to uncontrollable factors such as network failures, some production data may be lost. This data loss will have a negative impact on the operation, management and safety of thermal power plants. Therefore, effective measures need to be taken to ensure the integrity of the data.

[0003] In related technologies, in order to improve the integrity of power plant production data, it is necessary to first pull the original production data, and then fill in the missing data within a period of time. However, as the power plant continues to operate, the amount of data continues to increase, and the amount of missing data also gradually increases. Therefore, how to efficiently fill in the missing data in such a huge amount of data has become an urgent problem to be solved. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a method for filling missing production data, a device, equipment and medium for filling missing production data, so as to solve the problem that the method for filling missing production data in the prior art has low filling efficiency and difficulty in ensuring data integrity.

[0005] In order to achieve the above-mentioned purpose, the first aspect of the present application provides a method for filling missing production data, the method comprising: Acquire multiple original data sets of the power plant within a target time period; wherein each original data set includes a collection timestamp, and some original data sets include historical values ​​of at least one production parameter, and each historical value in the original data set corresponds to the collection timestamp; Generate original DataSet data according to each original data set; wherein the original DataSet data includes multiple rows and columns of data, each original data set corresponds to a row of data in the original DataSet data, and historical values ​​of the same data type in each original data set are located in the same column in the original DataSet data, wherein the data type is a collection timestamp or a production parameter; Determine whether the collection timestamp is missing in the original DataSet data according to the time difference between adjacent collection timestamps in the original DataSet data; If it is determined that the collection timestamp is missing in the original DataSet data, the timestamp is filled in the original DataSet data to obtain the first updated DataSet data, and the first updated DataSet data is numerically filled according to the preset data filling rule to obtain the target DataSet data; wherein the collection timestamps in the first updated DataSet data and the filled new timestamps are arranged in chronological order; If it is determined that the collection timestamp is not missing in the original DataSet data, the original DataSet data is numerically filled according to the preset data filling rule to obtain the target DataSet data.

[0006] In the embodiment of the present application, original DataSet data is generated according to each original data set, including: Convert each original data set into target data of DataSet[row] type; Based on the order of collection timestamps, the original DataSet data is assembled and generated according to each target data.

[0007] In an embodiment of the present application, determining whether the original DataSet data is missing a collection timestamp according to the time difference between adjacent collection timestamps in the original DataSet data includes: If the time difference between each two adjacent acquisition timestamps in the original DataSet data and the numerical relationship between the preset acquisition time step both meet the preset conditions, it is determined that there is no missing acquisition timestamp in the original DataSet data.

[0008] In the embodiment of the present application, the original DataSet data is timestamped to obtain the first updated DataSet data, including: For each two adjacent acquisition timestamps in the original DataSet data, if the numerical relationship between the time difference between the two acquisition timestamps and the preset acquisition time step does not meet the preset conditions, then the number of new timestamps that need to be filled between the two acquisition timestamps is determined based on the time difference between the two acquisition timestamps and the preset acquisition time step. Based on the number, the new timestamp that needs to be filled is determined according to any one of the two acquisition timestamps and the preset acquisition time step, and new row data is generated based on the new timestamp, and the new row data is inserted into the original DataSet data.

[0009] In the embodiment of the present application, according to the preset data filling rule, the first update DataSet data is numerically filled to obtain the target DataSet data, including: For each production parameter in the first updated DataSet data, the data whose historical values ​​are not empty in the column data corresponding to the production parameter are determined as the baseline data of the production parameter. Based on the baseline data of the production parameter and the column data corresponding to the collection timestamp in the first updated DataSet data, according to the preset data filling rules, the items whose historical values ​​are empty in the column data corresponding to the production parameter are filled with values ​​to obtain the target parameter value data; wherein the target parameter value data includes the baseline data of the production parameter and the new value obtained after the numerical filling.

[0010] In the embodiment of the present application, based on the baseline data of the production parameters and the column data corresponding to the acquisition timestamp in the first updated DataSet data, according to the preset data filling rule, the items with empty historical values ​​in the column data corresponding to the production parameters are numerically filled to obtain the target parameter value data, including: Based on the baseline data of the production parameters and the column data corresponding to the collection timestamp in the first updated DataSet data, according to the preset data filling rules, the lit function is used to fill in the items with empty historical values ​​in the column data corresponding to the production parameters to obtain the target parameter value data.

[0011] In the embodiment of the present application, according to the preset data filling rule, the original DataSet data is numerically filled to obtain the target DataSet data, including: For each production parameter in the original DataSet data, if there are items with empty values ​​in the column data corresponding to the production parameter, the data with non-empty historical values ​​in the column data corresponding to the production parameter are determined as the baseline data of the production parameter, and based on the baseline data of the production parameter and the column data corresponding to the collection timestamp in the original DataSet data, according to the preset data filling rules, the items with empty historical values ​​in the column data corresponding to the production parameter are filled with values ​​to obtain the target parameter value data; wherein, the target parameter value data includes the baseline data of the production parameter and the new value obtained after the value filling.

[0012] A second aspect of the present application provides a device for filling missing production data, the device comprising: The original data set acquisition module is used to acquire multiple original data sets of the power plant within a target time period; wherein each original data set includes a collection timestamp, and some original data sets include a historical value of at least one production parameter, and each historical value in the original data set corresponds to the collection timestamp; A DataSet data generation module is used to generate original DataSet data according to each original data set; wherein the original DataSet data includes multiple rows and columns of data, each original data set corresponds to a row of data in the original DataSet data, and historical values ​​of the same data type in each original data set are located in the same column in the original DataSet data, wherein the data type is a collection timestamp or a production parameter; A timestamp missing judgment module is used to determine whether the original DataSet data is missing a collection timestamp based on the time difference between adjacent collection timestamps in the original DataSet data; A first value filling module is used to fill the original DataSet data with the timestamp if it is determined that the collection timestamp is missing in the original DataSet data to obtain the first updated DataSet data, and to fill the first updated DataSet data with the values ​​according to the preset data filling rule to obtain the target DataSet data; wherein the collection timestamps in the first updated DataSet data and the filled new timestamps are arranged in chronological order; The second value filling module is used to fill in the original DataSet data with values ​​according to a preset data filling rule to obtain the target DataSet data if it is determined that the acquisition timestamp is not missing in the original DataSet data.

[0013] The third aspect of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method for filling missing production data as described in the first aspect when executing the computer program.

[0014] A fourth aspect of the present application provides a machine-readable storage medium having instructions stored thereon, which, when executed by a processor, configures the processor to execute the method for filling missing production data described in the first aspect above.

[0015] The method, device, equipment and medium for filling missing production data provided by the present application first convert each acquired original data set of the power plant into a data type supported by Spark SQL to obtain the original DataSet data, and then determine whether there is a breakpoint caused by the missing acquisition timestamp based on the time difference between adjacent acquisition timestamps in the original DataSet data. When it is determined that there is a missing acquisition timestamp, timestamp filling is first performed to ensure the continuity of the data time series. Finally, when the timestamp filling is completed or it is determined that there is no missing acquisition timestamp in the original DataSet data, based on the complete acquisition timestamp, numerical filling is performed according to the current DataSet data (i.e., the first updated DataSet data or the original DataSet data) and preset data filling rules to obtain the target DataSet data. On the premise of ensuring the integrity of the production data after filling, the data filling efficiency is improved based on the characteristics of the DataSet.

[0016] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present application, but do not constitute a limitation on the embodiments of the present application. In the accompanying drawings: Figure 1 A schematic diagram schematically shows a flow chart of a method for filling missing production data according to an embodiment of the present application; Figure 2 A schematic diagram of a structure of a device for filling missing production data according to an embodiment of the present application is shown; Figure 3 The internal structure diagram of the computer device according to the embodiment of the present application is schematically shown.

[0018] Description of Reference Numerals A01-processor; A02-network interface; A03-internal memory; A04-display screen; A05-input device; A06-non-volatile storage medium; B01-operating system; B02-computer program. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application, and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0020] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back, etc.), such directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0021] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0022] In view of the fact that the methods for filling missing production data in the related art have low filling efficiency and difficulty in ensuring the integrity of the data, the present application provides a method, device, equipment and medium for filling missing production data. The method, device, equipment and medium for filling missing production data provided by the present application are described in detail below in conjunction with the accompanying drawings through specific examples and implementation methods.

[0023] Figure 1 The flowchart of the method for filling missing production data of the embodiment of the present application is schematically shown. Figure 1 As shown, in one embodiment of the present application, a method for filling missing production data is provided, and the method may include the following steps.

[0024] Step 200: Acquire multiple original data sets of the power plant within a target time period.

[0025] Each of the original data sets includes a collection timestamp, and some of the original data sets include a historical value of at least one production parameter, and each historical value in the original data set corresponds to the collection timestamp.

[0026] Specifically, the collection timestamp indicates the time point when the historical value of the production parameter is collected, and the historical value of the production parameter is the production data; each historical value in the original data set corresponds to the collection timestamp, which means that for each original data set containing historical values, each historical value in the original data set is the value of the production parameter at the collection timestamp in the original data set.

[0027] It should be noted that the target time period refers to the time period corresponding to the current production data missing filling. By continuously updating the target time period, the production data generated during the entire operation cycle of the power plant can be filled. In other words, the embodiment of the present application can obtain multiple newly generated original data sets of the power plant once every preset time period (the length of which is the length of the target time period), and execute subsequent steps to continuously fill the missing of the power plant production data.

[0028] Step 300, generating original DataSet data according to each original data set.

[0029] The original DataSet data includes multiple rows and columns of data, each original data set corresponds to a row of data in the original DataSet data, and the historical values ​​of the same data type in each original data set are located in the same column in the original DataSet data, wherein the data type is a collection timestamp or a production parameter.

[0030] Specifically, DataSet is a powerful and flexible data processing abstraction in Spark. It is a strongly typed distributed data set that combines the flexibility of RDD (Resilient Distributed Dataset) and the optimization capability of DataFrame. Through its powerful optimization mechanism, it can significantly improve the efficiency and performance of data processing.

[0031] To generate the original DataSet data, before step 200, the method may further include the following steps: Step 100: Build a Spark cluster.

[0032] In a specific implementation, the step 100 includes the following steps: Step 110, determine the hardware and operating system: select multiple servers (such as more than 3) as Master nodes and Worker nodes. Each server has the same version of the operating system (such as Ubuntu, CentOS) installed, each server can access each other, and the firewall rules of each server allow necessary port communication.

[0033] Step 120, install Java: Install Java on each server through a package manager (such as yum) or other installation methods.

[0034] Step 130, download Spark: obtain the Spark binary release package by downloading it from the Apache Spark official website or other acquisition methods, and decompress the Spark binary release package to the same directory on each server.

[0035] Step 140, configure Spark: edit the conf / spark-env.sh file and set the necessary environment variables (such as JAVA_HOME); edit the conf / slaves file to list the host names or IP addresses of all Worker nodes; edit conf / spark-env.sh on the Master node and set SPARK_MASTER_HOST to the host name or IP address of the Master node; edit conf / spark-env.sh on the Worker node and set parameters such as SPARK_WORKER_CORES and SPARK_WORKER_MEMORY.

[0036] For example, the configuration of spark-env.sh includes: export SPARK_MASTER_HOST= <master-ip> export SPARK_WORKER_CORES=4 export SPARK_WORKER_MEMORY=8G export JAVA_HOME= / usr / lib / jvm / java-8-openjdk-amd64 Step 150, start the Spark cluster: Run sbin / start-master.sh on the Master node and sbin / start-slave.sh spark: / / on each Worker node <master-ip>:7077.

[0037] Step 160, verify the cluster: Open a browser and access http: / / <master-ip>:8080 to view the Spark cluster UI; run a Spark sample job on the Master node to verify that the cluster is working properly.

[0038] The embodiment of the present application performs a value addition operation (including collecting timestamps and parameter values ​​of production parameters) on a Spark cluster, which can fully utilize cluster resources for parallel processing and greatly improve the processing speed of the value addition.

[0039] In a specific implementation, step 300 includes the following steps: Step 310, convert each original data set into target data of DataSet[row] type.

[0040] In one example, when the original data set obtained is in JSON format, it is first converted into List type data of a Java entity class, and then the serialized List is converted into DataSet[row] type data supported by Spark SQL.

[0041] Step 320 , based on the order of the acquisition timestamps and according to the various target data, assemble and generate the original DataSet data.

[0042] In one example, part of the data in the original DataSet data obtained in step 320 is shown in Table 1 below.

[0043] Table 1 - Part of the data in the original DataSet

[0044] In Table 1, timestamp represents the acquisition timestamp, and x and y represent the production parameters of the power plant, respectively.

[0045] This specific implementation converts the original data set into the data type supported by Spark SQL to facilitate subsequent value filling and data merging.

[0046] The embodiment of the present application uses the Spark SQL DataSet data structure. DataSet is strongly typed, which means that type errors can be checked at compile time, which helps reduce runtime errors. Technicians can find type mismatch problems in data operations at compile time, which improves the robustness of the code; it stores data in binary format instead of Java objects, avoiding the Java object serialization and deserialization process, which can significantly reduce processing time and resource consumption for processing large-scale production data of power plants, especially when data needs to be transmitted across the network or written to disk.

[0047] Step 400: Determine whether a collection timestamp is missing in the original DataSet data according to the time difference between adjacent collection timestamps in the original DataSet data.

[0048] Specifically, due to equipment failure, network interruption and other reasons, there is a situation where the acquisition timestamps in the acquired original data set are discontinuous. Therefore, before filling in the production parameter values, the embodiment of the present application first determines whether there are missing acquisition timestamps, and when the acquisition timestamps are missing, fills in the missing acquisition timestamps first, and then fills in the production parameter values, thereby ensuring the integrity of the production data.

[0049] In a specific implementation, step 400 includes the following steps: Step 410: If the time difference between every two adjacent acquisition timestamps in the original DataSet data and the numerical relationship between the preset acquisition time step both meet the preset conditions, it is determined that the acquisition timestamp is not missing in the original DataSet data.

[0050] The preset acquisition time step represents the time interval for collecting production parameter values ​​preset by the system in the power plant.

[0051] In this specific implementation, the preset condition is set as the time difference being equal to the preset acquisition time step. That is, if the time difference between each two adjacent acquisition timestamps in the original DataSet data is equal to the preset acquisition time step, it is determined that the acquisition timestamp is not missing in the original DataSet data; if the time difference between two adjacent acquisition timestamps in the original DataSet data is greater than the preset acquisition time step, it is determined that the acquisition timestamp is missing in the original DataSet data.

[0052] For the example of step 300 above, the preset acquisition time step is 1000 (ie, 1 second), which indicates that the original DataSet data lacks an acquisition timestamp.

[0053] Step 500: If it is determined that the original DataSet data lacks the collection timestamp, the original DataSet data is timestamp-filled to obtain the first updated DataSet data, and the first updated DataSet data is value-filled according to the preset data filling rule to obtain the target DataSet data.

[0054] The collected timestamps and the filled new timestamps in the first updated DataSet are arranged in chronological order.

[0055] In a specific implementation, in step 500, timestamp padding is performed on the original DataSet data to obtain the first updated DataSet data, including the following steps: Step 510, for every two adjacent acquisition timestamps in the original DataSet data, if the numerical relationship between the time difference between the two acquisition timestamps and the preset acquisition time step does not meet the preset condition, then determine the number of new timestamps that need to be filled between the two acquisition timestamps according to the time difference between the two acquisition timestamps and the preset acquisition time step, based on the number, determine the new timestamp that needs to be filled according to any one of the two acquisition timestamps and the preset acquisition time step, and generate new row data based on the new timestamp, and insert the new row data into the original DataSet data.

[0056] In this specific implementation, if the time difference between two adjacent acquisition timestamps in the original DataSet data is greater than the preset acquisition time step, the time difference between the two acquisition timestamps is divided by the preset acquisition time step and then subtracted by 1 and rounded to the integer to obtain the number of new timestamps that need to be filled between the two acquisition timestamps. After determining the number of new timestamps that need to be filled, the preset acquisition time step is used as the timestamp growth unit, and the acquisition timestamp that is earlier in the time sequence of the two acquisition timestamps is gradually increased by one timestamp growth unit, or the acquisition timestamp that is later in the time sequence of the two acquisition timestamps is gradually reduced by one timestamp growth unit, and the total number of new timestamps that is the number of new timestamps that need to be filled is obtained.

[0057] It can be understood that in the generated first updated DataSet data, the new row data generated based on the new timestamp only contains the new timestamp data.

[0058] For the example of step 300 above, after step 510 above, it can be determined that the missing acquisition timestamps (i.e., new timestamps) of the original DataSet data include "1681488002000" and "1681488004000". Based on this, part of the data in the column where the acquisition timestamp is located in the first updated DataSet data is determined as shown in Table 2 below.

[0059] Table 2 - Part of the data in the column where the collection timestamp is located in the first updated DataSet data

[0060] In a specific implementation, in step 500, performing value filling on the first update DataSet data according to a preset data filling rule to obtain the target DataSet data includes the following steps: Step 520, for each production parameter in the first updated DataSet data, determine the data whose historical values ​​in the column data corresponding to the production parameter are not empty as the baseline data of the production parameter, based on the baseline data of the production parameter and the column data corresponding to the collection timestamp in the first updated DataSet data, according to the preset data filling rules, fill in the items whose historical values ​​in the column data corresponding to the production parameter are empty with numerical values ​​to obtain the target parameter value data.

[0061] The target parameter value data includes the baseline data of the production parameters and the new values ​​obtained after numerical filling.

[0062] In this specific implementation, the preset data filling rule is forward filling or backward filling. Among them, forward filling means taking the previous valid historical value of the column (i.e. the most adjacent non-null value at the front position) as the new value, and backward filling means taking the next valid historical value of the column (i.e. the most adjacent non-null value at the back position) as the new value. Of course, it can be understood that the preset data filling rule can also be adjusted accordingly according to the specific actual situation.

[0063] Specifically, based on the baseline data of the production parameters and the column data corresponding to the acquisition timestamp in the first updated DataSet data, according to the preset data filling rule, the items with empty historical values ​​in the column data corresponding to the production parameters are numerically filled to obtain the target parameter value data, which can be achieved by the following method: Based on the baseline data of the production parameters and the column data corresponding to the collection timestamp in the first updated DataSet data, according to the preset data filling rules, the lit function is used to fill in the items with empty historical values ​​in the column data corresponding to the production parameters to obtain the target parameter value data.

[0064] In one example, the specific operation of step 520 is: using the range function of Spark SQL, with the start timestamp (i.e., the first acquisition timestamp in the first updated DataSet data), the end timestamp (i.e., the last acquisition timestamp in the first updated DataSet data), the preset acquisition time step, and the number of partitions as parameters, to create a DataSet named timestamp column. The DataSet is as follows Figure 2 As shown; by adding a new constant item withColumn and combining it with the functions.lit(value) method, a DataSet filled with timestamp column data is obtained. The filled DataSet includes the target parameter value data.

[0065] The lit function creates a column containing fixed literal values. Spark's Catalyst optimizer can identify these constant expressions and perform constant folding, directly propagating the constant value to other parts of the expression, reducing the amount of running calculations. The lit function simply wraps a value and does not require any data reading or conversion operations, so the operation cost is extremely low; the lit value is displayed as a static value in the Spark execution plan, and does not need to occupy a large amount of serialization and deserialization resources like dynamically generated data, nor does it need to be transmitted between various Executors, reducing memory usage and network transmission overhead.

[0066] In this specific implementation, for each production parameter in the first updated DataSet data, after step 520, a new DataSet data is generated based on the column data corresponding to the acquisition timestamp and the column data corresponding to the production parameter (i.e., the target parameter value data). Finally, the new DataSet data are merged on the condition that the timestamp column data is equal to obtain the target DataSet data. Specifically, a join operation is performed on each target parameter value data (i.e., the filled DataSet), and multiple timestamp columns are generated after the join operation. The duplicate timestamp columns are removed by the drop method. It can be understood that when the number of new DataSet data is only 1, the new DataSet data is the target DataSet data.

[0067] For the example of step 300 above, when the preset data filling rule is selected as forward filling, part of the data in the obtained target DataSet data is shown in Table 3.

[0068] Table 3 - Part of the data in the target DataSet

[0069] Step 600: If it is determined that the acquisition timestamp is not missing in the original DataSet data, then the original DataSet data is numerically filled according to a preset data filling rule to obtain the target DataSet data.

[0070] In a specific implementation, step 600 includes the following steps: Step 610, for each production parameter in the original DataSet data, if there are items with empty values ​​in the column data corresponding to the production parameter, the data with non-empty historical values ​​in the column data corresponding to the production parameter are determined as the benchmark data of the production parameter, and based on the benchmark data of the production parameter and the column data corresponding to the collection timestamp in the original DataSet data, according to the preset data filling rules, the items with empty historical values ​​in the column data corresponding to the production parameter are filled with values ​​to obtain the target parameter value data.

[0071] The target parameter value data includes the baseline data of the production parameters and the new values ​​obtained after numerical filling.

[0072] The specific implementation method of step 610 in this specific implementation method is the same as the specific implementation method of the above-mentioned step 520, and will not be repeated here.

[0073] In this specific implementation, for each production parameter in which there is an item with an empty value in the corresponding column data in the original DataSet data, after the step 610, a new DataSet data is generated according to the column data corresponding to the acquisition timestamp and the column data corresponding to the production parameter (i.e., the target parameter value data). Finally, the target DataSet data is obtained based on the merging of each new DataSet data with the timestamp column data being equal as a condition. During the merging, if there is a production parameter in which each historical value in the corresponding column data is not empty in the original DataSet data, the column data corresponding to the production parameter is also merged, that is, the obtained target DataSet data includes the column data corresponding to the production parameter. Specifically, the merging method is to perform a join operation on the column data corresponding to each production parameter, and multiple timestamp columns will be generated after the join operation, and the duplicate timestamp columns are removed by the drop method. It can be understood that when the number of new DataSet data is only one and there is no production parameter in which each historical value in the corresponding column data is not empty in the original DataSet data, the new DataSet data is the target DataSet data.

[0074] In the embodiment of the present application, the target DataSet data finally obtained can be written into the MongoDb database through Spark SQL.

[0075] Specifically, Spark supports efficient serialization libraries, such as Kryo, which can reduce the time for data serialization and deserialization. In particular, when using the DataSet API, the serialization process is optimized to speed up data preparation and transmission. By using MongoDb connectors designed specifically for Spark (such as MongoDb Spark Connector), the data interaction between Spark and MongoDb databases is optimized, including data transmission protocols, error handling, and retry logic, which can reduce data movement delays and improve reliability. In addition, before writing data, reasonable partitioning and distribution design of data can ensure that data is evenly written to MongoDb's shard cluster, avoid data skew, and improve overall writing efficiency.

[0076] The existing method (i.e., slicing the data block, dividing it into multiple lists, traversing the list data to fill in the value, and calling the Java API to write into the MongoDb database) is compared with the method for filling the missing production data provided by the embodiment of the present application (hereinafter referred to as the present method): when the total amount of production data of 10 measuring points in a power plant in one day is 860,000, the time required for the existing method to fill the missing production data is about 120 seconds, and the time required for the present method to fill the missing production data is about 30 seconds, which is shortened to 1 / 4 of the original time; when the total amount of production data of 10 measuring points in a power plant in 15 days is 13 million, the time required for the existing method to fill the missing production data is about 2100 seconds, and the time required for the present method to fill the missing production data is about 360 seconds, which is shortened to 1 / 6 of the original time. According to the above-mentioned comparison of measured data, it can be seen that when the amount of data to be processed is larger, the average speed of data processing per second by the present method is faster, and the present method greatly improves the efficiency of filling the missing production data.

[0077] It is not difficult to see that the method for filling missing production data provided in the embodiment of the present application runs on the constructed Spark cluster, and uses the DataSet data structure of Spark SQL to package the data to be filled into a DataSet containing a "timestamp" field. The data is efficiently assembled through join, and Spark SQL is used to quickly write to the MongoDb database, thereby improving the efficiency of filling missing production data.

[0078] Figure 1 FIG. 1 is a flow chart of a method for filling missing production data in one embodiment. It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0079] Figure 2 The structure block diagram of the device for filling missing production data in the embodiment of the present application is schematically shown. Figure 2 As shown, in one embodiment of the present application, a device for filling missing production data is provided, and the device for filling missing production data may include the following functional modules.

[0080] The original data set acquisition module is used to acquire multiple original data sets of the power plant within the target time period, wherein each original data set includes a collection timestamp, and some original data sets include historical values ​​of at least one production parameter, and each historical value in the original data set corresponds to the collection timestamp.

[0081] The DataSet data generation module is used to generate original DataSet data according to each original data set. The original DataSet data includes multiple rows and columns of data, each original data set corresponds to a row of data in the original DataSet data, and the historical values ​​of the same data type in each original data set are located in the same column in the original DataSet data, wherein the data type is a collection timestamp or a production parameter.

[0082] The timestamp missing judgment module is used to determine whether the collection timestamp is missing in the original DataSet data according to the time difference between adjacent collection timestamps in the original DataSet data.

[0083] The first value filling module is used to fill the original DataSet data with the timestamp if it is determined that the collection timestamp is missing in the original DataSet data to obtain the first updated DataSet data, and fill the first updated DataSet data with the values ​​according to the preset data filling rule to obtain the target DataSet data. The collection timestamps in the first updated DataSet data and the filled new timestamps are arranged in chronological order.

[0084] The second value filling module is used to fill in the original DataSet data with values ​​according to a preset data filling rule to obtain the target DataSet data if it is determined that the acquisition timestamp is not missing in the original DataSet data.

[0085] In the embodiment of the present application, the DataSet data generation module may include: The data conversion unit is used to convert each original data set into target data of DataSet[row] type.

[0086] The data assembly unit is used to assemble and generate original DataSet data based on the order of collection timestamps and various target data.

[0087] In an embodiment of the present application, the timestamp missing judgment module may include: The row data missing judgment unit is used to determine that the original DataSet data does not lack the collection timestamp if the time difference between each two adjacent collection timestamps in the original DataSet data and the numerical relationship between the preset collection time step both meet the preset conditions.

[0088] In an embodiment of the present application, the first value filling module may include: The first numerical filling unit is used to determine the number of new timestamps that need to be filled between each two adjacent acquisition timestamps in the original DataSet data according to the time difference between the two acquisition timestamps and the preset acquisition time step if the numerical relationship between the time difference between the two acquisition timestamps and the preset acquisition time step does not meet the preset conditions. Based on the number, the new timestamp that needs to be filled is determined according to any one of the two acquisition timestamps and the preset acquisition time step, and new row data is generated based on the new timestamp, and the new row data is inserted into the original DataSet data.

[0089] In an embodiment of the present application, the first value filling module may further include: The second value filling unit is used to determine, for each production parameter in the first updated DataSet data, data whose historical values ​​in the column data corresponding to the production parameter are not empty as the reference data of the production parameter, and based on the reference data of the production parameter and the column data corresponding to the acquisition timestamp in the first updated DataSet data, perform value filling on the items whose historical values ​​in the column data corresponding to the production parameter are empty according to the preset data filling rule to obtain target parameter value data. The target parameter value data includes the reference data of the production parameter and the new value obtained after the value filling.

[0090] In the embodiment of the present application, the second value filling unit may further include: The numerical filling subunit is used to fill in the items with empty historical values ​​in the column data corresponding to the production parameters through the lit function based on the baseline data of the production parameters and the column data corresponding to the collection timestamp in the first updated DataSet data, according to the preset data filling rules, to obtain the target parameter value data.

[0091] In an embodiment of the present application, the second value filling module may include: The third value filling unit is used for, for each production parameter in the original DataSet data, if there is an item with an empty value in the column data corresponding to the production parameter, then the data with a non-empty historical value in the column data corresponding to the production parameter is determined as the reference data of the production parameter, and based on the reference data of the production parameter and the column data corresponding to the acquisition timestamp in the original DataSet data, according to the preset data filling rule, the items with empty historical values ​​in the column data corresponding to the production parameter are filled with values ​​to obtain target parameter value data. The target parameter value data includes the reference data of the production parameter and the new value obtained after the value filling.

[0092] Since the device for filling missing production data provided in the embodiment of the present application is a virtual device corresponding to the method for filling missing production data in the above-mentioned embodiment, it can also solve the problem that the method for filling missing production data in the prior art has low filling efficiency and difficulty in ensuring data integrity.

[0093] An embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method for filling missing production data described in the above embodiment when executing the computer program.

[0094] The electronic device provided in the embodiment of the present application includes a processor capable of running the method for filling missing production data of the aforementioned embodiment, and can therefore also solve the problem that the method for filling missing production data in the prior art has low filling efficiency and difficulty in ensuring data integrity.

[0095] An embodiment of the present application provides a machine-readable storage medium, on which instructions are stored. When the instructions are executed by a processor, the processor is configured to execute the method for filling missing production data described in the above embodiment.

[0096] The machine-readable storage medium provided in the embodiment of the present application stores instructions for enabling the machine to execute the method for filling missing production data of the above embodiment, and thus can also solve the problem that the method for filling missing production data in the prior art has low filling efficiency and difficulty in ensuring data integrity.

[0097] Figure 3 The internal structure diagram of the computer device of the embodiment of the present application is schematically shown. Figure 3 As shown, in one embodiment of the present application, a computer device is provided, which may be a terminal. The computer device includes a processor A01, a network interface A02, a display screen A04, an input device A05 and a memory (not shown in the figure) connected via a system bus. Among them, the processor A01 of the computer device is used to provide computing and control capabilities. The memory of the computer device includes an internal memory A03 and a non-volatile storage medium A06. The non-volatile storage medium A06 stores an operating system B01 and a computer program B02. The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 in the non-volatile storage medium A06. The network interface A02 of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor A01, a method for filling missing production data is implemented. The display screen A04 of the computer device may be a liquid crystal display or an electronic ink display, and the input device A05 of the computer device may be a touch layer covered on the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0098] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0099] In one embodiment, the production data missing filling device provided in the present application can be implemented in the form of a computer program, and the computer program can be used in Figure 3 The computer device is run on the computer device shown. The memory of the computer device can store various program modules constituting the production data missing filling device, and the computer program composed of various program modules enables the processor to execute the steps of the production data missing filling method of each embodiment of the present application described in this specification.

[0100] Figure 3 The computer device shown can be Figure 2 The original data set acquisition module in the production data missing filling device shown executes step 200, the DataSet data generation module executes step 300, the timestamp missing judgment module executes step 400, the first value filling module executes step 500 and the second value filling module executes step 600.

[0101] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0102] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0103] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0104] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0105] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0106] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0107] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0108] It should also be noted that the term "includes", "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the existence of other identical elements in the process, method, commodity or device including the element.

[0109] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for filling missing production data, characterized in that: The method comprises: Acquire multiple original data sets of the power plant within a target time period; wherein each original data set includes a collection timestamp, and some original data sets include historical values ​​of at least one production parameter, and each historical value in the original data set corresponds to the collection timestamp; Generate original DataSet data according to each original data set; wherein the original DataSet data includes multiple rows and columns of data, each original data set corresponds to a row of data in the original DataSet data, and historical values ​​of the same data type in each original data set are located in the same column in the original DataSet data, wherein the data type is a collection timestamp or a production parameter; Determine whether the collection timestamp is missing in the original DataSet data according to the time difference between adjacent collection timestamps in the original DataSet data; If it is determined that the collection timestamp is missing in the original DataSet data, the timestamp is filled in the original DataSet data to obtain the first updated DataSet data, and the first updated DataSet data is numerically filled according to the preset data filling rule to obtain the target DataSet data; wherein the collection timestamps in the first updated DataSet data and the filled new timestamps are arranged in chronological order; If it is determined that the collection timestamp is not missing in the original DataSet data, the original DataSet data is numerically filled according to the preset data filling rule to obtain the target DataSet data.

2. The method according to claim 1, characterized in that Generate original DataSet data based on each original data set, including: Convert each original data set into target data of DataSet[row] type; Based on the order of the collection timestamps, the original DataSet data is assembled according to each target data.

3. The method according to claim 1, characterized in that Determine whether the original DataSet data is missing a collection timestamp based on the time difference between adjacent collection timestamps in the original DataSet data, including: If the time difference between each two adjacent acquisition timestamps in the original DataSet data and the numerical relationship between the preset acquisition time step both meet the preset conditions, it is determined that there is no missing acquisition timestamp in the original DataSet data.

4. The method according to claim 1, characterized in that: Timestamp padding is performed on the original DataSet data to obtain the first updated DataSet data, including: For each two adjacent acquisition timestamps in the original DataSet data, if the numerical relationship between the time difference between the two acquisition timestamps and the preset acquisition time step does not meet the preset conditions, then the number of new timestamps that need to be filled between the two acquisition timestamps is determined based on the time difference between the two acquisition timestamps and the preset acquisition time step. Based on the number, the new timestamp that needs to be filled is determined according to any one of the two acquisition timestamps and the preset acquisition time step, and new row data is generated based on the new timestamp, and the new row data is inserted into the original DataSet data.

5. The method according to claim 1, characterized in that According to the preset data filling rule, the first updated DataSet data is numerically filled to obtain the target DataSet data, including: For each production parameter in the first updated DataSet data, the data whose historical values ​​are not empty in the column data corresponding to the production parameter are determined as the baseline data of the production parameter. Based on the baseline data of the production parameter and the column data corresponding to the collection timestamp in the first updated DataSet data, according to the preset data filling rules, the items whose historical values ​​are empty in the column data corresponding to the production parameter are filled with values ​​to obtain the target parameter value data; wherein the target parameter value data includes the baseline data of the production parameter and the new value obtained after the numerical filling.

6. The method according to claim 5, characterized in that Based on the benchmark data of the production parameters and the column data corresponding to the acquisition timestamp in the first updated DataSet data, according to the preset data filling rule, the items with empty historical values ​​in the column data corresponding to the production parameters are numerically filled to obtain the target parameter value data, including: Based on the baseline data of the production parameters and the column data corresponding to the collection timestamp in the first updated DataSet data, according to the preset data filling rules, the lit function is used to fill in the items with empty historical values ​​in the column data corresponding to the production parameters to obtain the target parameter value data.

7. The method according to claim 1, characterized in that According to the preset data filling rules, the original DataSet data is numerically filled to obtain the target DataSet data, including: For each production parameter in the original DataSet data, if there are items with empty values ​​in the column data corresponding to the production parameter, the data with non-empty historical values ​​in the column data corresponding to the production parameter are determined as the baseline data of the production parameter, and based on the baseline data of the production parameter and the column data corresponding to the collection timestamp in the original DataSet data, according to the preset data filling rules, the items with empty historical values ​​in the column data corresponding to the production parameter are filled with values ​​to obtain the target parameter value data; wherein, the target parameter value data includes the baseline data of the production parameter and the new value obtained after the value filling.

8. A device for filling missing production data, characterized in that: The device comprises: The original data set acquisition module is used to acquire multiple original data sets of the power plant within a target time period; wherein each original data set includes a collection timestamp, and some original data sets include a historical value of at least one production parameter, and each historical value in the original data set corresponds to the collection timestamp; A DataSet data generation module is used to generate original DataSet data according to each original data set; wherein the original DataSet data includes multiple rows and columns of data, each original data set corresponds to a row of data in the original DataSet data, and historical values ​​of the same data type in each original data set are located in the same column in the original DataSet data, wherein the data type is a collection timestamp or a production parameter; A timestamp missing judgment module is used to determine whether the original DataSet data is missing a collection timestamp based on the time difference between adjacent collection timestamps in the original DataSet data; A first value filling module is used to fill the original DataSet data with the timestamp if it is determined that the collection timestamp is missing in the original DataSet data to obtain the first updated DataSet data, and to fill the first updated DataSet data with the values ​​according to the preset data filling rule to obtain the target DataSet data; wherein the collection timestamps in the first updated DataSet data and the filled new timestamps are arranged in chronological order; The second value filling module is used to fill in the original DataSet data with values ​​according to a preset data filling rule to obtain the target DataSet data if it is determined that the acquisition timestamp is not missing in the original DataSet data.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for filling missing production data according to any one of claims 1 to 7 is implemented.

10. A machine-readable storage medium having instructions stored thereon, characterized in that: When the instruction is executed by a processor, the processor is configured to execute the method for filling production data gaps according to any one of claims 1 to 7.

Images