Data processing method and device, electronic equipment and storage medium

Abstract

The invention provides a data processing method and device, electronic equipment and a storage medium, and the data processing method comprises the steps: sending a query instruction to equipment to be monitored, so as to enable the equipment to be monitored to return first target data to a monitoring platform according to the query instruction. And according to the first target data, locking a calibration date with data exception, and according to the calibration date, executing a data calibration process to realize targeted repair of the corresponding data. The data calibration process comprises the following steps: acquiring second target data of a date corresponding to a calibration date in a monitoring platform and first historical data of the monitoring platform; and performing statistical processing on data of a date corresponding to the calibration date in the first target data to obtain third target data, and updating the second target data into the third target data. And when the second target data is updated to the third target data, performing data synchronization processing on the first historical data of the monitoring platform to obtain the second historical data, so that the integrity, accuracy and consistency of the data of the monitoring platform are improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, apparatus, electronic device and storage medium. Background Technology

[0002] With the widespread application of distributed energy systems, photovoltaic power plants, wind power plants, and other equipment generally interact with remote monitoring platforms through gateways to achieve real-time collection and centralized management of key parameters such as power generation, operating status, and fault information. Historical data can be used to evaluate system performance, specify operation and maintenance strategies, calculate revenue, and serve as a core basis for meeting regulatory requirements.

[0003] In actual operation, data upload typically relies on a two-tier communication architecture. The first tier is local communication between the device and the gateway, such as RS485 or Modbus protocols. The second tier is remote communication between the gateway and the platform server, such as 4G, Wi-Fi, or Ethernet. Mainstream monitoring platforms usually have fault-tolerance mechanisms for interruptions in second-tier communication, such as offline caching and retransmission after network recovery to achieve data retransmission. However, they lack the ability to update and repair related data in case of problems with first-tier communication. Summary of the Invention

[0004] This application aims to provide a data processing method, apparatus, electronic device, and storage medium to improve the ability to repair problematic data.

[0005] In a first aspect, this application proposes a data processing method applied to an energy monitoring system. The energy monitoring system includes a device to be monitored, a communication gateway, and a monitoring platform. The monitoring platform communicates with the device to be monitored through the communication gateway. The method includes: sending a query command to the device to be monitored, so that the device to be monitored returns first target data to the monitoring platform according to the query command; determining a calibration date based on the first target data, and executing a data calibration process based on the calibration date; wherein the data calibration process includes: acquiring second target data corresponding to the calibration date in the monitoring platform and first historical data of the monitoring platform; performing statistical processing on the data corresponding to the calibration date in the first target data to obtain third target data, and updating the second target data to the third target data; when the second target data is updated to the third target data, performing data synchronization processing on the first historical data of the monitoring platform to obtain second historical data.

[0006] In some embodiments, sending a query instruction to the device to be monitored, so that the device to be monitored returns first target data to the monitoring platform according to the query instruction, includes: the monitoring platform generating a query instruction including a target device identifier and a target time range, and sending it to the communication gateway; the communication gateway determining the corresponding target device to be monitored according to the target device identifier in the query instruction, and sending a data acquisition request to the target device to be monitored, the data acquisition request including the target time range; the communication gateway receiving the first target data returned by the target device to be monitored, which corresponds to the target time range, and forwarding the first target data to the monitoring platform.

[0007] In some embodiments, the step of performing statistical processing on the data in the first target data corresponding to the calibration date to obtain the third target data includes: querying and obtaining the data to be processed corresponding to the calibration date from the first target data according to the calibration date; generating a calibration task based on the calibration date, the calibration task including at least the data to be processed and the calibration date; calling a preset data processing service to execute the calibration task, recalculating and statistically analyzing the data to be processed, and generating the third target data.

[0008] In some embodiments, when the second target data is updated to the third target data, the first historical data of the monitoring platform is synchronized to obtain the second historical data, which includes: determining at least one statistical time period containing the calibration date based on the calibration date; for each statistical time period, recalculating the statistical data corresponding to the statistical time period based on the updated second target data and other data in the first historical data belonging to the statistical time period; updating the recalculated statistical data of each statistical time period to the database record corresponding to the monitoring platform to overwrite the original statistical data, wherein the updated statistical data constitutes the second historical data.

[0009] In some embodiments, before updating the second target data to the third target data, the data processing method further includes: performing permission verification on the current user to determine whether the user has permission to perform data updates; if the permission verification passes, displaying a confirmation prompt message, the confirmation prompt message being used to request whether to update the second target data corresponding to the calibration date to the third target data; receiving a confirmation instruction for the confirmation prompt message, and executing the step of updating the second target data to the third target data; if the permission verification fails, terminating the data calibration process.

[0010] In some embodiments, the data processing method further includes performing anomaly monitoring on the second historical data based on a preset anomaly data monitoring model to obtain monitoring results; and automatically initiating the data calibration process when the monitoring results meet the condition of being lower than a preset data range.

[0011] Secondly, this application also proposes a data processing apparatus, comprising: a first target data acquisition module, configured to send a query instruction to a device to be monitored, so that the device to be monitored returns first target data to a monitoring platform according to the query instruction; and a data calibration module, configured to determine a calibration date based on the first target data, and execute a data calibration process based on the calibration date; wherein the data calibration process includes: acquiring second target data corresponding to the calibration date in the monitoring platform and first historical data of the monitoring platform; performing statistical processing on the data corresponding to the calibration date in the first target data to obtain third target data, and updating the second target data to the third target data; when the second target data is updated to the third target data, performing data synchronization processing on the first historical data of the monitoring platform to obtain second historical data.

[0012] Thirdly, this application also proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the data processing method described in any of the first aspects.

[0013] Fourthly, this application also proposes a non-volatile computer-readable storage medium storing computer-executable instructions that, when executed by an electronic device, cause the electronic device to perform the data processing method described in any of the first aspects.

[0014] Fifthly, this application also proposes a computer program product comprising a computer program stored on a non-volatile computer-readable storage medium, the computer program comprising program instructions that, when executed by an electronic device, cause the electronic device to perform the data processing method described in any of the first aspects.

[0015] Unlike existing technologies, this application provides a data processing method. This method includes: sending a query command to the device under monitoring, causing the device to return first target data to the monitoring platform based on the query command. The monitoring platform directly obtains the first target data from the device under monitoring, reducing intermediate transmission links and ensuring the reliability and authenticity of the data source. Based on the first target data, the calibration date with data anomalies is accurately located, and a data calibration process is executed according to the calibration date to achieve targeted repair of the data corresponding to the calibration date. The data calibration process includes: obtaining second target data corresponding to the calibration date from the monitoring platform and first historical data from the monitoring platform; performing statistical processing on the data corresponding to the calibration date in the first target data to obtain third target data, and updating the second target data to the third target data; when the second target data is updated to the third target data, performing data synchronization processing on the first historical data from the monitoring platform to obtain second historical data. In the data calibration process, not only is the second target data corresponding to the calibration date updated and replaced, but the first historical data is also recalculated simultaneously to generate the second historical data. This effectively solves the problem of delayed updates to cumulative data, thereby fully ensuring the integrity, accuracy and consistency of the monitoring platform data and improving the energy monitoring system's ability to maintain historical data.

[0016] Additional aspects and advantages of the embodiments of this application will be described or shown in part in the following description, or illustrated by practice of the embodiments of this application. Attached Figure Description

[0017] One or more embodiments are illustrated by way of example with reference to the accompanying drawings, which are not intended to limit the embodiments, and elements having the same reference numerals in the drawings are designated as similar elements.

[0018] Figure 1 This is a schematic diagram of an energy monitoring system according to some embodiments of this application; Figure 2 This is a flowchart illustrating the data processing method of some embodiments of this application; Figure 3 These are schematic diagrams illustrating application scenarios for some embodiments of this application; Figure 4 These are schematic diagrams illustrating application scenarios for some embodiments of this application; Figure 5 These are schematic diagrams illustrating application scenarios for some embodiments of this application; Figure 6 These are schematic diagrams illustrating application scenarios for some embodiments of this application; Figure 7 This is a schematic diagram of the structure of a data processing apparatus according to some embodiments of this application; Figure 8This is a schematic diagram of the structure of an electronic device according to some embodiments of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of this application, but not all embodiments.

[0020] In this application, the reference to "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments.

[0021] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0022] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0023] The technical features involved in the different embodiments of this application described below can be combined with each other as long as they do not conflict with each other.

[0024] With the widespread application of distributed energy systems, photovoltaic power plants, wind power plants, and other equipment generally interact with remote monitoring platforms through communication gateways to achieve real-time collection and centralized management of key parameters such as historical data, operating status, and fault information. Historical data (e.g., power generation, current, power, temperature, etc.) serves as the core basis for evaluating system performance, formulating operation and maintenance strategies, calculating revenue, and meeting regulatory requirements; its completeness and accuracy are of paramount importance.

[0025] However, in actual operation, the existing energy monitoring platform still has some problems with the maintenance methods of historical data. For example, (1) it lacks the ability to repair data loss caused by local communication interruption. When communication between the communication gateway and the inverter is interrupted due to loose wiring, abnormal protocol or equipment failure, even if the equipment itself is still running normally and recording local historical data, the communication gateway cannot obtain this historical data, and the monitoring platform will display "zero value" or "missing". Since this type of interruption does not involve the connection between the monitoring platform and the communication gateway, the traditional "breakpoint resume" mechanism cannot detect and handle it. (2) There is no precise manual intervention to calibrate historical data. Most existing monitoring platforms do not allow users to manually modify historical data for certain dates, or only provide coarse-grained data synchronization functions, and cannot support selectively synchronizing local historical data to the monitoring platform on a daily level to achieve local data repair. (3) The data aggregation logic is not updated in conjunction with the data. For updated historical data, the existing monitoring platform often does not automatically trigger the recalculation of monthly, annual and total cumulative data, which has a data update lag.

[0026] The data processing method provided in this application can be applied to energy monitoring systems. Please refer to... Figure 1 The energy monitoring system mainly includes monitored devices 11, a communication gateway 12, and a monitoring platform 13. The monitored devices 11 can specifically refer to various devices deployed in energy production or transmission scenarios, such as generator sets, photovoltaic inverters, energy storage converters, or power metering devices. The communication gateway 12, as a key communication hub in the system, establishes communication connections with both the monitored devices 11 and the monitoring platform 13. The monitoring platform 13 can be a standalone server or a server cluster consisting of multiple servers, responsible for centralized processing and analysis of energy data.

[0027] The device to be monitored 11 connects to the communication gateway 12 via its built-in communication module, is controlled by the communication gateway 12, and interacts with it for data. The device to be monitored 11 can connect to the communication gateway 12 via a local area network, such as Zigbee, Bluetooth, or industrial bus. The communication gateway 12 then establishes a stable network connection with the remote monitoring platform 13 via a wide area network path, such as fiber optic, 4G / 5G, or Wi-Fi network, thereby constructing a data transmission channel from the field device to the cloud platform. Users or systems can send query or control commands to the device to be monitored 11 via the communication gateway 12 through the monitoring platform 13, and receive the operational data returned by it.

[0028] Specifically, the monitoring platform 13 sends a data query command to the designated monitored device 11 through the communication gateway 12. Upon receiving the command, the monitored device 11 returns its recorded first target data to the monitoring platform 13. The monitoring platform 13 determines a calibration date for data verification and correction based on the first target data and initiates the corresponding data calibration process. During this process, the energy monitoring system simultaneously acquires the second target data stored on the monitoring platform 13 at the calibration date, as well as the first historical data in the monitoring platform 13. Next, the energy monitoring system performs statistical and integrated processing on the first target data obtained from the monitored device 11 corresponding to the calibration date, generating third target data, and replacing the original second target data in the monitoring platform 13 with this third target data. After the target data in the monitoring platform 13 is updated, the energy monitoring system further synchronizes and statistically adjusts the relevant first historical data, thereby generating an internally consistent and calibrated second historical dataset to ensure the accuracy and continuity of the data records across the entire monitoring platform 13.

[0029] Specifically, the embodiments of this application will be further described below with reference to the accompanying drawings.

[0030] Firstly, embodiments of this application provide a data processing method, please refer to... Figure 2 The data processing method specifically includes the following steps: Step S1: Send a query command to the device to be monitored 11, so that the device to be monitored 11 returns the first target data to the monitoring platform 13 according to the query command.

[0031] The device to be monitored 11 refers to the terminal equipment monitored in the energy monitoring system. For example, an inverter. It has data acquisition, local storage, and communication functions.

[0032] Communication gateway 12 is a communication relay device deployed on the device to be monitored 11. It is used to convert between the local communication protocol (e.g., RS485, Modbus, etc.) of the device to be monitored 11 and the network protocol (e.g., TCP, MQTT) of the monitoring platform 13, and is the communication channel connecting the device to be monitored 11 and the monitoring platform 13.

[0033] The monitoring platform 13 is a software system deployed on a cloud server, used to provide a human-computer interaction interface, receive, store, process and display data.

[0034] The monitoring platform 13 sends a query command to the device under monitoring 11. The device under monitoring 11 retrieves the first target data corresponding to the query command from its local storage and feeds it back to the monitoring platform 13 via the communication gateway 12. The first target data includes the operating parameter data of the device under monitoring 11 within a preset time period, such as power generation, current, voltage, power, and temperature. By directly sending a query command to the device under monitoring 11 to obtain the first target data, redundant intermediate transmission links can be skipped, reducing distortion or lag issues that may occur during the transmission of the first target data, ensuring that the monitoring platform 13 obtains accurate energy operation data from the device under monitoring 11.

[0035] In some embodiments, sending a query instruction to the device to be monitored 11, so that the device to be monitored 11 returns first target data to the monitoring platform 13 according to the query instruction, includes: the monitoring platform 13 generating a query instruction including a target device identifier and a target time range, and sending it to the communication gateway 12. The communication gateway 12 determines the corresponding target device to be monitored 11 according to the target device identifier in the query instruction, and sends a data acquisition request to the target device to be monitored 11, the data acquisition request including the target time range. The communication gateway 12 receives the first target data returned by the target device to be monitored 11 corresponding to the target time range, and forwards the first target data to the monitoring platform 13.

[0036] Step S2: Determine the calibration date based on the first target data, and execute the data calibration process according to the calibration date; wherein, the data calibration process includes: Acquire the second target data corresponding to the calibration date and the first historical data of the monitoring platform 13 from the monitoring platform 13.

[0037] Statistical processing is performed on the data corresponding to the calibration date in the first target data to obtain the third target data, and the second target data is updated to the third target data.

[0038] When the second target data is updated to the third target data, the first historical data of the monitoring platform 13 is synchronized to obtain the second historical data.

[0039] The primary target data consists of the raw operating data recorded by the monitored device 11 in its local storage unit. This raw operating data is collected in real time by the sensors of the monitored device 11 itself and stored according to a preset time dimension (such as hourly or daily). Since the monitored device 11 is directly associated with the energy production process, its locally stored primary target data can accurately reflect the actual operating status of the device within a specific time period. Even during the first-level communication interruption with the communication gateway 12, this data will continue to be recorded, providing the most direct and accurate data source for subsequent data calibration.

[0040] The calibration date refers to the specific date on which data calibration needs to be performed in the monitoring platform 13. When the monitoring platform 13 receives the first target data returned by the monitored device 11, it compares and analyzes it with the historical data for the corresponding date stored in the monitoring platform 13 itself. For example, if the power generation data for a certain day shown in the first target data differs from the second target data recorded in the monitoring platform 13 for that day (e.g., the data in the monitoring platform 13 is zero or significantly deviates from a reasonable range, or needs adjustment), then that date is determined as the calibration date so that the data for that date can be accurately corrected subsequently. It is understood that there can be one or more calibration dates.

[0041] By determining the calibration date based on the first target data and carrying out a targeted data calibration process, the time node data that needs to be corrected can be accurately located, reducing the efficiency loss caused by indiscriminate calibration of all data. At the same time, by combining the calibration date with the corresponding second target data and first historical data in the monitoring platform 13, the object of data calibration is more clearly defined and the scope is clearer.

[0042] The second target data is the historical operating data corresponding to the calibration date stored in the database of the monitoring platform 13 before the data calibration process is started. This data may be missing or inaccurate due to previous local communication interruptions or other issues, such as being displayed as "zero value", "missing", or abnormal values ​​that do not match the actual operating conditions of the monitored device 11. This data is the object that needs to be checked and replaced in this data calibration process.

[0043] The first historical data consists of cumulative data stored in monitoring platform 13 that is associated with the calibration date. This includes data such as the monthly cumulative power generation for the month of the calibration date, the annual cumulative power generation for the year, and the total cumulative power generation since the equipment was put into operation. This data is typically calculated based on daily target data aggregation. During data calibration, when the second target data is updated to the accurate third target data, the first historical data also needs to be adjusted accordingly to obtain the second historical data. This ensures the consistency between the historical energy data of monitoring platform 13 and the calibrated data, reducing the occurrence of contradictions between old and new data and data chain breaks. It provides a consistent and accurate data basis for subsequent work such as energy consumption trend analysis, equipment energy efficiency assessment, and energy cost accounting.

[0044] This application provides a data processing method, which includes: sending a query command to a device 11 to be monitored, so that the device 11 returns first target data to a monitoring platform 13 according to the query command. The monitoring platform 13 directly obtains the first target data from the device 11 to be monitored, reducing intermediate transmission links and ensuring the reliability and authenticity of the data source. Based on the first target data, the calibration date with data anomalies is accurately located, and a data calibration process is executed according to the calibration date to achieve targeted repair of the data corresponding to the calibration date. The data calibration process includes: obtaining second target data corresponding to the calibration date and first historical data of the monitoring platform 13. Statistical processing is performed on the data corresponding to the calibration date in the first target data to obtain third target data, and the second target data is updated to the third target data. When the second target data is updated to the third target data, the first historical data of the monitoring platform 13 is synchronized to obtain second historical data. In the data calibration process, not only is the second target data corresponding to the calibration date updated and replaced, but the first historical data is also recalculated simultaneously to generate the second historical data. This effectively solves the problem of delayed updates to cumulative data, thereby fully ensuring the integrity, accuracy and consistency of the data on the monitoring platform and improving the energy monitoring system's ability to maintain historical data.

[0045] In some embodiments, statistical processing is performed on the data corresponding to the calibration date in the first target data to obtain the third target data. This includes: querying and retrieving the data to be processed corresponding to the calibration date from the first target data based on the calibration date. The data to be processed specifically refers to the raw operating parameter data recorded by the monitored device 11 within the calibration date at a preset acquisition frequency (e.g., once per hour). For example, the raw operating parameter data may be the hourly power generation, peak current for each time period, voltage fluctuation data, power curve data, and equipment temperature change data within the calibration date. When performing statistical processing on this data to be processed, the data validity is first verified, and outliers that clearly exceed the reasonable range (such as extreme values ​​caused by instantaneous sensor failure) and duplicate records are removed to ensure that the data basis for the statistics is true and reliable.

[0046] Subsequently, according to the storage format requirements of the monitoring platform 13 for daily target data, aggregation calculations are performed on the verified data to be processed to obtain third target data. This third target data can accurately reflect the actual operating status of the monitored device 11 on the calibration date and can be directly used to replace the original second target data in the monitoring platform 13. A calibration task is generated based on the calibration date, which includes at least the data to be processed and the calibration date. The preset data processing service is called to execute the calibration task, recalculate and statistically analyze the data to be processed, and generate the third target data. For example, the preset data processing service can be a data analysis script written in Python, with built-in algorithm modules such as data cleaning, outlier detection, and time series aggregation. When the calibration task is triggered, the system will pass the data to be processed and the calibration date as input parameters to the service. First, the original data for the corresponding time period is filtered according to the calibration date. Then, outliers are identified and removed using the IQR (interquartile range) method. Then, the data is summed, averaged, and extreme value extracted according to the time dimension (such as hourly or daily) defined by the monitoring platform 13. Finally, the third target data that conforms to the storage format of the monitoring platform 13 is output. This modular processing method not only improves the efficiency and accuracy of data statistics, but also allows for flexible adaptation to the data processing needs of different types of monitored devices 11 by updating preset algorithm modules.

[0047] In some embodiments, when the second target data is updated to the third target data, the first historical data of the monitoring platform 13 is synchronized to obtain the second historical data, including: determining at least one statistical time period containing the calibration date based on the calibration date. The statistical time period can be preset according to the data statistical dimensions of the monitoring platform 13, such as daily, weekly, monthly, quarterly, and annual periods. For example, if the calibration date is October 15, 2023, the statistical time period containing the calibration date includes the daily period (i.e., October 15, 2023), the weekly period (October 9 to October 15, 2023), the monthly period (October 2023), the quarterly period (the fourth quarter of 2023, i.e., October, November, and December), and the annual period (2023).

[0048] For each statistical time period, the statistical data corresponding to that statistical time period is recalculated based on the updated second target data and other data belonging to that statistical time period in the first historical data. For example, if the statistical time period is a monthly period (October 2023), when recalculating the monthly cumulative power generation for that month, the updated third target data for October 15, 2023, will be added to the daily power generation data already stored in the monitoring platform 13 for other dates in that month (such as October 1 to 14, October 16 to 31) to obtain the corrected monthly cumulative power generation for October 2023. This corrected data will then replace the original monthly cumulative power generation for October 2023 in the first historical data to generate the second historical data.

[0049] Similarly, for the annual cycle (2023), the corrected cumulative power generation for October 2023 will be re-added with the cumulative power generation for other months in 2023 (January to September, November to December) to generate a new annual cumulative power generation for 2023, i.e., the second historical data. Through this method, it is ensured that all cumulative data at all levels related to the calibration date in monitoring platform 13 can be accurately corrected, forming logically consistent and coherent second historical data.

[0050] In some embodiments, before updating the second target data to the third target data, the data processing method further includes: performing permission verification on the current operating user to determine whether the user has permission to perform the data update. If the permission verification passes, a confirmation prompt is displayed, which requests whether to update the second target data corresponding to the calibration date to the third target data. A confirmation instruction for the confirmation prompt is received, and the step of updating the second target data to the third target data is executed. If the permission verification fails, the data calibration process is terminated.

[0051] Permission verification ensures that only administrators or authorized personnel with specific operating permissions can perform data update operations, preventing unauthorized users from maliciously tampering with or misoperating the data on the monitoring platform 13, and guaranteeing data security and integrity. Specifically, when a user initiates a data update request, the monitoring platform 13 automatically obtains the current user's identity information (such as username, user ID, etc.) and compares this information with a preset permission configuration table. The permission configuration table records the operating permission range corresponding to different user roles. For example, the "system administrator" role has full permissions for data updates, while the "ordinary operator" role only has data viewing permissions and no update permissions. If the comparison result shows that the current user's role permissions include data update permissions, the permission verification passes, allowing the user to continue updating the second target data to the third target data. If the permission verification fails, the monitoring platform 13 rejects the update request and sends a "Insufficient permissions, unable to perform data update" message to the user, terminating the data calibration process. Simultaneously, this permission verification failure event is recorded in the system log for subsequent auditing and traceability.

[0052] In some embodiments, the data processing method further includes performing anomaly monitoring on the second historical data based on a preset anomaly data monitoring model to obtain monitoring results. When the monitoring results meet the condition of being below a preset data range, a data calibration process is automatically initiated.

[0053] The abnormal data monitoring model can be built using machine learning algorithms. For example, a data model trained on normal data samples from the first historical data set can be input into the second historical data set, and the model will output an anomaly probability value based on the data characteristics. If the anomaly probability value exceeds a preset threshold (e.g., 95%), the data is determined to be abnormal, triggering an early warning mechanism and sending an alert notification to the system administrator containing the specific timestamp, value, and possible cause analysis of the abnormal data. Simultaneously, the abnormal data is marked as "pending verification," and a notification is displayed in the historical data management interface of the monitoring platform 13, facilitating manual verification by the administrator. If manual verification confirms that the abnormal data does indeed contain calibration errors, the energy monitoring system can support re-triggering the calibration process for the corresponding date of the abnormal data. The newly acquired first target data is used to re-perform the calibration of the second target data and synchronize it with the second historical data until the abnormal data is corrected, further improving the reliability of the second historical data and ensuring the accuracy of the energy operation data stored on the monitoring platform 13 throughout its entire lifecycle.

[0054] In some embodiments, a data snapshot is generated for each calibration operation in the calibration process. The data snapshot includes the second target data and the first historical data before the calibration operation, and the third target data and the second historical data after calibration. It also records the execution time of the calibration operation, the operator's identification, the calibration date, and the calibration basis (such as the corresponding first target data source file number). These snapshot data are archived and stored in chronological order, forming a complete calibration operation history chain. When a historical version rollback is required, the user can select a specified snapshot version through the historical version management function of the monitoring platform 13. The system will automatically restore the current data state of the monitoring platform 13 to the historical state corresponding to that snapshot based on the pre-calibration data recorded in the snapshot. This effectively reduces the risk of data corruption due to calibration errors, provides a safe and reliable fault-tolerance mechanism for data calibration operations, and ensures the flexibility and security of data management on the monitoring platform 13.

[0055] In some embodiments, an approval mechanism is introduced before the data calibration process to ensure the standardization and traceability of the data calibration operation. The specific process includes: after a regular user discovers a data anomaly on the monitoring platform 13 and completes the acquisition of the first target data and a preliminary determination of the calibration date, they can submit a calibration application through the system. The application must include the calibration date, the number of the monitored device 11, a description of the source of the first target data, and a preliminary calibration plan. After logging into the monitoring platform 13, the senior administrator will see the calibration application in the "Pending Approval Tasks" list. The administrator clicks the "Agree" button on the approval interface, and the system automatically triggers the data calibration process. The entire approval process (including submission time, approver, approval comments, approval result, execution time, etc.) is recorded in detail in the approval log and associated with the corresponding data snapshot, forming a complete closed-loop management system, further strengthening the compliance and accountability of the data calibration operation.

[0056] In some embodiments, a visual comparison interface is provided, allowing users to intuitively view the numerical differences between the second and third target data corresponding to the calibration date, as well as the first and second historical data, on the monitoring platform 13. This interface uses a timeline as a reference, displaying data from the monitoring platform on the left and data from the monitored device 11 on the right, differentiated by different colors. This helps users quickly understand the impact range and correction effect of data calibration, improving the transparency and credibility of the data calibration results and facilitating subsequent manual review and verification of the rationality of the calibration operation.

[0057] In some embodiments, data synchronization and recalculation tasks are placed in a message queue (such as RabbitMQ / Kafka) to avoid the data calibration process occupying the system's main thread resources for a long time through asynchronous processing. This ensures that the monitoring platform 13 can still respond normally to other user queries and operation requests while performing data calibration operations, thereby improving the system's concurrent processing capabilities and operational stability.

[0058] In some embodiments, before the first historical data of the monitoring platform 13 is processed for data synchronization to obtain the second historical data, the data processing method further includes introducing an asynchronous task processing mechanism to enhance the stability and responsiveness of the system in high-concurrency scenarios.

[0059] Specifically, after determining that data synchronization processing is required, monitoring platform 13 does not directly perform complex recalculation in the main business thread. Instead, it creates a structured recalculation task message. This message contains all necessary information, such as the unique identifier of the target device, the calibration date, the updated third target data value, and a list of statistical time periods that need to be recalculated. Subsequently, monitoring platform 13 publishes this recalculation task message to the designated task queue of its internally integrated message middleware (such as RabbitMQ or Kafka). After the publication operation is completed, the main business thread of monitoring platform 13 can immediately return, signifying that the data calibration process is complete for the user, thus ensuring a smooth user experience.

[0060] Meanwhile, one or more independent recalculation service processes deployed in the system act as consumers of the message middleware, continuously listening to the task queue. When a recalculation service process retrieves a recalculation task message from the queue, it parses the message content and, based on the parameters, retrieves relevant data from the database, independently executing the recalculation logic required for data synchronization. After all calculations are completed, the recalculation service process updates the database of the monitoring platform 13 with the final obtained second historical data. Through the asynchronous decoupling design of the message queue described above, computationally intensive and time-consuming data recalculation work is transferred to the background service for asynchronous execution, reducing its blocking of the user's core operation process and significantly improving the overall throughput and service reliability of the monitoring platform 13.

[0061] In some embodiments, the data processing method also supports integration with external systems, encapsulating the core capabilities of the data calibration process into services. Specifically, the monitoring platform 13 provides and exposes a standard RESTful API interface. This interface defines a standardized request endpoint. When a third-party operation and maintenance system needs to initiate data calibration, it does not need to log in to the front-end interface of the monitoring platform 13, but can directly send a request to the request endpoint to trigger the data calibration process. This request must include key parameters such as the unique identifier of the device to be calibrated, the calibration date, the first target data, and the caller's authentication information. Through the above service encapsulation and API exposure, seamless integration with external systems (such as enterprise-level operation and maintenance management platforms, energy dispatching systems, etc.) is achieved, enabling the data calibration capability to be reused in a wider range of business scenarios and improving the data collaborative processing efficiency of the entire energy monitoring system.

[0062] In some practical application scenarios: A distributed photovoltaic power station uploads real-time power generation data from dozens of photovoltaic inverters to a remote energy monitoring platform 13 via a communication gateway 12 deployed on-site. The power station's maintenance personnel need to manually adjust the daily power generation data of inverter 01 (hereinafter referred to as inverter 01) on December 16, 2025. To solve this problem, an engineer with advanced maintenance privileges logged into the monitoring platform 13 and performed the following calibration procedure: Please refer to... Figure 3 On the historical data page of inverter 01, the engineer found the manual calibration entry and clicked it to trigger the calibration process. The monitoring platform 13 automatically sent a query command to inverter 01. This command was sent through the communication gateway 12 and directly read the daily historical power generation logs for December (i.e., the first target data) stored in the internal memory of inverter 01.

[0063] Please refer to Figure 3 and Figure 4 The monitoring platform 13 displays the acquired local device data (the first target data) to the engineer. Through comparison, the engineer finds that the monitoring platform 13 recorded 12.16 kWh on December 16th (the second target data), while the device's local record shows 36.7 kWh. Based on this, the calibration date is determined to be December 16, 2025. Combined with... Figure 5 and Figure 6After confirmation by the engineer, monitoring platform 13 processed the local 36.7kWh data of the device (to obtain the third target data) and updated the record of December 16th in monitoring platform 13 from 12.16kWh to 36.7kWh. Immediately afterwards, the energy monitoring system automatically triggered data synchronization processing. Based on the updated daily data, it recalculated the first historical data for inverter 01 in December 2025, including the monthly total power generation, annual cumulative power generation, and total power generation of the power station, and refreshed all updated second historical data to the database and statistical reports. Throughout the calibration operation, the system background automatically recorded a complete operation log, including the operator, time, modified device (inverter 01), modification date (December 16th), value before modification (12.16 kWh), value after modification (36.7kWh), and changes in related statistical values.

[0064] In addition, before the engineer clicks the "Update" button, the system displays a clear secondary confirmation dialog box, prompting, "This operation will permanently modify historical power generation data and affect monthly statistics and settlements. Please confirm." This key operation prompt effectively reduces the possibility of operational errors.

[0065] Through the above calibration process, the photovoltaic power station successfully repaired the historical data gaps caused by the local communication interruption, ensuring the integrity and accuracy of the platform data and providing a reliable data foundation for subsequent power generation performance analysis, electricity bill settlement, and government subsidy application.

[0066] Secondly, this application provides a data processing apparatus 100, please refer to... Figure 7 The data processing device 100 includes a first target data acquisition module 101 and a data calibration module 102.

[0067] The first target data acquisition module 101 is used to send a query command to the device to be monitored 11, so that the device to be monitored 11 returns the first target data to the monitoring platform 13 according to the query command.

[0068] The data calibration module 102 is used to determine the calibration date based on the first target data and to execute the data calibration process based on the calibration date. The data calibration process includes: acquiring the second target data corresponding to the calibration date and the first historical data of the monitoring platform 13 from the monitoring platform 13; performing statistical processing on the data corresponding to the calibration date in the first target data to obtain the third target data, and updating the second target data to the third target data; when the second target data is updated to the third target data, performing data synchronization processing on the first historical data of the monitoring platform 13 to obtain the second historical data.

[0069] It should be noted that the data processing apparatus 100 described above can execute the data processing method provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in the embodiments of the data processing apparatus 100 can be found in the data processing method provided in the embodiments of this application. Thirdly, this application also provides an electronic device, please refer to... Figure 8 The electronic device 200 includes: at least one processor 201; and a memory 202 communicatively connected to said at least one processor 201. Figure 8 The example uses a processor 201. The memory 202 stores instructions executable by the at least one processor 201, which, when executed by the at least one processor 201, enable the at least one processor 201 to perform the aforementioned data processing method. The processor 201 and the memory 202 can be connected via a bus or other means. Figure 8 Taking the example of a connection between China and Israel via a bus.

[0070] The memory 202, as a readable storage medium, can be used to store software programs, executable programs, and modules, such as the program instructions / modules corresponding to the data processing method in the embodiments of this application, for example, the appendix. Figure 7 The various modules are shown. The processor 201 executes various server functions and data processing by running software programs, instructions, and modules stored in the memory 202, that is, it implements the data processing method in the above embodiments. The memory 202 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the data processing device 100, etc. The memory 202 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 202 may optionally include memory remotely located relative to the processor 201, and these remote memories may be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0071] The one or more modules are stored in the memory 202, and when executed by the one or more processors 201, they perform the data processing methods described in any of the above method embodiments. For example, the above-described... Figure 2 The method and steps to achieve Figure 7 The functions of each module in the program.

[0072] Fourthly, embodiments of this application also provide a non-volatile computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are executed by an electronic device, the data processing method described above is implemented. For example, the above-described... Figure 2 The method and steps to achieve Figure 7 The functions of each module in the program.

[0073] In some embodiments, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the steps in the above-described method embodiments.

[0074] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software and a general-purpose hardware platform, or of course, using hardware. Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of this application as described above, which are not provided in detail for the sake of brevity; although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method applied to an energy monitoring system, the energy monitoring system comprising a device to be monitored, a communication gateway, and a monitoring platform, wherein the monitoring platform communicates with the device to be monitored through the communication gateway, characterized in that, include: A query command is sent to the device to be monitored, so that the device to be monitored returns first target data to the monitoring platform according to the query command; Based on the first target data, a calibration date is determined, and a data calibration process is executed according to the calibration date; wherein, the data calibration process includes: Obtain the second target data corresponding to the calibration date from the monitoring platform and the first historical data from the monitoring platform; Statistical processing is performed on the data in the first target data that corresponds to the calibration date to obtain the third target data, and the second target data is updated to the third target data; When the second target data is updated to the third target data, the first historical data of the monitoring platform is synchronized to obtain the second historical data.

2. The data processing method according to claim 1, characterized in that, Sending a query command to the device under monitoring, so that the device under monitoring returns first target data to the monitoring platform according to the query command, includes: The monitoring platform generates a query command including the target device identifier and the target time range, and sends it to the communication gateway; The communication gateway determines the corresponding target device to be monitored based on the target device identifier in the query instruction, and sends a data acquisition request to the target device to be monitored, wherein the data acquisition request includes the target time range; The communication gateway receives the first target data returned by the target device to be monitored, which corresponds to the target time range, and forwards the first target data to the monitoring platform.

3. The data processing method according to claim 1, characterized in that, The step of statistically processing the data in the first target data corresponding to the calibration date to obtain the third target data includes: Based on the calibration date, query and retrieve the data to be processed corresponding to the calibration date from the first target data; A calibration task is generated based on the calibration date, and the calibration task includes at least the data to be processed and the calibration date. The calibration task is executed by calling a preset data processing service to recalculate and statistically analyze the data to be processed, and generate the third target data.

4. The data processing method according to claim 1, characterized in that, When the second target data is updated to the third target data, the first historical data of the monitoring platform is synchronized to obtain the second historical data, including: Based on the calibration date, determine at least one statistical time period that includes the calibration date; For each statistical time period, the statistical data corresponding to that statistical time period is recalculated based on the updated second target data and other data belonging to that statistical time period in the first historical data. The recalculated statistical data for each statistical time period is updated in the database record corresponding to the monitoring platform to overwrite the original statistical data. The updated statistical data constitutes the second historical data.

5. The data processing method according to claim 1, characterized in that, Before updating the second target data to the third target data, the data processing method further includes: Perform permission verification on the current user to determine whether the user has the permission to perform data updates; If the permission verification passes, a confirmation prompt message is displayed. The confirmation prompt message is used to request whether to update the second target data corresponding to the calibration date to the third target data. Receive a confirmation instruction for the confirmation prompt information, and execute the step of updating the second target data to the third target data; If the permission verification fails, the data calibration process will be terminated.

6. The data processing method according to claim 1, characterized in that, The method also includes, Based on the preset abnormal data monitoring model, the second historical data is monitored for abnormalities to obtain monitoring results; When the monitoring result falls below the preset data range, the data calibration process is automatically initiated.

7. A data processing apparatus, characterized in that, The data processing device includes: The first target data acquisition module is used to send a query instruction to the device to be monitored, so that the device to be monitored returns the first target data to the monitoring platform according to the query instruction. A data calibration module is configured to determine a calibration date based on the first target data and execute a data calibration process based on the calibration date. The data calibration process includes: acquiring second target data corresponding to the calibration date from the monitoring platform and first historical data from the monitoring platform; performing statistical processing on the data corresponding to the calibration date in the first target data to obtain third target data, and updating the second target data to the third target data; when the second target data is updated to the third target data, performing data synchronization processing on the first historical data from the monitoring platform to obtain second historical data.

8. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1-6.

9. A non-volatile computer-readable storage medium, characterized in that, The non-volatile computer-readable storage medium stores computer-executable instructions that, when executed by an electronic device, cause the electronic device to perform the method described in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program stored on a non-volatile computer-readable storage medium, the computer program including program instructions that, when executed by an electronic device, cause the electronic device to perform the method of any one of claims 1-6.

Images