Automatic test and evaluation method and system for load identification algorithm

Abstract

The invention discloses an automatic test and evaluation method of a load identification algorithm, and discloses a system with the automatic test and evaluation method of the load identification algorithm. According to the automatic test and evaluation method of the load identification algorithm, system construction is carried out from three aspects of test scene construction, scene complexity grading and intelligent evaluation, and self-adaptive construction and complexity calculation of a load identification test scene and multi-dimensional evaluation of the load identification algorithm are realized. The training and verification of the load identification model can complete standardized test evaluation without depending on a large amount of home-entry actual measurement data, the problems that a traditional method is single in test scene, scene complexity cannot be evaluated, and evaluation standards are not uniform are solved, and the quantifiable evaluation level and engineering application reliability of the load identification technology are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of smart grid power consumption side information sensing and energy efficiency analysis technology, and in particular to an automated testing and evaluation method and system for load identification algorithms. Background Technology

[0002] In non-intrusive load monitoring (NILM) technology, the identification system infers the operating status and power consumption behavior of individual appliances by analyzing signals such as total power and voltage from residential or industrial users. Currently, various load identification algorithms have been proposed, covering different technical approaches such as event detection, probabilistic modeling, and deep learning.

[0003] As a crucial technology for energy efficiency sensing at smart grid terminals, NILM (Non-Induced Load Modeling) must undergo scientific and comprehensive performance evaluation before practical deployment. Existing load identification testing methods face the following challenges: testing processes largely rely on manually constructed scenarios, lacking automated testing technology; inconsistent methods for describing dataset and scenario complexity lead to difficulties in cross-sectional comparisons; the lack of a unified evaluation index system results in highly subjective evaluation results; and evaluation indicators struggle to comprehensively cover dimensions such as algorithm accuracy, generalization, and real-time performance. Existing research largely focuses on improving the accuracy of appliance status identification and power / energy decomposition. However, a unified standard has yet to be established within the industry for scientifically and effectively evaluating the performance of NILM-related algorithms or products. Summary of the Invention

[0004] This invention aims to address at least one of the technical problems existing in the prior art. To this end, this invention proposes an automated testing and evaluation method for load identification algorithms. The system is constructed from three levels: "test scenario construction - scenario complexity classification - intelligent evaluation," realizing adaptive construction and complexity calculation of load identification test scenarios, as well as multi-dimensional evaluation of load identification algorithms.

[0005] This invention also proposes a system for an automated testing and evaluation method with the above-mentioned load identification algorithm.

[0006] An automated testing and evaluation method for a load identification algorithm according to a first aspect of the present invention is characterized by comprising the following steps:

[0007] Acquire access data and perform unified formatting and unit normalization on the access data;

[0008] Based on testing requirements, industrial load data from different scenarios are combined with the processed access data and noise is injected to obtain data from mixed load scenarios of multiple types.

[0009] The data of the multi-type load mixed scenario is classified into several complex level scenarios.

[0010] The data of each complexity level scenario is input into the load identification algorithm to be evaluated, the load identification result output by the load identification algorithm to be evaluated is obtained, and the load identification result is compared with the access data corresponding to the scenario to obtain the accuracy of the load identification algorithm to be evaluated in each complex scenario.

[0011] The accuracy across different complexity levels is used as the evaluation result for the load identification algorithm.

[0012] The automated testing and evaluation method for load identification algorithms according to embodiments of the present invention has at least the following beneficial effects: The automated testing and evaluation method for load identification algorithms provided by the present invention is systematically constructed from three levels: "test scenario construction—scenario complexity classification—intelligent evaluation," realizing adaptive construction and complexity calculation of load identification test scenarios, as well as multi-dimensional evaluation of load identification algorithms. Through automated test scenario generation and performance evaluation methods, the training and verification of load identification models can be completed with standardized testing and evaluation without relying on a large amount of on-site measurement data. This solves the problems of single test scenarios, inability to evaluate scenario complexity, and inconsistent evaluation standards in traditional methods, effectively improving the quantifiable evaluation level and engineering application reliability of load identification technology.

[0013] According to some embodiments of the present invention, the classification criteria include eight dimensions: number of devices, proportion of low-power devices, start-stop concurrency, degree of weak feature flooding, feature similarity, load dynamics, signal-to-noise ratio, and multi-stage operation load ratio.

[0014] According to some embodiments of the present invention, in the step of classifying the data of the multi-type load mixed scenario to divide the data of the multi-type load mixed scenario into data of several complexity level scenarios, it is assumed that each dimension is scored as D. i Let w ∈{0,1,2}. i Then the total score satisfies:

[0015]

[0016] default w i =1, Total score range: 0-16. Based on the total score, the data of the multi-type load mixed scenario are divided into: 0-4 points: level 1; 5-8 points: level 2; 9-12 points: level 3; 13-16 points: level 4.

[0017] According to some embodiments of the present invention, the accuracy includes at least one of the following: start-up and shutdown time identification accuracy, start-up and shutdown duration identification accuracy, cycle identification accuracy, attribute identification accuracy, and load power decomposition accuracy.

[0018] According to some embodiments of the present invention, the formula for the accuracy of the start-up and stop-down time identification is as follows:

[0019]

[0020] In the formula, Tc is the predicted start / stop time in the load identification result; Ts is the actual start / stop time, i.e. the original access data corresponding to the scenario; PT is the time matching accuracy of a single event; Pr1 is the average time accuracy of all events, i is the i-th event, and n represents the total number of events.

[0021] According to some embodiments of the present invention, the formula for the accuracy of the start-up and shutdown duration identification is as follows:

[0022]

[0023] In the formula, Lc is the predicted running time in the load identification result; Ls is the actual running time, i.e. the access data corresponding to the scenario; PL is the single running time identification accuracy; Pr2 is the average running time identification accuracy, i is the i-th event, and n represents the total number of events.

[0024] According to some embodiments of the present invention, the formula for the period identification accuracy is as follows:

[0025]

[0026] In the formula, Q c Q is the predicted cycle length in the load identification results; s For the actual period, i.e., the original data; Q H Pr3 represents the accuracy for a single cycle; Pr3 represents the average recognition accuracy over the cycle, where i is the i-th event and n represents the total number of events.

[0027] According to some embodiments of the present invention, the formula for the accuracy of attribute identification is as follows:

[0028]

[0029] In the formula, TP i For the i-th device, this represents the number of attributes correctly identified by the load identification result compared to the access data corresponding to that scenario; FP i In the context of load identification, the number of attributes incorrectly identified compared to the access data corresponding to the scenario is denoted as ; Pr4 represents the average attribute identification accuracy of all devices, i represents the i-th event, and n represents the total number of events.

[0030] According to some embodiments of the present invention, the formula for the accuracy of load power decomposition is as follows:

[0031]

[0032] In the formula, E i This represents the actual battery level, i.e., the raw data. δ represents the estimated electrical quantity in the load identification results. i For the overall power error, δ ∑ The total average power decomposition accuracy is given by i, where i is the i-th event and n represents the total number of events.

[0033] An automated testing and evaluation system for a load identification algorithm according to a second aspect of the present invention is characterized in that it comprises:

[0034] The data access module is capable of acquiring access data and performing unified formatting and unit normalization processing on the access data;

[0035] The test scenario generation module can combine industrial load data under different scenarios with the processed access data and inject noise based on test requirements to obtain data of mixed load scenarios of multiple types.

[0036] The scenario complexity classification module can classify the data of the multi-type load mixed scenario into several complexity levels.

[0037] The accuracy evaluation index module can input the data of the multi-type load mixed scenario into the load identification algorithm to be evaluated, obtain the load identification result output by the load identification algorithm to be evaluated, and compare the accuracy of the load identification result with the access data corresponding to the scenario to obtain the accuracy of the load identification algorithm to be evaluated.

[0038] The intelligent evaluation module can apply the load identification algorithm to different complexity levels of scenarios and use the combined accuracy under different complexity levels as the evaluation result of the load identification algorithm.

[0039] According to some embodiments of the present invention, the classification criteria include eight dimensions: number of devices, proportion of low-power devices, start-stop concurrency, degree of weak feature flooding, feature similarity, load dynamics, signal-to-noise ratio, and multi-stage operation load ratio.

[0040] According to some embodiments of the present invention, the scene complexity grading module includes: assuming each dimension is rated as D. i Let w ∈{0,1,2}. i The total score satisfies the following formula:

[0041]

[0042] default w i =1, Total score range: 0-16. Based on the total score, the data of the multi-type load mixed scenario are divided into: 0-4 points: level 1; 5-8 points: level 2; 9-12 points: level 3; 13-16 points: level 4.

[0043] According to some embodiments of the present invention, the accuracy evaluation index module evaluates the load identification algorithm by including the following performance indicators: start-up and stop time identification accuracy, start-up and stop duration identification accuracy, cycle identification accuracy, attribute identification accuracy, and load power decomposition accuracy.

[0044] According to some embodiments of the present invention, the formula for the accuracy of the start-up and stop-down time identification is as follows:

[0045]

[0046] In the formula, Tc is the predicted start / stop time in the load identification result; Ts is the actual start / stop time, i.e. the access data corresponding to the scenario; PT is the time matching accuracy of a single event; Pr1 is the average time accuracy of all events, i is the nth event, and n represents the total number of events.

[0047] According to some embodiments of the present invention, the formula for the accuracy of the start-up and shutdown duration identification is as follows:

[0048]

[0049] In the formula, Lc is the predicted running time in the load identification result; Ls is the actual running time, i.e. the access data corresponding to the scenario; PL is the single running time identification accuracy; Pr2 is the average running time identification accuracy, i is the i-th event, and n represents the total number of events.

[0050] According to some embodiments of the present invention, the formula for the period identification accuracy is as follows:

[0051]

[0052] In the formula, Q c Q is the predicted cycle length in the load identification results; s For the actual period, i.e., the original data; Q H Pr3 represents the accuracy for a single cycle; Pr3 represents the average recognition accuracy over the cycle, where i is the i-th event and n represents the total number of events.

[0053] According to some embodiments of the present invention, the formula for the accuracy of attribute identification is as follows:

[0054]

[0055] In the formula, TPi is the number of attributes correctly identified in the load identification result of the i-th device compared with the access data corresponding to the scenario; FPi is the number of attributes incorrectly identified in the load identification result compared with the access data corresponding to the scenario; Pr4 is the average attribute identification accuracy of all devices, i is the i-th event, and n represents the total number of events.

[0056] According to some embodiments of the present invention, the formula for the accuracy of load power decomposition is as follows:

[0057]

[0058] In the formula, E i This represents the actual battery level, i.e., the raw data. δ represents the estimated electrical quantity in the load identification results. i For the overall power error, δ Σ Let represent the overall average power decomposition accuracy, i be the i-th event, and n be the total number of events.

[0059] According to a third aspect of the present invention, the terminal includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the automated testing and evaluation method of the above-described load identification algorithm.

[0060] According to a fourth aspect of the present invention, a computer-readable storage medium stores computer-executable instructions for performing the automated testing and evaluation method of the above-described load identification algorithm.

[0061] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0062] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0063] Figure 1 This is a schematic diagram illustrating the steps of an automated testing and evaluation method for the load identification algorithm according to an embodiment of the present invention;

[0064] Figure 2 This is a structural block diagram of the automated testing and evaluation system for the load identification algorithm according to an embodiment of the present invention. Detailed Implementation

[0065] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0066] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0067] In the description of this invention, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0068] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0069] Example 1

[0070] To address the issues of inconsistent evaluation standards and strong subjectivity in existing non-intrusive load identification algorithms, embodiments of this invention provide an automated testing and evaluation method for load identification algorithms. The system is constructed from three levels: "test scenario setup—scenario complexity grading—intelligent evaluation," achieving adaptive construction and complexity calculation of load identification test scenarios, as well as multi-dimensional evaluation of load identification algorithms. Figure 1 As shown, the method includes:

[0071] Step S100: Obtain access data and perform unified formatting and unit normalization processing on the access data.

[0072] First, the standard load dataset and self-collected data are accessed. The accessed data are then formatted, normalized in terms of units, and classified.

[0073] Step S200: Based on testing requirements, combine industrial load data from different scenarios with the processed access data and inject noise to obtain data from mixed load scenarios of multiple types.

[0074] By importing various industrial load data into the system, and automatically combining test data segments, injecting noise, and generating multiple mixed load scenarios according to specific test requirements, the comprehensiveness and robustness of the test are enhanced.

[0075] Step S300: Classify the data of the multi-type load mixed scenario to divide the data of the multi-type load mixed scenario into data of several complex level scenarios.

[0076] The complexity of data scenarios is graded based on factors such as the number of devices, the proportion of low-power devices, and the similarity of electrical appliances, providing a quantitative basis for subsequent testing. An embodiment of this application provides a complexity evaluation index system comprising eight dimensions, as shown in Table 1 below:

[0077] Table 1. Scene Complexity Indicators and Scoring Criteria

[0078] Dimension number Dimension Name Scoring criteria (0 / 1 / 2) D1 Number of devices ≤1(0)，2-3(1)，>4(2) D2 Low-power devices account for a certain percentage ≤20％(0)，20-50％(1)，>50％(2) D3 Start-stop concurrency <2 times / 5 minutes (0), 2-5 (1), >5 (2) D4 Weak feature submersion degree None (0), Occasionally (1), Frequently (2) D5 Feature similarity Clearly distinguishable (0), similar (1), highly overlapping (2) D6 Load dynamics <3 times / 15 minutes (0), 3-6 times (1), >6 times (2) D7 Signal-to-noise ratio ≥10dB(0),5dB-10dB(1),<5dB(2) D8 Multi-stage operating load ratio None (0), <50% of equipment has (1), ≥50% (2)

[0079] D8 multi-stage operation load ratio: None (0), <50% equipment has (1), ≥50% (2)

[0080] The number of devices refers to the actual number of active electrical appliance types within the test segment, i.e., the number of different device types detected with obvious start-stop behavior or continuous fluctuation characteristics. The more device types there are, the larger the combined state space becomes, requiring the load identification algorithm to process more overlapping and interference information, significantly increasing task complexity.

[0081] The proportion of low-power devices refers to the frequency or duration of loads with an average power below a certain threshold (e.g., 50W) appearing in the current segment. Due to their low signal amplitude, they are easily masked by high-power loads, causing significant interference to the detection.

[0082] The start-stop concurrency level indicates the degree to which multiple devices start and stop simultaneously within a unit time window. If device start-stop behaviors occur in a concentrated manner, it may lead to feature aliasing, reducing the accuracy of event separation and recognition.

[0083] The degree of weak feature submersion reflects whether low-power or short-term events occur during the operation fluctuation or switching phase of high-power equipment. If they overlap, their features are easily masked by "background" fluctuations, affecting observability.

[0084] Feature similarity refers to the degree of similarity between waveform characteristics (such as current amplitude, rate of change, duration, etc.) of different devices during start-up and shutdown. When multiple devices have similar characteristics, the algorithm faces higher requirements for its ability to distinguish between them, and misjudgment or confusion can easily occur.

[0085] Load dynamics describes the frequency and magnitude of power fluctuations during equipment operation. The more drastic the dynamic changes, the more the identification system needs to have good time-series modeling and feature tracking capabilities.

[0086] Signal-to-noise ratio (SNR) reflects the relative strength of stable components and high-frequency noise or perturbation in a total power or current waveform. The lower the SNR, the more difficult it is to separate the effective components in the signal, which can easily affect the recognition accuracy.

[0087] Multi-stage operating load ratio represents the proportion of equipment with multiple operating levels or continuous adjustment characteristics (such as air conditioners, washing machines, induction cookers, etc.) in the test segment. The operating status of such equipment is relatively complex, and atypical binary start-stop modes will increase the difficulty of model modeling and state recognition.

[0088] All the above dimensions are automatically extracted and quantified by algorithms, without relying on any device labels or manual annotations, ensuring the automation and objectivity of the testing process. During the analysis of each segment, the system assigns a discrete level to each dimension, scores it based on the strength of its corresponding features, and finally calculates the overall complexity score of the segment using a weighted scoring mechanism. This score can be further divided into four levels: level 1, level 2, level 3, and level 4.

[0089] Assume each dimension has a score Di∈{0,1,2}, and let the weight w i The total score is:

[0090]

[0091] w i =1, total score range: 0-16, the test segment is divided into the following levels according to the score: 0-4 points: level 1; 5-8 points: level 2; 9-12 points: level 3; 13-16 points: level 4.

[0092] Step S400: Input the data of the multi-type load mixed scenario of each complexity level into the load identification algorithm to be evaluated, obtain the load identification result output by the load identification algorithm to be evaluated, and compare the accuracy of the load identification result with the original data to obtain the accuracy of the load identification algorithm to be evaluated in each complex scenario.

[0093] Simulated voltage and current waveforms generated from mixed load scenarios are input into the load identification algorithm to be evaluated, and the output load identification results are obtained. The identification results are then output through a digital interface and compared with the original data for accuracy.

[0094] Furthermore, the evaluation of the identification results includes the following key performance indicators: accuracy of start-up and shutdown time identification, accuracy of start-up and shutdown duration identification, accuracy of cycle identification, accuracy of attribute identification, and accuracy of load power decomposition. Among these:

[0095] The formula for the accuracy of start-up and stop time identification satisfies:

[0096]

[0097]

[0098] In the formula, Tc is the predicted start / stop time; Ts is the actual start / stop time; PT is the time matching accuracy of a single event; and Pr1 is the average time accuracy of all events.

[0099] The formula for determining the accuracy of start-up and shutdown duration is as follows:

[0100]

[0101] In the formula, Lc is the predicted runtime; Ls is the actual runtime; PL is the single runtime recognition accuracy; and Pr2 is the average runtime recognition accuracy.

[0102] The formula for the accuracy of period identification is as follows:

[0103]

[0104] In the formula, Q c Q represents the predicted cycle length (e.g., operating cycle). s For the actual period; Q H Pr3 represents the accuracy for a single cycle; Pr4 represents the average recognition accuracy over the cycle.

[0105] The formula for attribute identification accuracy is as follows:

[0106]

[0107] In the formula, TP i FP represents the number of attributes that the i-th device is correctly identified. i Pr4 represents the number of incorrectly identified attributes; Pr4 represents the average attribute identification accuracy across all devices.

[0108] The formula for the accuracy of load power decomposition is as follows:

[0109]

[0110] In the formula, E i This is the actual battery level. To estimate the amount of electricity, δ i This is for overall battery level error. To decompose the error step by step; δ ∑ The accuracy of the total average charge decomposition.

[0111] The load identification algorithm to be evaluated is applied to scenarios with different complexity levels obtained from the classification. The final load identification accuracy rating is output as the evaluation result of the load identification algorithm.

[0112] Step S500: The accuracy under different complexity levels is used as the evaluation result of the load identification algorithm.

[0113] Preferably, for a specific industry (such as agricultural irrigation and drainage, industrial manufacturing, commercial buildings, etc.), the relevant load identification indicators are calculated separately and then comprehensively to obtain the identification accuracy level for that industry. These indicators are then weighted and averaged according to their respective weights (1 / 6 in this scheme) to obtain the overall accuracy score for a single industry.

[0114] A grade threshold is set based on the accuracy score (≥90% is A, 80%-90% is B, 70%-80% is C, <70% is D).

[0115]

[0116] Where Prk is the accuracy of the k-th load identification-related indicator in the industry, and wk is the weight.

[0117] Based on the above steps, a load identification algorithm is used to select mixed load scenarios with similar complexity for equipment identification in the agricultural irrigation and drainage industry. The test results of the load identification function of the blanket rolling machine are evaluated as follows:

[0118]

[0119]

[0120] As can be seen, in this single complexity evaluation scenario, four out of five items passed, resulting in an overall evaluation of A.

[0121] By repeating the above process in scenarios of different complexities and obtaining comprehensive evaluations, a comprehensive evaluation of the performance of the load identification algorithm under different complexity scenarios can be made.

[0122] This application embodiment utilizes multi-source data access, uniformly formatting and normalizing the units of standard load datasets and self-collected data, and then classifying the accessed data. Based on specific testing requirements, it automatically combines test data segments, injects noise, and generates various mixed load scenarios to enhance the comprehensiveness and robustness of the test. A scenario complexity calculation module constructs a data scenario complexity hierarchy based on factors such as the number of devices, the proportion of low-power devices, and the similarity of electrical appliances, providing a quantitative basis for subsequent testing. Finally, it automatically calculates key performance indicators such as the accuracy of start-up and shutdown time identification, start-up and shutdown duration identification, cycle identification, attribute identification, and load power decomposition based on a preset indicator library, providing a final load identification accuracy evaluation level. The system can be constructed from three levels: "test scenario construction—scenario complexity hierarchy—intelligent evaluation," achieving adaptive construction and complexity calculation of load identification test scenarios, as well as multi-dimensional evaluation of the load identification algorithm.

[0123] Example 2

[0124] Another aspect of the present invention provides an automated testing and evaluation system for a load identification algorithm, such as... Figure 2 As shown, the system 20 includes:

[0125] The data access module 201 is capable of acquiring access data and performing unified formatting and unit normalization processing on the access data;

[0126] The test scenario generation module 202 can combine industrial load data under different scenarios with the processed access data and inject noise based on test requirements to obtain data of mixed load scenarios of multiple types.

[0127] The scenario complexity classification module 203 can classify the data of the multi-type load mixed scenario to divide the data of the multi-type load mixed scenario into several complexity levels.

[0128] The accuracy evaluation index module 204 can input the data of the multi-type load mixed scenario into the load identification algorithm to be evaluated, obtain the load identification result output by the load identification algorithm to be evaluated, and compare the accuracy of the load identification result with the original data to obtain the accuracy of the load identification algorithm to be evaluated.

[0129] The intelligent evaluation module 205 can apply the load identification algorithm to different complexity levels of scenarios and take the accuracy under different complexity levels as the evaluation result of the load identification algorithm.

[0130] Furthermore, the classification criteria include eight dimensions: number of devices, proportion of low-power devices, start-stop concurrency, degree of weak feature flooding, feature similarity, load dynamics, signal-to-noise ratio, and multi-stage operation load ratio.

[0131] Furthermore, the scene complexity grading module 203 includes: assuming each dimension is rated as D. i Let w ∈{0,1,2}. i The total score satisfies the following formula:

[0132]

[0133] default w i =1, total score range: 0-16, the test segment is divided into the following levels according to the score: 0-4 points: level 1; 5-8 points: level 2; 9-12 points: level 3; 13-16 points: level 4.

[0134] Furthermore, in the accuracy evaluation index module 204, the performance indicators for evaluating the load identification algorithm include the following: accuracy of start-up and shutdown time identification, accuracy of start-up and shutdown duration identification, accuracy of cycle identification, accuracy of attribute identification, and accuracy of load power decomposition.

[0135] Furthermore, in the accuracy evaluation index module 204, the formula for the accuracy of start-up and shutdown time identification satisfies:

[0136]

[0137] In the formula, Tc is the predicted start / stop time; Ts is the actual start / stop time; PT is the time matching accuracy of a single event; and Pr1 is the average time accuracy of all events.

[0138] Furthermore, in the accuracy evaluation index module 204, the formula for the accuracy of start-up and shutdown duration identification is as follows:

[0139]

[0140] In the formula, Lc is the predicted runtime; Ls is the actual runtime; PL is the single runtime recognition accuracy; and Pr2 is the average runtime recognition accuracy.

[0141] Furthermore, in the accuracy evaluation index module 204, the formula for the period identification accuracy is as follows:

[0142]

[0143] In the formula, Q c Q represents the predicted cycle length (e.g., operating cycle). s For the actual period; QH Pr3 represents the accuracy for a single cycle; Pr4 represents the average recognition accuracy over the cycle.

[0144] Furthermore, in the accuracy evaluation index module 204, the formula for attribute identification accuracy is as follows:

[0145]

[0146] In the formula, TP i FP represents the number of attributes that the i-th device is correctly identified. i Pr4 represents the number of incorrectly identified attributes; Pr4 represents the average attribute identification accuracy across all devices.

[0147] Furthermore, in the accuracy evaluation index module 204, the formula for the accuracy of load power decomposition is as follows:

[0148]

[0149] In the formula, E i This is the actual battery level. To estimate the amount of electricity, δ i This is for overall battery level error. To decompose the error step by step; δ ∑ The accuracy of the total average charge decomposition.

[0150] The embodiments of this application utilize a data access module to uniformly format and normalize the accessed data, as well as classify it, for both standard load datasets and self-collected data. Secondly, an automatic test scenario generation module can automatically combine test data segments, inject noise, and generate various mixed load scenarios according to specific test requirements, enhancing the comprehensiveness and robustness of the test. Then, a scenario complexity calculation module constructs a data scenario complexity hierarchy based on factors such as the number of devices, the proportion of low-power devices, and the similarity of electrical appliances, providing a quantitative basis for subsequent testing. In the test evaluation phase, an intelligent evaluation module can automatically calculate key performance indicators such as the accuracy of start-up / stop time identification, the accuracy of start-up / stop duration identification, the accuracy of cycle identification, the accuracy of attribute identification, and the accuracy of load power decomposition, based on a preset indicator library. Finally, the scores of each indicator are merged to output the final load identification accuracy evaluation level.

[0151] Another aspect of the present invention provides a terminal, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the automated testing and evaluation method of the above-described load identification algorithm.

[0152] Specifically, the processor can be a CPU, a general-purpose processor, a DSP, an ASIC, an FPGA, or other programmable logic device, transistor logic device, hardware component, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0153] Specifically, the processor connects to the memory via a bus, which may include a path for transmitting information. The bus can be a PCI bus or an EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc.

[0154] The memory may be a ROM or other type of static storage device capable of storing static information and instructions, a ROM or other type of dynamic storage device capable of storing information and instructions, or an EEPROM, CD-ROM or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0155] Optionally, the memory stores the code of the computer program that executes the scheme of this application, and the execution is controlled by the processor. The processor executes the application code stored in the memory to implement... Figure 2 The embodiment shown illustrates the functionality of an automated testing and evaluation system for a load identification algorithm.

[0156] Another aspect of the present invention provides a computer-readable storage medium storing computer-executable instructions for performing the above-described... Figure 1 The method for automated testing and evaluation of the load identification algorithm is shown.

[0157] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0158] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0159] The above is a detailed description of the preferred embodiments of this application. However, this application is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. An automated testing and evaluation method for a load identification algorithm, characterized in that, Includes the following steps: Acquire access data and perform unified formatting and unit normalization on the access data; Based on testing requirements, industrial load data from different scenarios are combined with the processed access data and noise is injected to obtain data from mixed load scenarios of multiple types. The data of the multi-type load mixed scenario is classified into several complex level scenarios. The data of each complexity level scenario is input into the load identification algorithm to be evaluated, the load identification result output by the load identification algorithm to be evaluated is obtained, and the load identification result is compared with the access data corresponding to the scenario to obtain the accuracy of the load identification algorithm to be evaluated in each complex scenario. The accuracy across different complexity levels is used as the evaluation result for the load identification algorithm.

2. The method according to claim 1, characterized in that, The classification is based on eight dimensions: number of devices, proportion of low-power devices, start-stop concurrency, degree of weak feature flooding, feature similarity, load dynamics, signal-to-noise ratio, and multi-stage operation load ratio.

3. The method according to claim 2, characterized in that, In the step of classifying the data of the multi-type load mixed scenario into data of several complexity levels, it is assumed that each dimension is scored as D. i Let w ∈{0,1,2}. i Then the total score satisfies: default w i =1, Total score range: 0-16. Based on the total score, the data of the multi-type load mixed scenario are divided into: 0-4 points: level 1; 5-8 points: level 2; 9-12 points: level 3; 13-16 points: level 4.

4. The method according to claim 1, characterized in that, The accuracy includes at least one of the following: accuracy of start-up and shutdown time identification, accuracy of start-up and shutdown duration identification, accuracy of cycle identification, accuracy of attribute identification, and accuracy of load power decomposition.

5. The method according to claim 4, characterized in that, The formula for the accuracy of the start-up and stop-down time identification is as follows: In the formula, Tc is the predicted start / stop time in the load identification result; Ts represents the actual power-on / power-off time, i.e., the access data corresponding to this scenario; PT represents the time matching accuracy of a single event. Pr1 represents the average time precision of all events, i represents the i-th event, and n represents the total number of events.

6. The method according to claim 4, characterized in that, The formula for determining the accuracy of start-up and shutdown duration identification is as follows: In the formula, Lc is the predicted running time in the load identification result; Ls is the actual running time, i.e. the access data corresponding to the scenario; PL is the single running time identification accuracy; Pr2 is the average running time identification accuracy, i is the i-th event, and n represents the total number of events.

7. The method according to claim 4, characterized in that, The formula for the accuracy of period identification is as follows: In the formula, Q c Q is the predicted cycle length in the load identification results; s For the actual period, i.e., the original data; Q H Pr3 represents the accuracy for a single cycle; Pr3 represents the average recognition accuracy over the cycle, where i is the i-th event and n represents the total number of events.

8. The method according to claim 4, characterized in that, The formula for the accuracy of attribute identification is as follows: In the formula, TP i For the i-th device, this represents the number of attributes correctly identified by the load identification result compared to the access data corresponding to that scenario; FP i In this context, the number of attributes incorrectly identified by the load identification results compared to the access data corresponding to the scenario is considered. Pr4 represents the average attribute recognition accuracy across all devices, i represents the i-th event, and n represents the total number of events.

9. The method according to claim 4, characterized in that, The formula for the accuracy of load power decomposition is as follows: In the formula, E i This represents the actual battery level, i.e., the raw data. δ represents the estimated electrical quantity in the load identification results. i For the overall power error, δ ∑ Let represent the overall average power decomposition accuracy, i be the i-th event, and n be the total number of events.

10. An automated testing and evaluation system for a load identification algorithm, characterized in that, include: The data access module is capable of acquiring access data and performing unified formatting and unit normalization processing on the access data; The test scenario generation module can combine industrial load data under different scenarios with the processed access data and inject noise based on test requirements to obtain data of mixed load scenarios of multiple types. The scenario complexity classification module can classify the data of the multi-type load mixed scenario into several complexity levels. The accuracy evaluation index module can input the data of the multi-type load mixed scenario into the load identification algorithm to be evaluated, obtain the load identification result output by the load identification algorithm to be evaluated, and compare the accuracy of the load identification result with the access data corresponding to the scenario to obtain the accuracy of the load identification algorithm to be evaluated. The intelligent evaluation module can apply the load identification algorithm to different complexity levels of scenarios and use the combined accuracy under different complexity levels as the evaluation result of the load identification algorithm.

11. The system according to claim 10, characterized in that, The classification criteria include eight dimensions: number of devices, proportion of low-power devices, start-stop concurrency, degree of weak feature flooding, feature similarity, load dynamics, signal-to-noise ratio, and multi-stage operation load ratio.

12. The system according to claim 11, characterized in that, The scenario complexity grading module includes: assuming each dimension is rated as D. i Let w ∈{0,1,2}. i The total score satisfies the following formula: default w i =1, Total score range: 0-16. Based on the total score, the data of the multi-type load mixed scenario are divided into: 0-4 points: level 1; 5-8 points: level 2; 9-12 points: level 3; 13-16 points: level 4.

13. The system according to claim 10, characterized in that, The accuracy includes at least one of the following: accuracy of start-up and shutdown time identification, accuracy of start-up and shutdown duration identification, accuracy of cycle identification, accuracy of attribute identification, and accuracy of load power decomposition.

14. The system according to claim 13, characterized in that, The formula for the accuracy of the start-up and stop-down time identification is as follows: In the formula, Tc is the predicted start / stop time in the load identification result; Ts represents the actual power-on / power-off time, i.e., the access data corresponding to this scenario; PT represents the time matching accuracy of a single event. Pr1 represents the average time precision of all events, i represents the nth event, and n represents the total number of events.

15. The system according to claim 13, characterized in that, The formula for determining the accuracy of start-up and shutdown duration identification is as follows: In the formula, Lc is the predicted running time in the load identification result; Ls is the actual running time, i.e. the access data corresponding to the scenario; PL is the single running time identification accuracy; Pr2 is the average running time identification accuracy, i is the i-th event, and n represents the total number of events.

16. The system according to claim 13, characterized in that, The formula for the accuracy of period identification is as follows: In the formula, Q c Q is the predicted cycle length in the load identification results; s For the actual period, i.e., the original data; Q H Pr3 represents the accuracy for a single cycle; Pr3 represents the average recognition accuracy over the cycle, where i is the i-th event and n represents the total number of events.

17. The system according to claim 13, characterized in that, The formula for the accuracy of attribute identification is as follows: In the formula, TPi represents the number of attributes correctly identified in the load identification result of the i-th device compared with the access data corresponding to the scenario; FPi represents the number of attributes incorrectly identified in the load identification result compared with the access data corresponding to the scenario. Pr4 represents the average attribute recognition accuracy across all devices, i represents the i-th event, and n represents the total number of events.

18. The system according to claim 13, characterized in that, The formula for the accuracy of load power decomposition is as follows: In the formula, E i This represents the actual battery level, i.e., the raw data. δ represents the estimated electrical quantity in the load identification results. i For the overall power error, δ ∑ Let represent the overall average power decomposition accuracy, i be the i-th event, and n be the total number of events.

19. A terminal, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the method of any one of claims 1 to 9.

20. A computer-readable storage medium storing computer-executable instructions for performing the method of any one of claims 1 to 9.

Images