Model prediction performance evaluation method, device, equipment and medium

Abstract

The invention discloses a model prediction performance evaluation method and device, equipment and a storage medium, and the method comprises the steps: obtaining a test sample set, screening the test sample set, and obtaining a typical wind speed daily sample; the test sample set comprises at least one wind speed daily sample; respectively inputting the typical wind speed daily samples into a plurality of prediction models, and outputting wind speed prediction results; determining statistics between prediction models based on the wind speed prediction result and a preset asymmetric threshold loss function; constructing a test based on the asymmetric threshold loss function, and determining a probability value of the test; and determining prediction model performance for the typical wind speed daily sample based on the statistics and the probability value. By utilizing the method, the problems that an existing evaluation method is insufficient to evaluate the prediction capability of the model and whether significant difference exists or not are solved, the meteorological intelligent early warning capability for the transformer substation is improved, and technical support is provided for reliable operation of the transformer substation.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of meteorological technologies, and in particular, to a method, device, equipment, and storage medium for evaluating the prediction performance of a model. Background Art

[0002] Currently, unattended operation of substation in power grids of 220 kV and below and less-attended operation of substation in power grids above 500 kV are gradually becoming trends, which pose higher requirements for the operation and maintenance of primary equipment in substations.

[0003] In recent years, substation intelligent operation and maintenance technologies have been continuously developing. With the continuous application of various meteorological-related sensors, meteorological data are often used to form samples, an early warning model based on the samples is established, and then extrapolation is performed using the model to achieve early warning of a certain meteorological feature (such as temperature prediction, wind speed prediction). Generally speaking, the smaller the error of the prediction outside the sample, the stronger the early warning performance of the model. The most common approach is to use some statistical indicators to evaluate the early warning performance of the model. Practical experience shows that the early warning evaluation of the performance of the early warning model does not lie in evaluating the size of the error of a single data point, but rather in looking at the overall performance of the early warning model. Using statistical indicators to determine the comparison results has great randomness. Sometimes, the difference in the actual performance of two early warning models will be overwhelmed by this randomness, resulting in incorrect model selection and it is also difficult to obtain the best early warning model. Summary of the Invention

[0004] The embodiments of the present invention provide a method, device, equipment, and storage medium for evaluating the prediction performance of a model, which solve the problem that the existing evaluation method is insufficient to evaluate the prediction ability of the model and whether there are significant differences, improve the meteorological intelligent early warning ability for substations, and provide technical support for the reliable operation of substations.

[0005] In a first aspect, the embodiments of the present invention provide a method for evaluating the prediction performance of a model. The computer device is communicatively connected to an electronic scale, and the method includes:

[0006] Obtain the weight range of each product combination;

[0007] Obtain the factory weight of the current product combination weighed by the electronic scale;

[0008] Compare the factory weight with the weight range corresponding to the current product combination to obtain a test result.

[0009] In a second aspect, the embodiments of the present invention further provide a device for evaluating the prediction performance of a model. The device includes:

[0010] Obtain a pre-configured operation priority; the operation priority includes: the operation priority of judging the brightness comparison result is greater than the operation priority of judging the light distribution situation;

[0011] Judge the brightness comparison result and the light distribution according to the operation priority;

[0012] When it is judged that the brightness comparison result is that the ambient brightness is low and the light distribution is that there is only infrared light, obtain the control operation as the first configuration file for turning on the infrared lamp, and execute the first configuration file to complete the evaluation of the model prediction performance;

[0013] When it is judged that the brightness comparison result is that the ambient brightness is high and the light distribution is that visible light is included, obtain the control operation as the second configuration file for turning off the infrared lamp, and execute the second configuration file to complete the evaluation of the model prediction performance.

[0014] In a third aspect, an embodiment of the present disclosure further provides an electronic device, where the electronic device includes:

[0015] One or more processors;

[0016] A storage device for storing one or more programs,

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the model prediction performance evaluation method provided by the embodiment of the present disclosure.

[0018] In a fourth aspect, an embodiment of the present disclosure further provides a storage medium containing computer-executable instructions, and the computer-executable instructions are used to execute the model prediction performance evaluation method provided by the embodiment of the present disclosure when executed by a computer processor.

[0019] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, where the computer program product includes a computer program, and the computer program implements the model prediction performance evaluation method provided by the embodiment of the first aspect when executed by a processor.

[0020] The present invention discloses a method, apparatus, device and storage medium for evaluating the performance of a model prediction. The method includes: obtaining a test sample set and screening the test sample set to obtain typical wind speed daily samples; the test sample set includes at least one wind speed daily sample; inputting the typical wind speed daily samples into a plurality of prediction models respectively to output wind speed prediction results; determining a statistic between the prediction models based on the wind speed prediction results and a preset asymmetric threshold loss function; constructing a test based on the asymmetric threshold loss function to determine the probability value of the test; and determining the performance of the prediction model for the typical wind speed daily samples based on the statistic and the probability value. By using this method, the problem that the existing evaluation method is insufficient to evaluate the prediction ability of the model and whether there are significant differences is solved, the meteorological intelligent early warning ability for substations is improved, and technical support is provided for the reliable operation of substations. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In combination with the accompanying drawings and with reference to the following specific embodiments, the above and other features, advantages and aspects of the various embodiments of the present disclosure will become more apparent. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the original elements and elements are not necessarily drawn to scale.

[0022] Figure 1 It is a flowchart of a method for evaluating the performance of a model prediction provided by an embodiment of the present disclosure;

[0023] Figure 2 It is a graphical display of the asymmetric threshold loss function provided by an embodiment of the present disclosure;

[0024] Figure 3 It is a schematic structural diagram of an apparatus for evaluating the performance of a model prediction provided by an embodiment of the present disclosure;

[0025] Figure 4 It is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0026] The embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not used to limit the protection scope of the present disclosure.

[0027] It should be understood that the various steps described in the method embodiments of the present disclosure may be executed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this regard.

[0028] As used herein, the term "comprising" and its variants are open-ended, i.e., "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0029] It should be noted that the concepts such as "first", "second", etc. mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependent relationships.

[0030] It should be noted that the modifications of "one" and "plural" mentioned in the present disclosure are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly specified in the context, it should be understood as "one or more".

[0031] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only for illustrative purposes and are not used to limit the scope of these messages or information.

[0032] It can be understood that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, usage scopes, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0033] For example, when responding to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested by the user will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, an application program, a server, or a storage medium that performs the operations of the technical solutions of the present disclosure according to the prompt message.

[0034] As an optional but non-limiting implementation manner, for example, the manner of sending a prompt message to the user in response to receiving an active request from the user may be in the form of a pop-up window, and the prompt message may be presented in text in the pop-up window. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0035] It can be understood that the above-mentioned notification and the process of obtaining user authorization are only illustrative and do not limit the implementation manner of the present disclosure. Other manners that comply with relevant laws and regulations can also be applied to the implementation manner of the present disclosure.

[0036] It can be understood that the data involved in the present technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of the corresponding laws, regulations and related provisions.

[0037] Figure 1 The figure is a flowchart of a model prediction performance evaluation provided by an embodiment of the present disclosure. The embodiment of the present disclosure is applicable to a situation where the problem that the existing evaluation method is insufficient to evaluate the prediction ability of the model and whether there are significant differences is solved. This method can be executed by a model prediction performance evaluation device, and the device can be implemented in the form of software and / or hardware. Optionally, it is implemented by an electronic device, and the electronic device can be a mobile terminal, a PC or a server, etc.

[0038] As Figure 1 shown, an embodiment of the present disclosure provides a model prediction performance evaluation method, and the method can specifically include the following steps:

[0039] S110. Obtain a test sample set and screen the test sample set to obtain a typical wind speed day sample.

[0040] Wherein, the test sample set includes at least one wind speed day sample.

[0041] In this embodiment, the test sample set can be a set composed of multiple wind speed day samples for testing. The wind speed day sample can be a wind speed data sample obtained by sampling the wind speed of the day at an interval. The typical wind speed day sample can be a wind speed day with specific typical characteristics in the sample. The interval can be 5 minutes.

[0042] The typical wind speed day sample includes at least one of the following: the wind speed day with the maximum wind speed; the wind speed day with the largest wind speed change and the wind speed day with the smallest wind speed change.

[0043] Based on the above embodiment, obtaining a test sample set and screening the test sample set to obtain a typical wind speed day sample can include: respectively determining the mean and variance of each wind speed day sample, taking the wind speed day sample with the highest mean as the wind speed day with the maximum wind speed, taking the wind speed day sample with the largest variance as the wind speed day with the largest wind speed change, and taking the wind speed day sample with the smallest variance as the wind speed day with the smallest wind speed change.

[0044] Specifically, respectively determine the mean and variance of each wind speed day sample, take the wind speed day sample with the highest mean as the wind speed day with the maximum wind speed, take the wind speed day sample with the largest variance as the wind speed day with the largest wind speed change, and take the wind speed day sample with the smallest variance as the wind speed day with the smallest wind speed change.

[0045] S120. Input the typical daily wind speed samples into multiple prediction models respectively, and output the wind speed prediction results.

[0046] Among them, the prediction models include: a prediction model based on a convolutional neural network and a long short-term memory network; a prediction model based on a support vector machine, and a prediction model based on a backpropagation model.

[0047] Specifically, input the typical daily wind speed samples into multiple prediction models respectively, and output the wind speed prediction results.

[0048] S130. Determine the statistic between models based on the wind speed prediction results and a preset asymmetric threshold loss function.

[0049] The asymmetric threshold loss function is as follows:

[0050]

[0051] Where L aT is the loss function, is the model prediction value, y t i is the true value, e i,t is the prediction error of the prediction model, l aT,i is the asymmetric adjustment parameter, p1, p2, q1, q2 are positive integer power parameters respectively; a1, a2, b1, b2 are asymmetric index parameters respectively, T1 is the positive error threshold value, and T2 is the negative error threshold value.

[0052] Figure 2 This is the graphical display of the asymmetric threshold loss function provided by the embodiments of the present disclosure. As Figure 2 shown, it can be easily seen that the function is a continuous non-differentiable function, and et = -4, et = 0, et = 3 are the non-differentiable points of the function.

[0053] Based on the above embodiments, determining the statistic between models based on the wind speed prediction results and a preset asymmetric threshold loss function may include the following steps:

[0054] a1) Calculate the prediction error of each prediction model according to the wind speed prediction results and the actual wind speed corresponding to the wind speed daily samples.

[0055] b1) Determine the error difference between the prediction errors of each prediction model, and determine the average value and standard deviation of the error difference.

[0056] c1) Determine the statistic between models according to the average value and standard deviation.

[0057] Specifically, the prediction errors of each prediction model are calculated based on the wind speed prediction results and the actual wind speeds in the daily wind speed samples. Then, the prediction errors are substituted into the above-mentioned preset asymmetric threshold loss function to determine the average value and standard deviation of the error differences. Finally, the average value and standard deviation are substituted into the statistical quantity calculation formula to determine the statistical quantity between the models.

[0058] S140. Construct a test based on the asymmetric threshold loss function to determine the p-value of the test.

[0059] Specifically, the null hypothesis is that the prediction efficiencies of the two models are the same, and the alternative hypothesis is that the prediction efficiencies of the two models are different. Through multiple random samplings (with replacement), a large number (such as 1000) of pseudo-samples are generated from the original dataset. The statistical quantity is calculated for each pseudo-sample, and the empirical distribution function of the statistical quantity is constructed. Then, the p-value of the actual statistical quantity is calculated based on this empirical distribution function.

[0060] S150. Determine the performance of the prediction model for the typical daily wind speed samples based on the statistical quantity and the p-value.

[0061] Based on the above embodiments, determining the performance of the prediction model for the typical daily wind speed samples based on the statistical quantity and the p-value may include the following steps:

[0062] a2) Compare the p-value with the preset probability threshold. If it is less than the probability threshold, there is a performance difference between the prediction models for the typical daily wind speed samples; if it is greater than or equal to the probability threshold, the performance difference between the prediction models for the typical daily wind speed samples is small.

[0063] b2) When there is a performance difference, determine the performance of the prediction model for the typical daily wind speed samples according to the statistical quantity.

[0064] In this embodiment, the probability threshold may be a preset threshold, which may be 0.05 or 0.01 in this embodiment.

[0065] Exemplarily, as shown in Table 1, Model A is a prediction model based on a convolutional neural network and a long short-term memory network, Model B is a prediction model based on a support vector machine, and Model C is a prediction model based on a backpropagation model. Based on three typical daily wind speeds, the performance of Model A is significantly higher than that of other models. Based on the wind speed data with the smallest wind speed change, the prediction ability of Model B2 is better than that of Model C2; based on the wind speed data with the largest wind speed and the smallest wind speed change, the warning abilities of Models B and C are equivalent.

[0066] This is in line with general experience, that is: Model A generally performs better than Models B and C, and the prediction ability of Model B is better than that of C when the wind speed fluctuates violently (representative day 2).

[0067] Table 1 Test results of three groups of asymmetric threshold loss functions

[0068] Maximum wind speed Model A1 and Model B1 Model B1 and Model C1 Model C1 and Model A1 Statistic -2.7 -0.34 2.5 Probability value 0.012 0.451 0.003 Minimum wind speed change Model A2 and Model B2 Model B2 and Model C2 Model C2 and Model A2 Statistic -3.2 -0.79 4.2 Probability value 0.012 0.089 0.005 Maximum wind speed change Model A3 and Model B3 Model B3 and Model C3 Model C3 and Model A3 Statistic -3.6 0.64 3.2 Probability value 0.023 0.272 0.030

[0069] The present invention discloses a method for evaluating the prediction performance of a model. The method includes: obtaining a test sample set and screening the test sample set to obtain typical wind speed daily samples; the test sample set includes at least one wind speed daily sample; inputting the typical wind speed daily samples into multiple prediction models respectively to output wind speed prediction results; determining a statistic between the prediction models based on the wind speed prediction results and a preset asymmetric threshold loss function; constructing a test based on the asymmetric threshold loss function to determine the probability value of the test; and determining the performance of the prediction model for the typical wind speed daily samples based on the statistic and the probability value. By using this method, the problems that the existing evaluation methods are insufficient to evaluate the prediction ability of the model and whether there are significant differences are solved, the meteorological intelligent early warning ability for substations is improved, and technical support is provided for the reliable operation of substations.

[0070] Figure 3 The present invention also provides a schematic structural diagram of a model prediction performance evaluation device, as Figure 3 shown. The device includes: an information acquisition module 210, a prediction result module 220, a statistic determination module 230, a test module 240, and a performance analysis module 250.

[0071] The information acquisition module 210 is configured to obtain a test sample set and screen the test sample set to obtain typical wind speed daily samples; the test sample set includes at least one wind speed daily sample;

[0072] The prediction result module 220 is configured to input the typical wind speed daily samples into multiple prediction models respectively to output wind speed prediction results;

[0073] The statistic determination module 230 is configured to determine a statistic between the prediction models based on the wind speed prediction results and a preset asymmetric threshold loss function;

[0074] The test module 240 is configured to construct a test based on the asymmetric threshold loss function to determine the probability value of the test;

[0075] The performance analysis module 250 is configured to determine the performance of the prediction model for the typical wind speed daily samples based on the statistic and the probability value.

[0076] The technical solution provided by the embodiments of the present disclosure solves the problems that the existing evaluation methods are insufficient to evaluate the prediction ability of the model and whether there are significant differences, improves the meteorological intelligent early warning ability for substations, and provides technical support for the reliable operation of substations.

[0077] Further, the information acquisition module 210 can be used for:

[0078] The typical wind speed daily samples include at least one of the following: the wind speed day with the maximum wind speed; the wind speed day with the largest wind speed change and the wind speed day with the smallest wind speed change.

[0079] Furthermore, the information acquisition module 210 can be used for:

[0080] Respectively determine the mean and variance of each wind speed daily sample, take the wind speed daily sample with the highest mean as the wind speed day with the maximum wind speed, take the wind speed daily sample with the largest variance as the wind speed day with the largest wind speed change, and take the wind speed daily sample with the smallest variance as the wind speed day with the smallest wind speed change.

[0081] Furthermore, the statistic determination module 230 can be used for:

[0082] Calculate the prediction error of each prediction model according to the wind speed prediction result and the actual wind speed corresponding to the wind speed daily sample;

[0083] Determine the error difference between the prediction errors of each prediction model and determine the average value and standard deviation of the error difference;

[0084] Determine the statistic between the models according to the average value and the standard deviation.

[0085] Furthermore, the performance analysis module 250 can be used for:

[0086] Compare the probability value with a preset probability threshold. If it is less than the probability threshold, there is a performance difference between the prediction models for the typical wind speed daily sample; if it is greater than or equal to the probability threshold, the performance difference between the prediction models for the typical wind speed daily sample is small;

[0087] When there is a performance difference, determine the performance of the prediction model for the typical wind speed daily sample according to the statistic.

[0088] Furthermore, the statistic determination module 230 can be used for:

[0089] The asymmetric threshold loss function is as follows:

[0090]

[0091] Where L aT is the loss function, is the model prediction value, y t i is the true value, e i,t is the prediction error of the prediction model, l aT,iis an asymmetric adjustment parameter, p1, p2, q1, and q2 are positive integer power parameters respectively; a1, a2, b1, and b2 are asymmetric index parameters respectively, T1 is a positive error threshold, and T2 is a negative error threshold.

[0092] Further, the prediction result module 220 can be used for: The prediction model includes: a prediction model based on a convolutional neural network and a long short-term memory network; a prediction model based on a support vector machine, and a prediction model based on a backpropagation model.

[0093] The above device can execute the methods provided in all the foregoing embodiments of the present invention, and has corresponding functional modules and beneficial effects for executing the above methods. Technical details not described in detail in this embodiment can be found in the methods provided in all the foregoing embodiments of the present invention.

[0094] Figure 4 The structural schematic diagram of the electronic device 10 that can be used to implement the embodiments of the present invention is given. The electronic device is intended to represent various forms of digital computers, such as, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, personal digital processors, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0095] As Figure 4 shown, the electronic device 10 includes at least one processor 11, and a memory communicatively connected to at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc. Among them, the memory stores a computer program executable by at least one processor, and the processor 11 can execute various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0096] Multiple components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disc, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0097] The processor 11 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the model prediction performance evaluation method.

[0098] In some embodiments, the model prediction performance evaluation method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the model prediction performance evaluation method described above can be executed. Alternatively, in other embodiments, the processor 11 can be configured to execute the model prediction performance evaluation method by any other suitable means (e.g., by means of firmware).

[0099] The various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special or general programmable processor, and can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0100] A computer program for implementing the method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowchart and / or block diagram to be implemented. The computer programs can be executed entirely on the machine, partially on the machine, as a stand-alone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0101] In the context of the present invention, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0102] In order to provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the electronic device. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0103] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected to each other by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), blockchain network, and the Internet.

[0104] The computing system can include a client and a server. The client and the server are generally remote from each other and typically interact via a communication network. The client-server relationship is created by computer programs running on respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system, solving the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS services.

[0105] It should be understood that various forms of the processes shown above can be used, steps can be reordered, added, or deleted. For example, the steps recited in the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved, and no limitation is made herein.

[0106] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for evaluating the performance of model prediction, characterized in that, The method includes: Obtaining a test sample set and screening the test sample set to obtain typical wind speed daily samples; the test sample set includes at least one wind speed daily sample; Inputting the typical wind speed daily samples into multiple prediction models respectively, and outputting wind speed prediction results; Determining a statistic between the prediction models based on the wind speed prediction results and a preset asymmetric threshold loss function; Constructing a test based on the asymmetric threshold loss function and determining the probability value of the test; Determining the performance of the prediction models for the typical wind speed daily samples based on the statistic and the probability value.

2. The method according to claim 1, characterized in that The typical wind speed daily samples include at least one of the following: the wind speed daily with the maximum wind speed; the wind speed daily with the largest wind speed change and the wind speed daily with the smallest wind speed change. Correspondingly, the step of obtaining a test sample set and screening the test sample set to obtain typical wind speed daily samples includes: Respectively determining the mean and variance of each wind speed daily sample, taking the wind speed daily sample with the highest mean as the wind speed daily with the maximum wind speed, taking the wind speed daily sample with the largest variance as the wind speed daily with the largest wind speed change, and taking the wind speed daily sample with the smallest variance as the wind speed daily with the smallest wind speed change.

3. The method according to claim 1, wherein The step of determining a statistic between the models based on the wind speed prediction results and a preset asymmetric threshold loss function includes: Calculating the prediction error of each prediction model according to the wind speed prediction results and the actual wind speed in the corresponding wind speed daily sample; Determining the error difference between the prediction errors of each prediction model and determining the average value and standard deviation of the error difference; Determining the statistic between the models according to the average value and the standard deviation.

4. The method according to claim 1, characterized in that The step of determining the performance of the prediction models for the typical wind speed daily samples based on the statistic and the probability value includes: Comparing the probability value with a preset probability threshold. If it is less than the probability threshold, there is a performance difference between the prediction models for the typical wind speed daily samples; if it is greater than or equal to the probability threshold, the performance difference between the prediction models for the typical wind speed daily samples is small; When there is a performance difference, determining the performance of the prediction models for the typical wind speed daily samples according to the statistic.

5. The method according to claim 1, characterized in that, The asymmetric threshold loss function is as follows: Among them, L aT is the loss function, is the model prediction value, is the true value, and e i,t is the prediction error of the prediction model, and l aT,i is the asymmetric adjustment parameter, p1, p2, q1, q2 are positive integer power parameters; a1, a2, b1, b2 are asymmetric index parameters, T1 is the positive error threshold, and T2 is the negative error threshold.

6. The method according to claim 1, characterized in that The prediction models include: a prediction model based on a convolutional neural network and a long short-term memory network; a prediction model based on a support vector machine and a prediction model based on a backpropagation model.

7. A model prediction performance evaluation device, characterized in that, It includes: An information acquisition module, configured to obtain a test sample set and screen the test sample set to obtain typical wind speed daily samples; The test sample set includes at least one wind speed daily sample; A prediction result module, configured to input the typical wind speed daily samples into multiple prediction models respectively and output wind speed prediction results; A statistic determination module, configured to determine a statistic between the prediction models based on the wind speed prediction results and a preset asymmetric threshold loss function; A test module, configured to construct a test based on the asymmetric threshold loss function and determine the probability value of the test; A performance analysis module, configured to determine the performance of the prediction models for the typical wind speed daily samples based on the statistic and the probability value.

8. An electronic device, characterized in that, The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and when the computer program is executed by the at least one processor, the at least one processor is enabled to execute the model prediction performance evaluation method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a processor to implement the model prediction performance evaluation method according to any one of claims 1-6 when executed.

10. A computer program product, characterized in that, The computer program product includes a computer program which, when executed by a processor, implements the model prediction performance evaluation method according to any one of claims 1-6.

Images