Performance prediction model accuracy evaluation method and device and electronic equipment

Abstract

The invention discloses a performance prediction model accuracy evaluation method and device and electronic equipment. The specific implementation scheme comprises the steps of obtaining an initial data set; preprocessing the initial data set to obtain a synthetic data set; inputting the training synthesis data into the tested model, and outputting performance prediction data; and performing comparative analysis and deviation calculation on the performance prediction data and the test synthesis data to obtain an evaluation result of the performance prediction accuracy of the tested model. When the model accuracy is evaluated, the problem of insufficient data is solved, the operation complexity is reduced, the evaluation accuracy is improved, by calculating the evaluation result of the performance prediction accuracy of the tested model, the performance prediction accuracy of the tested model is effectively verified, the practicability of the tested model is enhanced, and the evaluation accuracy is improved. And the initial data set obtained by the method can be repeatedly used, so that the use efficiency of the data is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and particularly to a method, device, and electronic device for evaluating the accuracy of a performance prediction model. Background Art

[0002] The quality of the performance of an optical layer transmission system has always been an important evaluation index for the operation, maintenance, and management of the entire life cycle of an optical network. In the past, it mainly relied on manual methods with the help of auxiliary tools and optical network management and control systems for the design, monitoring, and optimization of optical layer transmission performance, consuming a large amount of human and time costs and having low work efficiency. With the continuous improvement of the intelligent level of optical networks, new technologies such as artificial intelligence (AI), big data, and digital twins are introduced in optical layer operation, and an AI model is used to realize the simulation, analysis, prediction, and optimization of the performance of an optical transmission system. On the one hand, it can effectively improve the intelligent operation and management efficiency of the entire life cycle of an optical network, reduce a large amount of manual work, and improve work efficiency and accuracy; on the other hand, it promotes the transformation from the original passive operation and maintenance mode of dealing with faults after the fact to the active operation and maintenance mode of pre-performance prediction and optimization. Therefore, the accuracy evaluation result of an AI data-driven optical network transmission performance prediction model provides an important reference basis for the application and popularization of such AI models in optical networks.

[0003] Regarding the accuracy evaluation of a data-driven AI optical transmission performance prediction model (hereinafter referred to as the performance prediction model), the traditional method is to collect in-service network data or actual physical device data collected in a laboratory environment as the main sources of the data set to train and verify the accuracy of such AI models.

[0004] However, there are common problems in the evaluation of the accuracy of AI models, including that the existing technology consumes a large amount of human and time costs during accuracy evaluation, with extremely low efficiency, and the existing accuracy evaluation means mainly rely on offline tools + manual assistance. The statistical analysis of a large amount of data requires manual assistance to complete, and the accuracy of the results is also difficult to guarantee. Summary of the Invention

[0005] The present invention provides a method, device, and electronic device for evaluating the accuracy of a performance prediction model to solve the problem of lack of effective performance data during model accuracy evaluation, reduce operation complexity, and improve the evaluation accuracy.

[0006] According to one aspect of the present invention, a method for evaluating the accuracy of a performance prediction model is provided, including:

[0007] Obtaining an initial data set, where the initial data set includes data indicating the performance of a transmission system, and the performance of the transmission system is predicted by the model to be tested;

[0008] Preprocess the initial data set to obtain a synthetic data set, where the synthetic data set includes training synthetic data and test synthetic data;

[0009] Input the training synthetic data into the model under test to output performance prediction data, where the performance prediction data includes prediction values for predicting the performance of the transmission system;

[0010] Perform comparative analysis and deviation calculation on the performance prediction data and the test synthetic data to obtain an evaluation result of the performance prediction accuracy of the model under test.

[0011] According to another aspect of the present invention, there is provided an apparatus for evaluating the accuracy of a performance prediction model, including:

[0012] An acquisition module for acquiring an initial data set, where the initial data set includes data indicating the performance of a transmission system, and the prediction of the performance of the transmission system is implemented by a model under test;

[0013] A preprocessing module for preprocessing the initial data set to obtain a synthetic data set, where the synthetic data set includes training synthetic data and test synthetic data;

[0014] An output module for inputting the training synthetic data into the model under test to output performance prediction data, where the performance prediction data includes prediction values for predicting the performance of the transmission system;

[0015] An analysis module for performing comparative analysis and deviation calculation on the performance prediction data and the test synthetic data to obtain an evaluation result of the performance prediction accuracy of the model under test.

[0016] According to another aspect of the present invention, there is provided an electronic device, where the electronic device includes:

[0017] At least one processor; and

[0018] A memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program executable by the at least one processor, and when the computer program is executed by the at least one processor, the at least one processor is enabled to execute the method for evaluating the accuracy of a performance prediction model according to any embodiment of the present invention.

[0020] According to another aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions for causing a processor to execute the method for evaluating the accuracy of a performance prediction model according to any embodiment of the present invention when executed.

[0021] According to another aspect of the present invention, there is provided a computer program product, which includes a computer program that, when executed by a processor, implements the method for evaluating the accuracy of the performance prediction model according to any embodiment of the present invention.

[0022] In the technical solution of the embodiment of the present invention, by obtaining an initial data set, the initial data set includes data indicating the performance of a transmission system, and the prediction of the transmission system performance is implemented by a model under test; preprocessing the initial data set to obtain a synthetic data set, the synthetic data set includes training synthetic data and test synthetic data; inputting the training synthetic data into the model under test, and outputting performance prediction data, the performance prediction data includes predicted values for predicting the performance of the transmission system; performing comparative analysis and deviation calculation on the performance prediction data and the test synthetic data to obtain an evaluation result of the performance prediction accuracy of the model under test. When evaluating the model accuracy, the problem of insufficient performance data is solved, the operation complexity is reduced, and the evaluation accuracy is improved. By calculating the evaluation result of the performance prediction accuracy of the model under test, the accuracy of the performance prediction of the model under test is effectively verified, the practicability of the model under test is enhanced, and the initial data set obtained by the present invention can be reused, improving the data usage efficiency.

[0023] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings without creative efforts based on these drawings.

[0025] Figure 1 is a flowchart of a method for evaluating the accuracy of a performance prediction model according to Embodiment 1 of the present invention;

[0026] Figure 2 is a schematic diagram of the evaluation process of a method for evaluating the accuracy of a performance prediction model according to Embodiment 1 of the present invention;

[0027] Figure 3 is a flowchart of a method for constructing an initial data set according to Embodiment 2 of the present invention;

[0028] Figure 4 is a schematic structural diagram of a device for evaluating the accuracy of a performance prediction model according to Embodiment 3 of the present invention;

[0029] Figure 5 It is a block diagram of an electronic device provided according to Embodiment 4 of the present invention. Detailed implementation manners

[0030] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data used in appropriate cases can be interchanged so that the embodiments of the present invention described here can be implemented in an order other than those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0032] Embodiment 1

[0033] Figure 1 This is a flowchart of a method for evaluating the accuracy of a performance prediction model provided in Embodiment 1 of the present invention. This embodiment is applicable to the situation of evaluating the performance prediction accuracy of a model to be measured. This method can be executed by a device for evaluating the accuracy of a performance prediction model. The device for evaluating the accuracy of a performance prediction model can be implemented in the form of hardware and / or software. The device for evaluating the accuracy of a performance prediction model can be configured in an electronic device, and the electronic device can be a mobile terminal, a PC or a server, etc. As Figure 1 shown, the method includes:

[0034] S110. Obtain an initial data set, where the initial data set includes data indicating the performance of a transmission system, and the prediction of the transmission system performance is implemented by the model to be measured.

[0035] In this embodiment, the initial data set can be understood as a set composed of data indicating the performance of the transmission system. The data in the initial data set can be directly collected data or processed data. The transmission system can be understood as a system for transmitting data, signals, etc. The transmission system can be an optical network transmission system for transmitting optical signals. The performance of the transmission system can be understood as an index for measuring the efficiency, reliability, and practicality of the transmission system. The performance of the transmission system can be the change in the optical power, OSNR, dispersion, etc. of the transmission system. The model under test can predict the performance of the transmission system and output the predicted value of the transmission system performance.

[0036] Specifically, determine the model under test that needs to be evaluated for the accuracy of performance prediction. This model under test realizes the performance prediction of the transmission system. Collect the data indicating the performance of the transmission system output by the transmission system, judge this data, determine the acquisition method of the initial data set, and thus obtain the initial data set.

[0037] Exemplarily, for an optical network transmission system, the in-network data of the optical network transmission system or the actual physical device data collected in a laboratory environment can be collected. Figure 2 It is a schematic diagram of the evaluation process for the accuracy of a performance prediction model provided in Embodiment 1 of the present invention. As Figure 2 , the initial data set can be obtained through a data generator.

[0038] S120. Preprocess the initial data set to obtain a synthetic data set, where the synthetic data set includes training synthetic data and test synthetic data.

[0039] In this embodiment, the synthetic data set can be understood as being obtained by preprocessing the initial data set. The synthetic data set is a data set suitable for the model under test. The test synthetic data can be understood as a set composed of the data with a later time series in the synthetic data set. The training synthetic data can be understood as a set composed of the data with an earlier time series in the synthetic data set.

[0040] Specifically, according to the data required for the input of the model under test, preprocess the synthetic data set so that the data in the synthetic data set can be input into the model under test. Proportionally, the data with an earlier time series in the synthetic data set is used as the training synthetic data, and the data with a later time series in the synthetic data set is used as the test synthetic data. The training synthetic data is input into the model under test for processing.

[0041] Exemplarily, the initial data set can be preprocessed through a data processor to generate a synthetic data set adapted to the model under test. As Figure 2 shown, the initial data set is processed through a data processor to obtain a synthetic data set.

[0042] S130. Input the training synthetic data into the model under test, and output performance prediction data, where the performance prediction data includes prediction values for predicting the performance of the transmission system.

[0043] In this embodiment, the performance prediction data can be understood as the data generated by the model under test, and this data can represent the prediction result of the transmission system performance.

[0044] Specifically, input the training synthetic data in the synthetic dataset into the model under test to complete the training, verification, and testing of the model under test, and the model under test outputs performance prediction data. The performance prediction data is the prediction value of the transmission system performance.

[0045] Exemplarily, as Figure 2 shown, the model under test can be the AI model under test. Input the training synthetic data in the synthetic dataset into the AI model under test to obtain performance prediction data.

[0046] S140. Conduct comparative analysis and deviation calculation on the performance prediction data and the test synthetic data to obtain the evaluation result of the performance prediction accuracy of the model under test.

[0047] Specifically, conduct comparative analysis and deviation calculation on the performance prediction data output by the model under test and the test synthetic data in the synthetic dataset. At this time, the performance prediction data is the performance prediction data output after the training synthetic data passes through the model under test, and it is the data in the same time series as the test synthetic data. Therefore, by conducting comparative analysis and deviation calculation on the performance prediction data and the test synthetic data, the evaluation result of the performance prediction accuracy of the model under test can be obtained. If the analysis result of the performance prediction data and the test synthetic data indicates that the deviation between the two is smaller, it can be considered that the performance prediction accuracy of the model under test is higher.

[0048] Exemplarily, as Figure 2 shown, analyze the performance prediction data and the test synthetic data through a data statistical analysis tool to output the evaluation result of the performance prediction accuracy.

[0049] The technical solution of the embodiment of the present invention is as follows: an initial data set is obtained, where the initial data set includes data indicating the performance of the transmission system, and the prediction of the performance of the transmission system is implemented by the model under test; the initial data set is preprocessed to obtain a synthetic data set, where the synthetic data set includes training synthetic data and test synthetic data; the training synthetic data is input into the model under test, and performance prediction data is output, where the performance prediction data includes a predicted value for predicting the performance of the transmission system; the performance prediction data and the test synthetic data are compared and analyzed and deviation is calculated to obtain an evaluation result of the performance prediction accuracy of the model under test. When evaluating the accuracy of the performance prediction model, the problem of insufficient data set is solved, the operation complexity is reduced, and the evaluation accuracy is improved. By calculating the evaluation result of the performance prediction accuracy of the model under test, the accuracy of the performance prediction of the model under test is effectively verified, the practicability of the model under test is enhanced, and the initial data set obtained by the present invention can be reused, improving the data usage efficiency.

[0050] Based on the above embodiment, a variant embodiment of the above embodiment is proposed. Here, it should be noted that for the sake of brief description, only the differences from the above embodiment are described in the variant embodiment.

[0051] In one embodiment, the obtaining of the initial data set includes:

[0052] Obtain a raw data set, where the raw data set includes data indicating the performance of the transmission system output by the transmission system;

[0053] Evaluate the raw data set to obtain data valid information, where the data valid information includes information indicating whether the data in the raw data set has a data change trend;

[0054] According to the data valid information, obtain the initial data set corresponding to the raw data set.

[0055] In this embodiment, the raw data set can be understood as the transmission performance data directly collected from the output of the transmission system. The data in the raw data set can be actual in-network data or data of physical devices collected in a laboratory environment. The data valid information can be understood as information indicating whether the data in the raw data set is valid, and the validity can be defined as whether the data in the raw data set has a data change trend.

[0056] Specifically, collect the transmission performance data output by the transmission system to form an original data set. Evaluate the data in the original data set to determine whether the data in the original data set has a data change trend. The data change trend can be a gradual decrease, a gradual increase, a periodic change, a sudden increase or decrease, etc. Use whether there is a data change trend as the data valid information of the original data set, and determine the acquisition method of the initial data set corresponding to the original data set according to the data valid information, and obtain the initial data set accordingly.

[0057] Exemplarily, from the existing network or the actual physical system, collect the original data set formed by the transmission performance data of the optical network transmission system. The transmission performance data can be performance data such as OMS optical power spectrum, OCh OSNR, end-to-end BER, nonlinearity, delay, SOP, CD, and PMD. Determine whether the data in the original data set has a data change trend. Only the data with a data change trend can be used for the evaluation of the accuracy of the performance prediction model. For the data without a data change trend, the original data set can be expanded through data enhancement technology to obtain an initial data set that can be used for the evaluation of the accuracy of the performance prediction model.

[0058] Optionally, the obtaining of the initial data set corresponding to the original data set according to the data valid information includes:

[0059] Judge whether the data valid information indicates that the data in the original data set has a data change trend;

[0060] If so, use the original data set as the initial data set;

[0061] If not, expand the original data set to construct an initial data set.

[0062] Specifically, judge whether the data in the original data set has a data change trend. If it has, the original data set can be directly used as the initial data set for subsequent operations. Otherwise, the data in the original data set cannot meet the requirements of model performance evaluation, and the original data set needs to be expanded. The expansion technology can be data enhancement technology.

[0063] Exemplarily, as Figure 2 shown, load and evaluate the original data set. For the data with a data change trend, directly use the original data set as the initial data set. For the data without a data change trend, input the original data set into the data generator, and the original data set can be expanded through the function generator and the optical network simulator to obtain the initial data set.

[0064] In one embodiment, the preprocessing of the initial data set to obtain a synthetic data set includes:

[0065] Obtain a data format that meets the input requirements of the model under test;

[0066] Adapt the data in the initial dataset according to the data format to obtain an adapted dataset;

[0067] Preprocess the data in the adapted dataset to obtain a synthetic dataset.

[0068] In this embodiment, the data format can be understood as data that meets the input requirements of the model under test, the required data format, and the data format can include information such as the range of the data. The adapted dataset can be understood as a set composed of the data obtained by adapting the data in the initial dataset according to the data format.

[0069] Specifically, obtain the data format required for the data to meet the training, validation, and testing requirements of the model under test, and adapt the data in the initial dataset according to the data format, which can be to normalize the data in terms of format range, etc., and use the set composed of the adapted data as the adapted dataset. Then perform preprocessing operations on the data in the adapted dataset to obtain a synthetic dataset.

[0070] Exemplarily, as Figure 2 shown, the operation of preprocessing the initial dataset to obtain a synthetic dataset can be completed in a data processor, and the data processor includes two modules: data adaptation and data preprocessing. Through data adaptation, the mapping between the initial dataset and the model under test is completed to obtain an adapted dataset. The data in the adapted dataset is preprocessed through data preprocessing to obtain a synthetic dataset, which can meet the requirements of training, validation, and testing of the model under test.

[0071] In one embodiment, the comparative analysis and deviation calculation of the performance prediction data and the test synthetic data to obtain the evaluation result of the performance prediction accuracy of the model under test includes:

[0072] Input the performance prediction data and the test synthetic data into an evaluation system, the evaluation system is connected to the model under test, and the evaluation system includes a system with data analysis functions;

[0073] The evaluation system performs comparative analysis and deviation calculation on the performance prediction data and the test synthetic data, and outputs the evaluation result of the performance prediction accuracy.

[0074] In this embodiment, the evaluation system can be understood as a system with data statistics and analysis functions, which can realize the evaluation of the performance prediction accuracy of the model under test.

[0075] Exemplarily, as Figure 2As shown, the evaluation system can be regarded as an evaluation subsystem. The performance prediction data and test synthesis data are input into the evaluation system, and statistical analysis is carried out using data statistical analysis tools. The evaluation system can automatically output the performance evaluation results. Then, the model under test can be optimized according to the evaluation results.

[0076] Embodiment 2

[0077] Figure 3 The figure is a flowchart of a method for constructing an initial data set provided in Embodiment 2 of the present invention. This embodiment expands on the construction operation of the initial data set in the above embodiment. As Figure 3 shown, the method includes:

[0078] S210. Determine the size of the initial data set according to the data characteristics of the original data set output by the transmission system, where the data characteristics include the variation range and quantity of the data in the original data set.

[0079] Specifically, the original data set is loaded and evaluated to determine data characteristics such as the data variation range and quantity of the original data set. The data characteristics may also include the number of data points in the original data set, and accordingly, the size of the initial data set to be constructed is determined.

[0080] S220. Set a data change trend for the initial data set according to the performance of the transmission system, and determine the basis function for the change of specific parameters of the transmission system.

[0081] Exemplarily, as Figure 2 shown, the original data set is input into a data generator, which may include a function generator, an optical network simulator, etc. According to the performance characteristics of the transmission system, a data change trend is set for the initial data set to be constructed. And a suitable basis function and the parameters of the basis function are selected through the function generator.

[0082] S230. Introduce data random characteristics into the initial data set.

[0083] In this embodiment, the data random characteristics can be understood as the uncertainty or unpredictability of the data. Introducing the data random characteristics can enhance the authenticity of the initial data set.

[0084] Specifically, data random characteristics are introduced into the initial data set to be constructed. The data random characteristics may be random noise, such as white noise, etc.

[0085] S240. Output target performance data through simulation by the simulator according to the original data set and the basis function.

[0086] In this embodiment, the target performance data can be understood as the initial data in the initial data set to be constructed.

[0087] Specifically, the basis functions selected and generated by the data generator change, and the changed data of specific system parameters in the transmission system are input into the emulator as input data, and the target performance data is output as the initial data in the initial data set.

[0088] Optionally, the simulation output of the target performance data according to the original data set and the basis function by the emulator includes:

[0089] Determine the system parameters of the transmission system;

[0090] Set the parameters of the emulator to the system parameters of the transmission system;

[0091] Construct the original data set into an input original data set through the basis function;

[0092] Input the data in the input original data set into the emulator, and simulate and output the target performance data.

[0093] Specifically, to ensure the rationality of the data in the initial data set, the configuration of the emulator needs to be consistent with the actual transmission system and parameter settings, and the parameters of the emulator can be set to the system parameters of the transmission system. Input the data in the original data set into the emulator to obtain the target performance data.

[0094] S250. Superimpose the data random characteristics on the target performance data to obtain the initial performance data.

[0095] Specifically, first superimpose the data random characteristics on the target performance data, which can be adding random noise to the target performance data to obtain the initial performance data. The initial performance data is the data generated by the data generator and can be put into the initial data set.

[0096] S260. Use the initial performance data as the data in the initial data set, and combine the size of the initial data set and the data change trend of the initial data set to construct the initial data set.

[0097] Specifically, according to the size and data change trend of the initial data set, operate on the initial performance data to make it meet the above two conditions, and finally complete the construction of the initial data set.

[0098] The technical solution of the embodiment of the present invention determines the size of the initial data set according to the data characteristics of the original data set output by the transmission system, where the data characteristics include the change range and quantity of the data in the original data set; sets the data change trend for the initial data set according to the performance of the transmission system, and determines the basis function for the change of specific parameters of the transmission system; introduces data random characteristics to the initial data set; outputs target performance data through simulation by a simulator according to the original data set and the basis function; superimposes the data random characteristics on the target performance data to obtain initial performance data; and constructs an initial data set with the initial performance data as the data in the initial data set, in combination with the size of the initial data set and the data change trend of the initial data set. By elaborating on the construction operation of the initial data set, the problems of data set shortage and "data wall" in the implementation process of the performance prediction model accuracy evaluation method are solved. The present invention can generate data that meets specific requirements according to the performance characteristics of the transmission system, can perform data expansion and increase as needed, and can meet the training and testing requirements of the model under test.

[0099] Embodiment III

[0100] Figure 4 FIG. is a schematic structural diagram of an apparatus for evaluating the accuracy of a performance prediction model provided in Embodiment III of the present invention. As Figure 4 shown, the apparatus includes:

[0101] An acquisition module 310, configured to acquire an initial data set, where the initial data set includes data indicating the performance of a transmission system, and the prediction of the transmission system performance is implemented by a model under test;

[0102] A preprocessing module 320, configured to preprocess the initial data set to obtain a synthetic data set, where the synthetic data set includes training synthetic data and test synthetic data;

[0103] An output module 330, configured to input the training synthetic data into the model under test and output performance prediction data, where the performance prediction data includes a predicted value for predicting the performance of the transmission system;

[0104] An analysis module 340, configured to perform comparative analysis and deviation calculation on the performance prediction data and the test synthetic data to obtain an evaluation result of the performance prediction accuracy of the model under test.

[0105] The evaluation device for the accuracy of the performance prediction model provided by the embodiments of the present invention obtains an initial data set through an acquisition module. The initial data set includes data indicating the performance of a transmission system, and the prediction of the transmission system performance is implemented by a model under test. The initial data set is preprocessed by a preprocessing module to obtain a synthetic data set, which includes training synthetic data and test synthetic data. The training synthetic data is input into the model under test through an output module to output performance prediction data, which includes prediction values for predicting the performance of the transmission system. The performance prediction data and the test synthetic data are compared and analyzed, and deviation calculation is performed through an analysis module to obtain an evaluation result of the performance prediction accuracy of the model under test. Through the mutual cooperation of each module, when evaluating the model accuracy, the problem of insufficient data is solved, the operation complexity is reduced, and the evaluation accuracy is improved. By calculating the evaluation result of the performance prediction accuracy of the model under test, the accuracy of the performance prediction of the model under test is effectively verified, the practicability of the model under test is enhanced, and the initial data set obtained by the present invention can be reused, improving the data usage efficiency.

[0106] In one embodiment, the acquisition module 310 includes:

[0107] A first acquisition unit for acquiring an original data set, where the original data set includes data indicating the performance of the transmission system output by the transmission system;

[0108] An evaluation unit for evaluating the original data set to obtain data valid information, where the data valid information includes information indicating whether the data in the original data set has a data change trend;

[0109] A second acquisition unit for acquiring the initial data set corresponding to the original data set according to the data valid information.

[0110] In one embodiment, the second acquisition unit is specifically configured to:

[0111] Determine whether the data valid information indicates that the data in the original data set has a data change trend;

[0112] If so, use the original data set as the initial data set;

[0113] If not, expand the original data set to construct an initial data set.

[0114] In one embodiment, the model performance evaluation device further includes a construction module for the initial data set, including:

[0115] A determination unit, configured to determine the size of the initial data set according to the data characteristics of the original data set output by the transmission system, where the data characteristics include the variation range and quantity of the data in the original data set;

[0116] A setting unit, configured to set a data variation trend for the initial data set according to the performance of the transmission system, and determine a basis function for the change of specific parameters of the transmission system;

[0117] An introduction unit, configured to introduce data random characteristics to the initial data set;

[0118] A simulation unit, configured to output target performance data through simulation by a simulator according to the original data set and the basis function;

[0119] A superimposing unit, configured to superimpose the data random characteristics onto the target performance data to obtain initial performance data;

[0120] A construction unit, configured to use the initial performance data as the data in the initial data set, and construct an initial data set in combination with the size of the initial data set and the data variation trend of the initial data set.

[0121] In one embodiment, the simulation unit is specifically configured to:

[0122] Determine the system parameters of the transmission system;

[0123] Set the parameters of the simulator to the system parameters of the transmission system;

[0124] Construct the original data set into an input original data set through the basis function;

[0125] Input the data in the input original data set into the simulator, and simulate and output target performance data.

[0126] In one embodiment, the preprocessing module 320 is specifically configured to:

[0127] Obtain a data format that meets the input requirements of the device under test;

[0128] Adapt the data in the initial data set according to the data format to obtain an adapted data set;

[0129] Perform preprocessing on the data in the adapted data set to obtain a synthetic data set.

[0130] In one embodiment, the analysis module 340 is specifically configured to:

[0131] Input the performance prediction data and the test synthesis data into an evaluation system, which is connected to the model under test. The evaluation system includes a system with data analysis capabilities;

[0132] The evaluation system performs comparative analysis and deviation calculation on the performance prediction data and the test synthesis data, and outputs an evaluation result of the performance prediction accuracy.

[0133] The evaluation device for the accuracy of the performance prediction model provided by the embodiments of the present invention can execute the evaluation method for the accuracy of the performance prediction model provided by any embodiment of the present invention. Through the mutual cooperation and collaborative work among the modules, the accuracy evaluation of the performance prediction model is completed, and it has the corresponding functional modules and beneficial effects for executing the method.

[0134] Embodiment Four

[0135] According to an embodiment of the present invention, the present invention also provides an electronic device, a computer-readable storage medium, and a computer program product.

[0136] Figure 5 FIG. is a block diagram of an electronic device provided according to Embodiment Four of the present invention. This electronic device can implement the evaluation method for the accuracy of the performance prediction model described in the embodiments of the present invention. The electronic device is intended to represent various forms of digital computers, such as, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, personal digital processors, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0137] As Figure 5 shown, the electronic device 410 includes at least one processor 411, and a memory communicatively connected to at least one processor 411, such as a read-only memory (ROM) 412, a random access memory (RAM) 413, etc. Among them, the memory stores a computer program executable by at least one processor. The processor 411 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 412 or the computer program loaded from the storage unit 418 into the random access memory (RAM) 413. In the RAM 413, various programs and data required for the operation of the electronic device 410 can also be stored. The processor 411, the ROM 412, and the RAM 413 are connected to each other through a bus 414. The input / output (I / O) interface 415 is also connected to the bus 414.

[0138] Multiple components in the electronic device are connected to the I / O interface 415, including: an input unit 416, such as a keyboard, a mouse, etc.; an output unit 417, such as various types of displays, speakers, etc.; a storage unit 418, such as a magnetic disk, an optical disc, etc.; and a communication unit 419, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 419 allows the electronic device to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0139] The processor 411 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 411 include but are not limited to a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 411 executes the various methods and processes described above, such as the method for evaluating the accuracy of the performance prediction model.

[0140] In some embodiments, the method for evaluating the accuracy of the performance prediction model can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as the storage unit 418. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 410 via the ROM 412 and / or the communication unit 419. When the computer program is loaded into the RAM 413 and executed by the processor 411, one or more steps of the method for evaluating the accuracy of the performance prediction model described above can be executed. Alternatively, in other embodiments, the processor 411 can be configured to execute the method for evaluating the accuracy of the performance prediction model by any other suitable means (e.g., by means of firmware).

[0141] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, the programmable processor can be a special or general programmable processor, can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0142] A computer program for implementing the method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer programs are executed by the processor, the functions / operations specified in the flowchart and / or block diagram are implemented. The computer programs can be executed entirely on the machine, partially on the machine, as a stand-alone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0143] In the context of the present invention, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0144] In order to provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the electronic device. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and the input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0145] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected to each other by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), blockchain network, and the Internet.

[0146] A computing system can include a client and a server. The client and the server are generally far from each other and usually interact through a communication network. The client-server relationship is created by computer programs running on respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system and solves the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS services.

[0147] In some embodiments, a computer program product includes a computer program that, when executed by a processor, implements the method for evaluating the accuracy of the performance prediction model provided by the embodiments of the present invention.

[0148] The technical solution of the embodiments of the present invention is directed to a method, apparatus, and electronic device for evaluating the accuracy of a performance prediction model. By obtaining an initial data set; preprocessing the initial data set to obtain a synthetic data set; inputting the training synthetic data into the model under test to output performance prediction data; and performing comparative analysis and deviation calculation on the performance prediction data and the test synthetic data to obtain an evaluation result of the performance prediction accuracy of the model under test. It realizes the solution to the problem of insufficient data during model accuracy evaluation, reduces the operation complexity, and improves the evaluation accuracy. By calculating the evaluation result of the performance prediction accuracy of the model under test, it effectively verifies the accuracy of the performance prediction of the model under test, enhances the practicality of the model under test, and the initial data set obtained by the present invention can be reused, improving the data usage efficiency.

[0149] It should be understood that various forms of the processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved, and no limitation is imposed herein.

[0150] The above specific embodiments do not constitute a limitation to the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for evaluating the accuracy of a performance prediction model, characterized in that: include: Acquiring an initial data set, the initial data set comprising data indicating transmission system performance, the prediction of the transmission system performance being implemented by the model under test; Preprocessing the initial data set to obtain a synthetic data set, wherein the synthetic data set includes training synthetic data and test synthetic data; Inputting the training synthetic data into the model under test, and outputting performance prediction data, wherein the performance prediction data includes a predicted value for predicting the performance of the transmission system; The performance prediction data and the test synthetic data are compared and analyzed and deviations are calculated to obtain an evaluation result of the performance prediction accuracy of the tested model.

2. The method according to claim 1, characterized in that The obtaining of the initial data set comprises: Acquire an original data set, the original data set comprising data output by the transmission system indicating the performance of the transmission system; Evaluating the original data set to obtain data validity information, wherein the data validity information includes information indicating whether the data in the original data set has a data change trend; According to the data validity information, an initial data set corresponding to the original data set is obtained.

3. The method according to claim 2, characterized in that The step of acquiring an initial data set corresponding to the original data set according to the data valid information includes: Determining whether the data validity information indicates that the data in the original data set has a data change trend; If yes, the original data set is used as the initial data set; If not, the original data set is expanded to construct an initial data set.

4. The method according to claim 1, characterized in that The operation of constructing the initial data set includes: Determining the size of the initial data set according to data characteristics of the original data set output by the transmission system, wherein the data characteristics include a variation range and quantity of data in the original data set; According to the performance of the transmission system, setting a data change trend for the initial data set and determining a basis function for the change of a specific parameter of the transmission system; Introducing data random characteristics into the initial data set; Outputting target performance data through simulation by a simulator according to the original data set and the basis function; superimposing the random characteristics of the data onto the target performance data to obtain initial performance data; The initial performance data is used as data in the initial data set, and the initial data set is constructed in combination with the size of the initial data set and the data change trend of the initial data set.

5. The method according to claim 4, characterized in that The step of outputting target performance data through simulation by a simulator according to the original data set and the basis function comprises: determining system parameters of the transmission system; Setting the parameters of the simulator to system parameters of the transmission system; The original data set is constructed as an input original data set through the basis function; The data in the input original data set is input into the simulator, and the simulation outputs the target performance data.

6. The method according to claim 1, characterized in that The preprocessing of the initial data set to obtain a synthetic data set includes: Acquire a data format that meets the input requirements of the model under test; According to the data format, adapting the data in the initial data set to obtain an adapted data set; The data in the adapted data set is preprocessed to obtain a synthetic data set.

7. The method according to claim 1, characterized in that The comparative analysis and deviation calculation of the performance prediction data and the test synthetic data to obtain the evaluation result of the performance prediction accuracy of the tested model includes: Inputting the performance prediction data and the test synthesis data into an evaluation system, wherein the evaluation system is connected to the model under test, and the evaluation system includes a system having a data analysis function; The evaluation system performs comparative analysis and deviation calculation on the performance prediction data and the test synthesis data, and outputs an evaluation result of the performance prediction accuracy.

8. A device for evaluating the accuracy of a performance prediction model, characterized in that: include: An acquisition module, configured to acquire an initial data set, wherein the initial data set includes data indicating the performance of a transmission system, wherein the prediction of the performance of the transmission system is implemented by a model under test; A preprocessing module, used for preprocessing the initial data set to obtain a synthetic data set, wherein the synthetic data set includes training synthetic data and test synthetic data; An output module, used for inputting the training synthetic data into the model under test, and outputting performance prediction data, wherein the performance prediction data includes a prediction value for predicting the performance of the transmission system; The analysis module is used to perform comparative analysis and deviation calculation on the performance prediction data and the test synthesis data to obtain an evaluation result of the performance prediction accuracy of the tested model.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for evaluating the accuracy of a performance prediction model described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for evaluating the accuracy of a performance prediction model according to any one of claims 1 to 7 when executed.

11. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the method for evaluating the accuracy of a performance prediction model according to any one of claims 1 to 7.

Images