Model testing method, device, equipment and storage medium

Abstract

This application provides a model testing method, apparatus, device, and storage medium, which can be applied in the computer field. The model testing method includes: inputting a benchmark dataset for the current time period into a test model and a model under test to obtain multiple first diagnostic results for the test model regarding any first test parameter of multiple first server logs and multiple second diagnostic results for the model under test regarding any first test parameter; obtaining benchmark test results based on the multiple first diagnostic results and multiple second diagnostic results; inputting a baseline test dataset for the current time period into the model under test to obtain multiple third diagnostic results for any second test parameter of multiple second server logs; obtaining baseline test results based on multiple baseline values ​​for any second test parameter in the baseline test sample and multiple third diagnostic results; and determining that the model under test has an anomaly if the benchmark test results and the baseline test results do not meet preset conditions.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a model testing method, apparatus, device, and storage medium. Background Technology

[0002] With the rapid development of information technology, servers are increasingly widely used in various fields, and their stability and reliability are crucial for the normal operation of business. Automated fault diagnosis systems have become one of the key technologies for ensuring server stability and reliability. The diagnostic models used by these systems can quickly diagnose uploaded server logs and output diagnostic conclusions to assist operations engineers in quickly diagnosing and repairing servers. Therefore, the accuracy and reliability of the diagnostic model's results directly affect the diagnosis of the server.

[0003] In related technologies, in order to ensure the reliability of the diagnostic results of the diagnostic model, it is usually necessary to periodically check a large number of logs to ensure that the diagnostic accuracy of the diagnostic model meets the requirements. However, this results in tens of thousands of test parameters that need to be maintained, and the number of test parameters that need to be maintained changes dynamically with the total amount of logs, which greatly increases the maintenance difficulty of the diagnostic model. Summary of the Invention

[0004] In view of the above problems, this application provides a model testing method, apparatus, device, medium and program product.

[0005] According to a first aspect of this application, a model testing method is provided, comprising: inputting a benchmark test dataset for the current time period into a test model and a model under test respectively, to obtain multiple first diagnostic results of the test model for any first test parameter of multiple first server logs and multiple second diagnostic results of the model under test for any first test parameter of the multiple first server logs, wherein the benchmark test dataset includes first server logs from a previous time period and first server logs sampled from newly added first server logs in the current time period at a preset ratio; obtaining a benchmark test result based on the number of identical diagnostic results for the same first server log in the multiple first diagnostic results and the multiple second diagnostic results; inputting a baseline test dataset for the current time period into the model under test to obtain multiple third diagnostic results for any second test parameter of multiple second server logs, wherein the baseline test dataset includes second server logs from a previous time period and second server logs for development requirements in the current time period; obtaining a baseline test result based on the number of multiple baseline values ​​for any second test parameter in the baseline test samples and the number of identical values ​​for the same second server log in the multiple third diagnostic results; and determining that the model under test has an anomaly if the benchmark test result and the baseline test result do not meet preset conditions.

[0006] A second aspect of this application provides a model testing apparatus, comprising: a first input module, a first acquisition module, a second input module, a second acquisition module, and a first determination module. The first input module is configured to input a benchmark dataset for the current time period into a test model and a model under test, respectively, to obtain multiple first diagnostic results of the test model for any first test parameter of multiple first server logs and multiple second diagnostic results of the model under test for any first test parameter of the multiple first server logs. The benchmark dataset includes first server logs from a previous time period and first server logs sampled from newly added first server logs in the current time period at a preset ratio. The first acquisition module is configured to obtain benchmark results based on the number of identical diagnostic results for the same first server log among the multiple first diagnostic results and the multiple second diagnostic results. The second input module is used to input the baseline test dataset for the current time period into the model under test to obtain multiple third diagnostic results for any second test parameter of multiple second server logs, wherein the baseline test dataset includes second server logs from the previous time period and second server logs for development requirements of the current time period; the second obtaining module is used to obtain the baseline test result based on the number of multiple baseline values ​​for any second test parameter in the baseline test sample and the number of identical values ​​for the same second server log in the multiple third diagnostic results; the first determining module is used to determine that the model under test has an anomaly if the benchmark test result and the baseline test result do not meet the preset conditions.

[0007] A third aspect of this application provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0008] A fourth aspect of this application also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0009] The fifth aspect of this application also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0010] According to the model testing method provided in this application, the model under test is tested before release to ensure the diagnostic accuracy of the model before release. Based on this, benchmark testing is introduced. Using a benchmark dataset of first server logs from previous periods and newly added first server logs from the current period sampled at a preset ratio, the diagnostic accuracy and reliability of the model under test can be flexibly and dynamically tested. Baseline testing is also introduced. Using a baseline test dataset of second server logs from previous periods and second server logs used for development requirements in the current period, baseline testing is performed on the model under test to ensure that the model under test meets the development requirements of the current period before release, while also ensuring that the model under test meets historical development requirements. Attached Figure Description

[0011] Figure 1 An application scenario diagram of the model testing method according to an embodiment of this application is shown;

[0012] Figure 2 A flowchart of a model testing method according to an embodiment of this application is shown;

[0013] Figure 3 A flowchart illustrating the process of obtaining benchmark test results according to an embodiment of this application is shown;

[0014] Figure 4 A flowchart illustrating the process of obtaining baseline test results according to an embodiment of this application is shown;

[0015] Figure 5 A schematic diagram illustrating the process of benchmarking and baseline testing of the model under test according to an embodiment of this application is shown;

[0016] Figure 6 A schematic diagram of a benchmark test according to an embodiment of this application is shown;

[0017] Figure 7 A schematic diagram of a baseline test according to an embodiment of this application is shown;

[0018] Figure 8 A structural block diagram of a model testing apparatus according to an embodiment of this application is shown;

[0019] Figure 9 A block diagram of an electronic device suitable for implementing a model testing method according to an embodiment of this application is shown. Detailed Implementation

[0020] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0021] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0022] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0023] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0024] To ensure the reliability of diagnostic model results, it is generally necessary to periodically verify large volumes of logs to ensure that the diagnostic accuracy meets requirements. Taking a diagnostic system with a total of 1 million logs as an example, sampling at 1% of the sample size requires more than 10,000 sample logs each time to verify the reliability of the diagnostic model's results. Furthermore, since the test parameters include multiple parameters such as fault type, fault level, and machine model SN (Serial Number), the total number of test parameters that need to be maintained exceeds tens of thousands, and they dynamically change with the total amount of logs. This significantly increases the difficulty of writing and maintaining test cases for the diagnostic model.

[0025] Furthermore, current verification of diagnostic model reliability primarily occurs after the diagnostic model is released. Leveraging the online and multi-platform advantages of the diagnostic system, it interfaces with various systems, such as work order systems, to verify the accuracy of the diagnostic model's output results. This feedback is then relayed to the model and algorithm developers, enabling them to further optimize the diagnostic code or algorithm. Testing before the diagnostic model's release largely involves manually maintaining a small sample of test cases and corresponding test code for small-batch testing.

[0026] The online integration solution for diagnostic systems can effectively verify the accuracy and reliability of diagnostic models after release. However, it can lead to a delay in the discovery of diagnostic model issues, which may take several months after deployment to identify and optimize problems, posing a potential threat to the reliability of the diagnostic model.

[0027] Besides the lag in problem detection, the integration system also has many pain points. Taking the integration of the diagnostic system and the work order system as an example, there are complex correspondences between the two systems, such as one log corresponding to multiple work orders or multiple logs corresponding to one work order, making differentiation difficult. Furthermore, the repair sources in the work order system and the logs in the diagnostic system may not match. For example, an engineer might replace a component based on a red indicator light on a server component in the field, but the server fault log uploaded to the diagnostic system may not show a corresponding fault record for that component. Therefore, these situations require manual intervention to resolve inaccuracies caused by the integration of the work order system.

[0028] Moreover, the continuous increase in diagnostic parameters also increases maintenance difficulty. As the diagnostic model adapts to various diagnostic systems, the number of output parameters for the diagnostic conclusions of the diagnostic model will continue to increase, ranging from a dozen to dozens. For example, the number of output parameters of the diagnostic model increased from 10 to 31. Initially, it only included basic information such as the model number, serial number, fault level, and faulty component. Subsequently, parameters such as AI-based operation and maintenance accuracy, expert rule accuracy, and component serial numbers were gradually added. However, many parameters do not exist in other systems, such as work order systems, making it impossible to verify them through interface integration.

[0029] Based on the above, existing solutions struggle to accurately maintain the diagnostic model. Therefore, embodiments of this application provide a model testing method to ensure the diagnostic accuracy of the diagnostic model before release.

[0030] Figure 1 An application scenario diagram of the model testing method according to an embodiment of this application is shown.

[0031] like Figure 1As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0032] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0033] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0034] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0035] For example, server 105 can input the benchmark dataset for the current time period into the test model and the model under test, respectively, to obtain multiple first diagnostic results of the test model for any first test parameter of multiple first server logs and multiple second diagnostic results of the model under test for any first test parameter of multiple first server logs. Based on the number of identical diagnostic results for the same first server log in the multiple first diagnostic results and multiple second diagnostic results, the benchmark test result can be obtained. The baseline test dataset for the current time period can be input into the model under test to obtain multiple third diagnostic results for any second test parameter of multiple second server logs. Based on the number of baseline values ​​for any second test parameter in the baseline test sample and the number of identical values ​​for the same second server log in the multiple third diagnostic results, the baseline test result can be obtained. If the benchmark test result and the baseline test result do not meet the preset conditions, it can be determined that the model under test has an anomaly.

[0036] It should be noted that the model testing method provided in this application embodiment can generally be executed by server 105. Correspondingly, the model testing device provided in this application embodiment can generally be located in server 105. The model testing method provided in this application embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the model testing device provided in this application embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0037] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0038] The following will be based on Figure 1 The described scene, through Figures 2-7 The model testing method of the embodiments of this application will be described in detail.

[0039] Figure 2 A flowchart of a model testing method according to an embodiment of this application is shown.

[0040] like Figure 2 As shown, the method 200 includes operations S210 to S250.

[0041] In operation S210, the benchmark test dataset for the current time period is input into the test model and the model under test respectively, to obtain multiple first diagnostic results of the test model for any first test parameter of multiple first server logs and multiple second diagnostic results of the model under test for any first test parameter of multiple first server logs.

[0042] The benchmark dataset includes the first server logs from the previous time period and the first server logs obtained by sampling the newly added first server logs in the current time period at a preset ratio.

[0043] According to embodiments of this application, the model under test can be a diagnostic model used for fault diagnosis of the server. Testing the model under test is to ensure the diagnostic accuracy of the model. The test model can characterize the model used to test the model under test.

[0044] According to an embodiment of this application, since the version of the model under test is iterative, such as releasing a version of the model under test in each cycle, during the testing of the updated version of the model under test in the current cycle, the benchmark test dataset used to test the previously released version of the model under test and the server logs added in the current cycle relative to the previous cycle can be used as the benchmark test dataset for the current version of the model under test.

[0045] In one embodiment, since the log increment is large, the newly added server logs can be sampled according to a preset ratio. The preset ratio is set as needed.

[0046] Specifically, the current time period can be represented as the current cycle. Since issues may arise after the version of the model under test is updated, it is necessary to test the model to ensure its diagnostic accuracy. Within the current time period, there may be multiple tests on the model under test, but the benchmark dataset used for each test remains unchanged within the current time period.

[0047] According to an embodiment of this application, the first server log in the previous time period represents all the first server logs in the benchmark dataset used to test the previous version of the model under test; the newly added first server log in the current time period represents the first server log added in the current time period relative to the time period when the previous version of the model under test was tested.

[0048] For example, with a preset ratio of 1%, if the number of first server logs in the benchmark dataset of the previous period was 10,000, and the number of new logs between the current period and the previous period was 30,000, then the benchmark dataset of the current period includes 1% of the new logs, which is 300 new logs. Therefore, the number of first server logs in the benchmark dataset of the current period is 10,300.

[0049] According to an embodiment of this application, benchmarking the model under test involves testing the model as a whole, that is, ensuring that the model under test as a whole meets certain diagnostic quality. Therefore, the benchmark dataset used for benchmarking is sampled from the total number of new log entries.

[0050] Specifically, by inputting the benchmark dataset into the test model and the model under test respectively, multiple first diagnostic results of the test model for any first test parameter of multiple first server logs and multiple second diagnostic results of the model under test for any first test parameter can be obtained.

[0051] In operation S220, benchmark results are obtained based on the number of identical diagnostic results for the same first server log among multiple first diagnostic results and multiple second diagnostic results.

[0052] According to the embodiments of this application, in order to determine the diagnostic accuracy of the model under test, it is necessary to obtain the benchmark test results based on the number of identical diagnostic results for the same first server log among multiple first diagnostic results and multiple second diagnostic results. That is, the benchmark test results can characterize the diagnostic accuracy of the model under test.

[0053] In one embodiment, performing the above operations S210 and S220 means performing benchmark testing on the model under test.

[0054] In operation S230, the baseline test dataset for the current time period is input into the model under test to obtain multiple third diagnostic results for any second test parameter of multiple second server logs.

[0055] The baseline test dataset includes the second server logs from the previous period and the second server logs from the current period for development needs.

[0056] According to embodiments of this application, baseline testing of the model under test is to test whether the model under test meets the development requirements of the current time period, that is, to ensure that the model under test meets the development requirements of the current time period. Therefore, the baseline test dataset used for baseline testing includes second server logs based on the development requirements for the current time period.

[0057] In one embodiment, the second server logs in the previous period may represent all the second server logs in the baseline test dataset used to test the previous version of the model under test.

[0058] Specifically, by inputting the baseline test dataset into the model under test, multiple third diagnostic results can be obtained for any second test parameter of multiple second server logs.

[0059] In operation S240, the baseline test result is obtained based on the number of baseline values ​​for any second test parameter in the baseline test sample and the number of identical values ​​for the same second server log in multiple third diagnostic results.

[0060] According to the embodiments of this application, in order to determine whether the model under test meets the development requirements, it is necessary to obtain the baseline test result based on the number of multiple baseline values ​​for any second test parameter in the baseline test sample and the number of identical values ​​for the same second server log in multiple third diagnostic results. That is, the baseline test result can characterize whether the model under test meets the development requirements.

[0061] In one embodiment, performing the above operations S230 and S240 is equivalent to performing a baseline test on the model under test.

[0062] Based on the above, baseline testing is performed on the model under test using the baseline test dataset for the current time period. The baseline test results can then be used to determine whether the model meets the development requirements for the current time period. Furthermore, since the baseline test dataset also includes second server logs from previous time periods, the baseline test results can also be used to determine whether the model under test simultaneously meets all previous development requirements, i.e., whether it meets historical development requirements.

[0063] When operating S250, if the benchmark test results and baseline test results do not meet the preset conditions, it is determined that the model under test has an anomaly.

[0064] According to embodiments of this application, the preset conditions are set based on the diagnostic accuracy requirements of the model under test. If the benchmark test results and baseline test results do not meet the preset conditions, i.e., at least one of the baseline test results and baseline test results does not meet the preset conditions, it can be determined that the model under test is abnormal.

[0065] According to an embodiment of this application, if it is determined that the model under test has an anomaly, it can be indicated that the currently updated model under test does not meet the current requirements and needs to be updated again.

[0066] Specifically, the above operations S210 to S250 are performed before the model to be tested is released.

[0067] According to embodiments of this application, before the model under test is released, testing is performed on the model to ensure the diagnostic accuracy of the model before release. Based on this, benchmark testing is introduced. Using a benchmark dataset of first server logs from previous periods and newly added first server logs from the current period sampled at a preset ratio, the diagnostic accuracy and reliability of the model under test can be flexibly and dynamically tested. Baseline testing is also introduced. Using a baseline test dataset of second server logs from previous periods and second server logs used for development requirements in the current period, baseline testing is performed on the model under test to ensure that the model under test meets the development requirements of the current period before release, while also ensuring that the model under test meets historical development requirements.

[0068] According to an embodiment of this application, the above-mentioned model testing method further includes: determining the model to be tested as the model to be released when the benchmark test results and baseline test results meet preset conditions.

[0069] According to an embodiment of this application, if both the benchmark test results and the baseline test results meet the preset conditions during the current time period, it can be indicated that the current model under test meets the current requirements. Therefore, the model under test can be determined as the model to be released.

[0070] In one embodiment, the current model under test meeting the current requirements can specifically mean that the current model under test meets the requirements for diagnostic accuracy and development.

[0071] According to an embodiment of this application, during the current time period, if the benchmark test results and baseline test results of the model under test do not meet the preset conditions, the R&D personnel can update and optimize the model under test based on the benchmark test results, baseline test results, and development requirements. This allows the aforementioned operations S210 to S240 to be executed again after the R&D personnel have completed updating and optimizing the model under test, so that the benchmark test results and baseline test results of the model under test are re-performed until they meet the preset conditions.

[0072] According to the embodiments of this application, based on preset conditions, benchmark tests and baseline tests are performed on the model under test to ensure that the model under test is determined to be the model to be released only when the benchmark test results and baseline test results of the model under test both meet the preset conditions, so as to ensure the diagnostic accuracy of the model under test before release.

[0073] According to embodiments of this disclosure, a benchmark dataset for the current time period is input into a test model and a model under test, respectively, to obtain multiple first diagnostic results of the test model for any first test parameter of multiple first server logs and multiple second diagnostic results of the model under test for any first test parameter of multiple first server logs. This includes: inputting multiple first server logs from the benchmark dataset into the test model to obtain a first diagnostic result for each of the multiple first test parameters of any first server log; and inputting multiple first server logs from the benchmark dataset into the model under test to obtain a second diagnostic result for each of the multiple first test parameters of any first server log.

[0074] According to embodiments of this application, a test model can be used to identify multiple first test parameters in a first server log. Thus, by inputting multiple first server logs from a benchmark dataset into the model under test, the model under test can output a first diagnostic result for each of the multiple first test parameters of any given first server log.

[0075] In one embodiment, by inputting all the first server logs in the benchmark dataset into the model under test, the first diagnostic results of each first server log for multiple first test parameters can be obtained.

[0076] According to an embodiment of this application, the model under test can be used to identify multiple parameters in a first server log. The multiple first test parameters are a part of the multiple parameters in the first server log that the model under test can identify, and the multiple first test parameters are the core parameters that the model under test can identify.

[0077] According to an embodiment of this application, by inputting multiple first server logs from the benchmark dataset into the model under test, diagnostic results for multiple parameters of any one first server log can be obtained. Since the test model identifies multiple first test parameters, and these multiple first test parameters are core parameters, a second diagnostic result for each of the multiple first test parameters of any one first server log can be selected from the output of the model under test.

[0078] According to embodiments of this application, by inputting a benchmark dataset into a test model and then into the model under test, first diagnostic results can be obtained for each of the multiple first test parameters of the test model for any first server log, and second diagnostic results can be obtained for each of the multiple first test parameters of the model under test for any first server log. Therefore, based on multiple first diagnostic results and multiple second diagnostic results, benchmark testing of the model under test can be achieved, thereby facilitating the determination of the diagnostic accuracy of the model under test.

[0079] Figure 3 A flowchart illustrating the benchmark test results according to an embodiment of this application is shown.

[0080] like Figure 3 As shown, the method 300 includes operations S310 to S330.

[0081] According to embodiments of this application, benchmark test results can characterize the diagnostic accuracy of the model under test for each of the multiple first test parameters.

[0082] In operation S310, for any first test parameter in any first server log, the first diagnostic result of any first test parameter is compared with the second diagnostic result to obtain the first comparison result of any first test parameter.

[0083] According to an embodiment of this application, since the diagnostic accuracy of the model under test is tested using a test model, it is necessary to compare the first diagnostic result of the test model for any first test parameter of any first server log with the second diagnostic result of the model under test for the corresponding first test parameter of the corresponding first server log, so as to obtain the first comparison result of the model under test and the test model for the first test parameter.

[0084] According to an embodiment of this application, the benchmark test results of the model under test can be obtained based on the first comparison results of each of the first test parameters of each first server log in the benchmark test dataset.

[0085] In operation S320, for any first test parameter, determine the first number of first server logs whose first comparison result indicates that the first diagnostic result is consistent with the second diagnostic result.

[0086] According to an embodiment of this application, if the first comparison result indicates that the first diagnostic result is consistent with the second diagnostic result, it can be determined that the diagnosis of a certain first test parameter of a certain first server log by the model under test is correct.

[0087] Specifically, for any first test parameter, the first comparison result can be used to determine the first number of first server logs that are consistent with the first diagnostic result and the second diagnostic result, that is, to determine the number of first server logs that are correctly diagnosed by the model under test for any first test parameter.

[0088] In operation S330, the diagnostic accuracy of the model under test for any first test parameter is obtained based on the first number and the number of first server logs in the benchmark dataset.

[0089] According to an embodiment of this application, based on the first number and the number of first server logs in the benchmark dataset, the diagnostic accuracy of the model under test for any first test parameter can be obtained, so as to determine whether the diagnostic accuracy of the model under test meets the requirements.

[0090] In one embodiment, the ratio of a first number to the number of first server logs in the benchmark dataset can be calculated, and this ratio can represent the diagnostic accuracy of the test subject for any of the first test parameters.

[0091] According to an embodiment of this application, the multiple first test parameters may include log type, server fault type, and server type; the test model is trained using the log type, server fault type, and server type involved in the server log, and the test model is used to identify the multiple first test parameters in the first server log.

[0092] In one embodiment, the plurality of first test parameters may further include a serial number and a diagnostic identifier.

[0093] Specifically, the log type can represent the type of the first server log; the server fault type can represent the fault type of the server corresponding to the first server log, such as memory fault, hard disk fault, etc.; the server type can represent the model of the server corresponding to the first server log; and the serial number can represent the serial number of the server corresponding to the first server log.

[0094] In one embodiment, the model under test and the test model include expert rule diagnosis, algorithm rule diagnosis, and a decision module. Each log entry goes to these two diagnostic sections and outputs a result for each. The decision model then determines whether the expert rule diagnosis or the algorithm rule diagnosis is correct and outputs the corresponding result. Thus, the diagnostic identifier can represent the degree of matching between the expert rule diagnosis and the algorithm rule diagnosis in the model under test or the test model.

[0095] Since multiple first test parameters are core parameters, the test model is trained only on multiple first test parameters involved in the server logs in order to identify multiple first test parameters in the first server logs.

[0096] According to an embodiment of this application, the first comparison results between the model under test and the test model regarding multiple first test parameters of multiple first server logs can be shown in Table 1 below.

[0097] Table 1

[0098]

[0099] In Table 1, "Pass" indicates that the diagnostic results of the test model and the test model are consistent with the first test parameter, and "Fail" indicates that the diagnostic results of the test model and the test model are inconsistent with the first test parameter.

[0100] Among them, CPU stands for Central Processing Unit; BMC stands for Baseboard Management Controller.

[0101] According to the embodiments of this application, the diagnostic accuracy of the model under test with respect to each of the multiple first test parameters can be shown in Table 2 below.

[0102] Table 2

[0103]

[0104] In Table 2, the test logs can represent the number of logs from the first server in the benchmark dataset.

[0105] According to embodiments of this application, by comparing the diagnostic results of the model under test and the test model for any first test parameter of any first server log, the diagnostic accuracy of the model under test with respect to each of the multiple first test parameters can be determined. This diagnostic accuracy can then be used to indicate the diagnostic accuracy of the model under test with respect to the first test parameters, thereby determining whether the benchmark test results of the model under test meet preset conditions, i.e., whether the diagnostic accuracy of the model under test meets the requirements. Furthermore, the diagnostic accuracy of the model under test can be determined by comparing the diagnostic results of the multiple first test parameters, which are core parameters.

[0106] Based on the above, benchmarking is conducted using a batch benchmark dataset. By comparing the diagnostic differences between the model under test and the test model, the accuracy of the model under test's diagnosis for multiple primary test parameters is assessed. Benchmarking primarily aims to prevent significant drops in diagnostic accuracy due to serious logical defects introduced by changes in software structure or diagnostic logic, thereby ensuring the reliability and accuracy of the diagnosis.

[0107] According to an embodiment of this application, the second server log for development requirements in the current time period includes server logs of diagnostic requirements added in the current time period and server logs of diagnostic errors added in the current time period.

[0108] According to embodiments of this application, the second server log for development requirements in the current time period is the second server log newly added in the current time period relative to the previous time period. Specifically, the server log for diagnostic requirements newly added in the current time period may include, for example, the server log corresponding to the model under test needing to be adapted to a new machine model; the server log for diagnostic errors newly added in the current time period may include, for example, the log corresponding to rules that the model under test failed to cover in the previous time period, such as the log of diagnostic errors of the published model under test in the previous time period.

[0109] For example, during the current period, the developers have compiled 10 diagnostic error logs and 10 diagnostic requirement logs. Therefore, the second server log used for the development requirements during the current period can be the aforementioned 20 server logs.

[0110] According to an embodiment of this application, since the second server log for the development requirements of the current time period includes server logs of newly added diagnostic requirements and server logs of newly added diagnostic errors in the current time period, it is possible to detect whether the model under test has met the requirements for the newly added diagnostic requirements and the existing diagnostic errors.

[0111] According to embodiments of this disclosure, inputting a baseline test dataset for the current time period into the model under test to obtain multiple third diagnostic results for any second test parameter of multiple second server logs includes: inputting multiple second server logs from the baseline test dataset into the model under test to obtain a third diagnostic result for each of the multiple second test parameters of any second server log.

[0112] According to an embodiment of this application, the model under test can be used to identify multiple parameters in a second server log. The multiple second test parameters are a part of the multiple parameters in the second server log that the model under test can identify, and the multiple second test parameters are the core parameters that the model under test can identify.

[0113] According to an embodiment of this application, by inputting multiple second server logs from the baseline test dataset into the model under test, diagnostic results for multiple parameters of any given second server log can be obtained. Since these multiple second test parameters are core parameters, third diagnostic results for each of the multiple second test parameters of any given second server log can be selected from the output of the model under test.

[0114] According to embodiments of this application, by inputting a baseline test dataset into the model under test, third diagnostic results for each of the multiple second test parameters of the model under test for any second server log can be obtained. Therefore, based on multiple third diagnostic results and baseline test samples, baseline testing of the model under test can be performed, thereby facilitating the determination of whether the model under test meets development requirements.

[0115] Figure 4 A flowchart illustrating the process of obtaining baseline test results according to an embodiment of this application is shown.

[0116] According to embodiments of this application, baseline test results can characterize the diagnostic accuracy of the model under test for each of the multiple second test parameters.

[0117] like Figure 4 As shown, the method 400 includes operations S410 to S430.

[0118] In operation S410, for any second test parameter of any second server log, the baseline value of any second test parameter is compared with the third diagnostic result based on the baseline test sample to obtain the second comparison result of any second test parameter.

[0119] According to an embodiment of this application, since the diagnostic accuracy of the model under test regarding development requirements is tested using a baseline test sample, it is necessary to compare the baseline value of any second test parameter for any second server log with the third diagnostic result based on the baseline test sample to obtain a second comparison result of the baseline test sample and the model under test for that second test parameter.

[0120] According to embodiments of this application, a plurality of second test parameters may include server failure types and baseline test cases, wherein the baseline test sample includes baseline values ​​for server failure types and baseline values ​​for baseline test cases for any second server log.

[0121] In one embodiment, the multiple second test parameters may further include a serial number and a server type. Specifically, the baseline test case may represent a specific fault under a server fault type, such as fault level, fault reason, solution, and comment.

[0122] According to an embodiment of this application, the third diagnostic result of the baseline use case of the model under test with respect to multiple second server logs and the baseline value of the baseline test sample with respect to multiple second server logs can be shown in Table 3 below.

[0123] Table 3

[0124]

[0125] In Table 3, for log 1, the diagnostic result of the model under test for the baseline use case is to provide a conclusion suggestion. For example, if the specific diagnostic result is to replace RAID card 1, then the diagnostic result of the model under test for the baseline use case is inconsistent with the baseline value. For log 5, the diagnostic result of the model under test for the baseline use case is the level of hard drive failure. The baseline value of the baseline use case in the baseline test sample is a potential problem. Therefore, the diagnostic result of the model under test for the baseline use case is inconsistent with the baseline value. The model under test has identified the potential problem in log 5 as a hard drive failure.

[0126] Among them, RAID card is Redundant Array of Independent Disks; BBU is Baseband Unit.

[0127] According to an embodiment of this application, taking Table 3 above as an example, the second comparison results of the test model and the baseline test sample in Table 3 regarding multiple second test parameters of multiple second server logs can be shown in Table 4 below.

[0128] Table 4

[0129]

[0130] According to an embodiment of this application, the baseline test result of the model under test can be obtained based on the second comparison results of each of the second test parameters of each second server log in the baseline test dataset.

[0131] In operation S420, for any second test parameter, determine the second number of second server logs that indicate the baseline value is consistent with the third diagnostic result.

[0132] According to embodiments of this application, baseline test results can characterize the diagnostic accuracy of the model under test for each of the multiple second test parameters.

[0133] According to an embodiment of this application, if the second comparison result indicates that the baseline value is consistent with the third diagnostic result, it can be determined that the diagnosis of a certain second test parameter of a certain second server log by the model under test is correct.

[0134] Specifically, for any second test parameter, the second number of second server logs whose baseline value is consistent with the third diagnostic result can be determined, that is, the number of first server logs whose diagnosis is correct for any second test parameter.

[0135] In operation S430, the diagnostic accuracy of the model under test for any second test parameter is obtained based on the second number and the number of second server logs in the baseline test dataset.

[0136] According to an embodiment of this application, based on the second number and the number of second server logs in the baseline test dataset, the diagnostic accuracy of the model under test for any second test parameter can be obtained, so as to determine whether the model under test meets the development requirements.

[0137] In one embodiment, the ratio of the second number to the number of second server logs in the baseline test dataset can be calculated, and this ratio can represent the diagnostic accuracy of the test subject for any of the second test parameters.

[0138] According to embodiments of this application, by comparing the third diagnostic result of the model under test for any first test parameter of any first server log with the baseline value of that first test parameter in the baseline test sample, the diagnostic accuracy of the model under test for each of the multiple second test parameters can be determined. This diagnostic accuracy can then indicate the diagnostic accuracy of the model under test for the second test parameters, which is used to determine whether the baseline test result of the model under test meets preset conditions, i.e., whether the model under test meets the development requirements of the current period and historical development requirements. Furthermore, by comparing the diagnostic results of the multiple second test parameters (which are core parameters) with the baseline values, it can be determined whether the model under test meets the development requirements.

[0139] Figure 5 A schematic diagram illustrating the process of benchmarking and baseline testing of the model under test according to an embodiment of this application is shown.

[0140] like Figure 5 As shown, benchmark testing is performed on the model under test based on the benchmark dataset. Specifically, the model under test outputs a first diagnostic result on the benchmark dataset, and the test model outputs a second diagnostic result on the benchmark dataset. The first and second diagnostic results are compared to obtain the first comparison result, thereby obtaining the benchmark test result for the model under test.

[0141] Based on the baseline test dataset, baseline testing is performed on the model to be tested. Specifically, the model to be tested outputs a third diagnostic result on the baseline test dataset, and a second comparison result is obtained by comparing the third diagnostic result with the baseline test sample, thereby obtaining the baseline test result on the model to be tested.

[0142] Figure 6 A schematic diagram of a benchmark test according to an embodiment of this application is shown.

[0143] like Figure 6 As shown, the initial number of first server logs in the benchmark dataset can be 6000, meaning the number of first server logs in the benchmark dataset increases periodically from this. The periodic incremental logs specifically involve adding 1% of new first server logs.

[0144] For example, the number of logs from the first server in the benchmark dataset dynamically increases with the total number of logs in the diagnostic system, with each increase representing 1% of the randomly selected log increment. The number of logs from the first server in the benchmark dataset initially was 6000, and has increased to 9900 in the current time period.

[0145] According to an embodiment of this application, multiple first server logs from the benchmark dataset are input into the test model and also into the model under test. Based on the outputs of the test model and the model under test, benchmark tests are performed on multiple first parameters under test to obtain the diagnostic accuracy rates for each parameter, specifically including: log type accuracy, fault type accuracy, machine model accuracy, serial number accuracy, and diagnostic identifier accuracy. Based on this, benchmark test results for the model under test can be obtained.

[0146] Figure 7 A schematic diagram of a baseline test according to an embodiment of this application is shown.

[0147] like Figure 7 As shown, the initial number of second server logs in the baseline test dataset can be 100, meaning the number of second server logs in the baseline test dataset increases periodically from this. The periodic incremental logs specifically involve adding second server logs for development needs specific to the current time period.

[0148] For example, the number of second server logs in the baseline test dataset increases dynamically with each iteration of the model under test, with an increment of 30 to 100 per period. The number of second server logs in the baseline test dataset can start at 100 and increase to 470 in the current time period.

[0149] According to an embodiment of this application, multiple second server logs from the baseline test dataset are input into the model under test. Based on the output of the model under test and the baseline test samples, baseline tests are performed on multiple second parameters under test to obtain the diagnostic accuracy rates for each parameter, specifically including: test case accuracy, fault type accuracy, machine model accuracy, and serial number accuracy. Based on this, baseline test results for the model under test can be obtained.

[0150] According to an embodiment of this application, the preset conditions include at least one of the following: among a plurality of first test parameters, the diagnostic accuracy of at least one first test parameter is less than a first preset threshold; among a plurality of second test parameters, the diagnostic accuracy of at least one second test parameter is less than a second preset threshold.

[0151] According to an embodiment of this application, a first preset threshold is set for the diagnostic accuracy of the first test parameter, and a second preset threshold is set for the diagnostic accuracy of the second test parameter.

[0152] If the diagnostic accuracy of at least one of the multiple first test parameters is less than a first preset threshold, or if the diagnostic accuracy of at least one of the multiple second test parameters is less than a second preset threshold, then the model under test is determined to have a problem.

[0153] Specifically, since benchmark testing and baseline testing have different purposes, benchmark testing is to verify the overall quality of the entire model under test to ensure that the model under test does not have major bugs (errors). Therefore, it is sufficient to ensure that the diagnostic results of the test model and the model under test regarding the first test parameter are not too different. Thus, for example, the first preset threshold can be set to 98%. Baseline testing is to verify that the model under test meets the development requirements of the current period to ensure that the model under test can fully meet the development requirements of the current period. Thus, the second preset threshold is set to 100%.

[0154] According to the embodiments of this application, the setting of preset conditions based on the first preset threshold and the second preset threshold can ensure that the diagnostic accuracy of the model under test meets the requirements and that the model under test meets the development requirements.

[0155] According to an embodiment of this application, the test model for the current time period is obtained by optimizing the test model for the previous time period using the development requirements for the current time period.

[0156] According to an embodiment of this application, the model to be tested in the previous period can represent the model to be tested released in the previous period. For the current period, new development requirements exist, therefore, it is necessary to optimize the model to be tested released in the previous period based on these new development requirements. Thus, the optimized model to be tested is the model that needs to be tested in the current period.

[0157] In one embodiment, while optimizing the model under test based on development requirements, it is also necessary to optimize the test model accordingly based on development requirements to ensure that the test model is compatible with the model under test.

[0158] Specifically, to achieve benchmark testing, a test model was developed simultaneously with the model under test. This test model, similar in principle to the diagnostic system, is based on in-band and out-of-band text logs, performing data standardization, rule matching, and fault diagnosis, ultimately outputting core key data such as fault type, device model, and serial number. Unlike the diagnostic system's stringent requirements regarding development language, multi-platform compatibility, code security, parameter output, and algorithm models, the test model uses a scripting language, supports fixed output parameters and periodic maintenance of diagnostic rules, enabling rapid development and iteration.

[0159] Based on this, the test model can be maintained, updated, and iterated according to the development requirements of the model under test and the benchmark test results.

[0160] For example, during benchmark testing, the reliability of the test model can also be verified in reverse. For instance, the diagnostic accuracy of server fault types in a certain version of the test model dropped to 97%. After testing, it was found that the test model had added expansion card type adaptation. This adaptation was introduced to adapt to new models, but the test model had not yet added it. Thus, it can drive the test model to adapt to new rules and new models.

[0161] Meanwhile, the test model has certain development advantages over the model under test: compared to the model under test, the test model has simpler functions, faster development and iteration, and lower fault tolerance.

[0162] Specifically, this includes several aspects: The test model does not need to include complex algorithm models; it only needs to focus on developing and maintaining the corresponding diagnostic rules. The logic is relatively simple and adapts quickly. Conversely, the model under test requires a significant amount of time to maintain and test algorithm models. It does not need to focus on complex parameters; the test model focuses on only five key parameters: device model, serial number, fault type, diagnostic identifier, and log type. In contrast, the model under test currently has dozens of parameters, and the identification and development of these parameters requires a significant amount of time for development and maintenance. It does not need to focus on log decompression; after the business test model executes, it directly transmits the decompressed logs to the test model. There is no need to focus on numerous log types and encryption methods. Conversely, the model under test needs to identify and decompress various standard and non-standard formats. It does not need to focus on performance; the test diagnostic architecture and instructions are relatively flexible. There is no need to focus on CPU usage time and data read / write times. Conversely, the model under test needs to focus on performance parameters such as CPU and memory usage time, and data read / write times.

[0163] According to the embodiments of this application, since the model to be tested in the current time period is obtained by optimizing the model to be tested in the previous time period using the development requirements for the current time period, testing the model to be tested in the current time period is to test whether the diagnostic accuracy of the optimized model to be tested meets the requirements and whether it meets the development requirements, that is, to test whether the optimization of the model to be tested is successful.

[0164] According to an embodiment of this application, the above-mentioned model testing method further includes: determining that the diagnostic accuracy of the model under test is not affected by the model optimization in the current period when the benchmark test results meet preset conditions; and determining that the model under test meets the development requirements of the current period and the development requirements of the previous period when the baseline test results meet preset conditions.

[0165] According to an embodiment of this application, if the benchmark test results meet the preset conditions, that is, the diagnostic accuracy of multiple first test parameters is greater than or equal to the first preset threshold, it can be determined that the diagnostic accuracy of the model under test is not affected by the model optimization in the current time period.

[0166] For example, when testing a certain version of the model under test, the accuracy of server fault types in the benchmark test results dropped to 91%. This result was reported to the developers of the model under test. Analysis revealed that this issue was caused by a newly added function to obtain the component's serial number (SN). Introducing this function into the model also involved significant adjustments to the logic for obtaining component types, which resulted in a defect in the component type acquisition logic. This demonstrates that current model optimizations can affect the diagnostic accuracy of the model under test.

[0167] Therefore, if the benchmark test results do not meet the preset conditions, the model under test should be adjusted in a timely manner to ensure that the diagnostic accuracy of the model under test is not affected by the model optimization in the current period and to avoid diagnostic accuracy problems after the model under test is released.

[0168] According to an embodiment of this application, if the baseline test results meet the preset conditions, that is, the diagnostic accuracy of multiple second test parameters is equal to the second preset threshold, it can be determined that the model under test meets the development requirements of the current period and the development requirements of the previous period.

[0169] According to the embodiments of this application, benchmark testing and baseline testing test the model under test from different aspects, so that if the benchmark test results meet the preset conditions, it can be determined that the diagnostic accuracy of the model under test is not affected by the model optimization in the current period; if the baseline test results meet the preset conditions, it can be determined that the model under test meets the development requirements of the current period and the development requirements of the previous period.

[0170] In one embodiment, when testing a certain version of the model under test, the accuracy of the server model in the benchmark test results dropped to 85%, and the feedback was sent to the developers of the model under test. After analysis, it was found that the problem was caused by a new FRU (Field Replaceable Unit) log acquisition model introduced in the model under test. This FRU format only supports specific models and the output format is non-standard. The model under test did not strictly limit this method, which led to the introduction of model acquisition adaptation problems.

[0171] Based on the model testing method of this application, based on a batch of benchmark test datasets, benchmark testing can quickly identify problems introduced in the development of the model under test before its release, ensuring the diagnostic reliability and accuracy of the model under test; based on an incremental baseline test dataset, baseline testing can quickly confirm the implementation of the version iteration requirements of the model under test before its release, and ensure that historical version iterations are not affected, thus ensuring the reliability of version iterations. That is, it ensures that the model under test meets the development requirements of the current period before release, and also ensures that the model under test meets the historical development requirements.

[0172] Based on the above model testing method, this application also provides a model testing apparatus. The following will combine... Figure 8 The device is described in detail.

[0173] Figure 8 A structural block diagram of a model testing apparatus according to an embodiment of this application is shown.

[0174] like Figure 8 As shown, the model testing device 800 of this embodiment includes a first input module 810, a first acquisition module 820, a second input module 830, a second acquisition module 840, and a first determination module 850.

[0175] The first input module 810 is used to input the benchmark test dataset for the current time period into the test model and the model under test, respectively, to obtain multiple first diagnostic results of the test model for any first test parameter of multiple first server logs and multiple second diagnostic results of the model under test for any first test parameter of multiple first server logs. The benchmark test dataset includes first server logs from previous time periods and first server logs obtained by sampling newly added first server logs in the current time period according to a preset ratio. In one embodiment, the first input module 810 can be used to perform the operation S210 described above, which will not be repeated here.

[0176] The first obtaining module 820 is used to obtain benchmark test results based on the number of identical diagnostic results for the same first server log among multiple first diagnostic results and multiple second diagnostic results. In one embodiment, the first obtaining module 820 can be used to perform the operation S220 described above, which will not be repeated here.

[0177] The second input module 830 is used to input the baseline test dataset for the current time period into the model under test to obtain multiple third diagnostic results for any second test parameter of multiple second server logs. The baseline test dataset includes second server logs from previous time periods and second server logs for development requirements in the current time period. In one embodiment, the second input module 830 can be used to perform the operation S230 described above, which will not be repeated here.

[0178] The second obtaining module 840 is used to obtain a baseline test result based on the number of baseline values ​​for any second test parameter in the baseline test sample and the number of identical values ​​for the same second server log in the multiple third diagnostic results. In one embodiment, the second obtaining module 840 can be used to perform the operation S240 described above, which will not be repeated here.

[0179] The first determining module 850 is used to determine that the model under test has an anomaly when the benchmark test results and baseline test results do not meet preset conditions. In one embodiment, the first determining module 850 can be used to perform the operation S250 described above, which will not be repeated here.

[0180] According to an embodiment of this application, the first input module 810 includes a first input unit and a second input unit.

[0181] The first input unit is used to input multiple first server logs from the benchmark dataset into the test model to obtain the first diagnostic results for each of the multiple first test parameters of any one first server log.

[0182] The second input unit is used to input multiple first server logs from the benchmark dataset into the model under test to obtain second diagnostic results for each of the multiple first test parameters of any first server log.

[0183] According to an embodiment of this application, the benchmark test result characterizes the diagnostic accuracy of the model under test for each of the multiple first test parameters; the first obtaining module 820 includes a first comparison unit, a first determining unit, and a first obtaining unit.

[0184] The first comparison unit is used to compare the first diagnostic result of any first test parameter with the second diagnostic result for any first test parameter in any first server log, and obtain the first comparison result of any first test parameter.

[0185] The first determining unit is configured to determine, for any first test parameter, a first number of first server logs indicating that the first diagnostic result is consistent with the second diagnostic result.

[0186] The first obtaining unit is used to obtain the diagnostic accuracy of the model under test for any first test parameter based on the first number and the number of first server logs in the benchmark dataset.

[0187] According to an embodiment of this application, the second input module 830 includes a third input unit.

[0188] The third input unit is used to input multiple second server logs from the baseline test dataset into the model under test to obtain the third diagnostic results for each of the multiple second test parameters of any second server log.

[0189] According to an embodiment of this application, the baseline test result characterizes the diagnostic accuracy of the model under test for each of the multiple second test parameters; the second acquisition module 840 includes a second comparison unit, a second determination unit, and a second acquisition unit.

[0190] The second comparison unit is used to compare the baseline value of any second test parameter with the third diagnostic result based on the baseline test sample for any second test parameter of any second server log, so as to obtain the second comparison result of any second test parameter.

[0191] The second determining unit is used to determine, for any second test parameter, the second number of second server logs whose second comparison result characterizes the baseline value and the third diagnostic result as consistent.

[0192] The second obtaining unit is used to obtain the diagnostic accuracy of the model under test for any second test parameter based on the second number and the number of second server logs in the baseline test dataset.

[0193] According to an embodiment of this application, the model testing apparatus 800 further includes a second determining module.

[0194] The second determination module is used to determine the model to be tested as the model to be released if the benchmark test results and baseline test results meet preset conditions.

[0195] According to embodiments of this application, the model testing apparatus 800 further includes a third determining module and a fourth determining module.

[0196] The third determination module is used to determine whether the diagnostic accuracy of the model under test is affected by model optimization in the current period, provided that the benchmark test results meet the preset conditions.

[0197] The fourth determination module is used to determine whether the model under test meets the development requirements of the current period and the development requirements of the previous period, provided that the baseline test results meet the preset conditions.

[0198] According to embodiments of this application, any plurality of modules among the first input module 810, first obtaining module 820, second input module 830, second obtaining module 840, and first determining module 850 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of this application, at least one of the first input module 810, first obtaining module 820, second input module 830, second obtaining module 840, and first determining module 850 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in any one of software, hardware, and firmware methods, or in a suitable combination of any of these methods. Alternatively, at least one of the first input module 810, the first obtaining module 820, the second input module 830, the second obtaining module 840, and the first determining module 850 may be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0199] Figure 9 A block diagram of an electronic device suitable for implementing a model testing method according to an embodiment of this application is shown.

[0200] like Figure 9 As shown, an electronic device 900 according to an embodiment of this application includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage portion 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0201] RAM 903 stores various programs and data required for the operation of electronic device 900. Processor 901, ROM 902, and RAM 903 are interconnected via bus 904. Processor 901 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 902 and / or RAM 903. It should be noted that the programs may also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0202] According to embodiments of this application, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to a bus 904. The electronic device 900 may also include one or more of the following components connected to the input / output (I / O) interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the input / output (I / O) interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 910 as needed so that computer programs read from it can be installed into the storage section 908 as needed.

[0203] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0204] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 902 and / or RAM 903 and / or one or more memories other than ROM 902 and RAM 903 described above.

[0205] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the model testing method provided in the embodiments of this application.

[0206] When the computer program is executed by the processor 901, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0207] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 909, and / or installed from a removable medium 911. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0208] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, it performs the functions defined in the system of this application embodiment. According to the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0209] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0210] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0211] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

[0212] The embodiments of this application have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of this application. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Without departing from the scope of this application, those skilled in the art can make various substitutions and modifications, all of which should fall within the scope of this application.

Claims

1. A model testing method, characterized in that, The method includes: The benchmark dataset for the current time period is input into the test model and the model under test, respectively, to obtain multiple first diagnostic results of the test model for any first test parameter of multiple first server logs and multiple second diagnostic results of the model under test for any first test parameter of the multiple first server logs. The benchmark dataset includes first server logs from the previous time period and first server logs sampled from newly added first server logs in the current time period according to a preset ratio. The benchmark test results are obtained based on the number of identical diagnostic results for the same first server log among the multiple first diagnostic results and the multiple second diagnostic results. The baseline test dataset for the current time period is input into the model under test to obtain multiple third diagnostic results for any second test parameter of multiple second server logs, wherein the baseline test dataset includes second server logs from the previous time period and second server logs for the development requirements of the current time period; Based on multiple baseline values ​​for any second test parameter in the baseline test sample and the number of identical values ​​for the same second server log in the multiple third diagnostic results, a baseline test result is obtained, including: for any second test parameter of any second server log, comparing the baseline value for any second test parameter with the third diagnostic result based on the baseline test sample to obtain a second comparison result for any second test parameter; for any second test parameter, determining a second number of second server logs whose baseline value matches the third diagnostic result; and obtaining the diagnostic accuracy of the model under test for any second test parameter based on the second number and the number of second server logs in the baseline test dataset; wherein, the baseline test result characterizes the diagnostic accuracy of the model under test for each of the multiple second test parameters; If the benchmark test results and the baseline test results do not meet the preset conditions, it is determined that the model under test has an anomaly.

2. The method according to claim 1, characterized in that, The step of inputting the benchmark dataset used for the current time period into the test model and the model under test respectively, to obtain multiple first diagnostic results of the test model for any first test parameter of multiple first server logs and multiple second diagnostic results of the model under test for any first test parameter of multiple first server logs, includes: Input multiple first server logs from the benchmark dataset into the test model to obtain first diagnostic results for each of the multiple first test parameters for any first server log. Multiple first server logs from the benchmark dataset are input into the model under test to obtain second diagnostic results for each of the multiple first test parameters of any one first server log.

3. The method according to claim 2, characterized in that, The benchmark test results characterize the diagnostic accuracy of the model under test for each of the multiple first test parameters; the benchmark test results are obtained based on the number of identical diagnostic results for the same first server log among the multiple first diagnostic results and the multiple second diagnostic results, including: For any first test parameter in any first server log, compare the first diagnostic result of the first test parameter with the second diagnostic result to obtain the first comparison result of the first test parameter; For any first test parameter, determine the first comparison result representing the first number of first server logs that are consistent with the first diagnostic result and the second diagnostic result; Based on the first number and the number of first server logs in the benchmark dataset, the diagnostic accuracy of the model under test for any of the first test parameters is obtained.

4. The method according to claim 3, characterized in that, The multiple first test parameters include log type, server fault type, and server type; The test model is trained using the log types, server fault types, and server types involved in the server logs. The test model is used to identify multiple first test parameters in the first server log.

5. The method according to claim 1, characterized in that, The process involves inputting the baseline test dataset for the current time period into the model under test to obtain multiple third diagnostic results for any second test parameter of multiple second server logs, including: By inputting multiple second server logs from the baseline test dataset into the model under test, a third diagnostic result is obtained for each of the multiple second test parameters of any second server log.

6. The method according to claim 5, characterized in that, The plurality of second test parameters include server failure types and baseline test cases. The baseline test sample includes baseline values ​​for server failure types and baseline values ​​for baseline test cases for any second server log.

7. The method according to claim 1, characterized in that, The preset conditions include at least one of the following: Among multiple first test parameters, the diagnostic accuracy of at least one first test parameter is less than a first preset threshold; Among multiple second test parameters, the diagnostic accuracy of at least one second test parameter is less than a second preset threshold.

8. The method according to claim 1, characterized in that, The method further includes: If the benchmark test results and the baseline test results meet the preset conditions, the model to be tested will be determined as the model to be released.

9. The method according to any one of claims 1 to 8, characterized in that, The test model for the current time period is obtained by optimizing the test model for the previous time period using the development requirements for the current time period.

10. The method according to claim 9, characterized in that, The method further includes: If the benchmark test results meet the preset conditions, it is determined that the diagnostic accuracy of the model under test is not affected by the model optimization in the current time period; If the baseline test results meet the preset conditions, it is determined that the model under test meets the development requirements of the current period and the development requirements of the previous period.

11. The method according to claim 1, characterized in that, The second server log for development needs in the current time period includes server logs of diagnostic needs added in the current time period and server logs of diagnostic errors added in the current time period.

12. A model testing device, characterized in that, The device includes: The first input module is used to input the benchmark test dataset for the current time period into the test model and the model under test respectively, to obtain multiple first diagnostic results of the test model for any first test parameter of multiple first server logs and multiple second diagnostic results of the model under test for any first test parameter of the multiple first server logs, wherein the benchmark test dataset includes first server logs from the previous time period and first server logs sampled from newly added first server logs in the current time period according to a preset ratio; The first obtaining module is used to obtain benchmark test results based on the number of identical diagnostic results for the same first server log among the plurality of first diagnostic results and the plurality of second diagnostic results; The second input module is used to input the baseline test dataset for the current time period into the model under test to obtain multiple third diagnostic results for any second test parameter of multiple second server logs, wherein the baseline test dataset includes second server logs from the previous time period and second server logs for development requirements in the current time period; The second obtaining module is used to obtain the baseline test result based on the number of baseline values ​​for any second test parameter in the baseline test sample and the number of identical values ​​for the same second server log in the multiple third diagnostic results; The baseline test results characterize the diagnostic accuracy of the model under test for each of the multiple second test parameters; the second acquisition module includes a second comparison unit, a second determination unit, and a second acquisition unit. The second comparison unit is used to compare the baseline value of any second test parameter with the third diagnostic result based on the baseline test sample for any second test parameter of any second server log, and obtain the second comparison result of any second test parameter. The second determining unit is used to determine, for any second test parameter, the second number of second server logs that indicate that the second comparison result is consistent with the baseline value and the third diagnostic result; The second obtaining unit is used to obtain the diagnostic accuracy of the model under test for any of the second test parameters based on the second number and the number of second server logs in the baseline test dataset. The first determining module is used to determine that the model under test has an anomaly when the benchmark test results and the baseline test results do not meet preset conditions.

13. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 11.

14. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 11.

Images