Deep learning model testing method, device, system and storage medium

By applying container engine technology and the use of information summary values, the problem of poor environmental isolation in deep learning model testing is solved, the testing efficiency and accuracy are improved, and the fairness and stability of the test are ensured.

CN116263737BActive Publication Date: 2024-11-01ZHONGKE NANJING SOFTWARE TECH RES INST
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111530253.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-14
Publication Date
2024-11-01
Estimated Expiration
2041-12-14

AI Technical Summary

Technical Problem

In traditional deep learning model testing methods, the isolation between the test objects and the test software and hardware environment is poor, resulting in low testing efficiency and frequent replacement of the test environment, which affects efficiency.

Method used

The test data set and test environment are encapsulated in a mirror file by applying container engine technology, and the completeness judgment is made using information summary values ​​to ensure the isolation of the test environment and the model, and the potential tampering behavior is identified through summary value comparison, so as to achieve stability and accuracy test results.

Benefits of technology

It realizes effective isolation between the deep learning model and the test environment, improves testing efficiency, ensures the fairness of the test and the accuracy of the results, and reduces dependence on the test environment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116263737B_ABST
    Figure CN116263737B_ABST
Patent Text Reader

Abstract

The present application relates to the field of artificial intelligence, and specifically to a method, device, system and storage medium for testing a deep learning model, the method comprising: a testing server sends a first image file to a testing platform server; the testing platform server pre-tests the deep learning model based on the first image file, and encapsulates the first image file and the deep learning model into a second image file; performs information digest calculation on the second image file to obtain a third summary value; sends the second image file and the third summary to the testing server; the testing server performs information digest calculation on the first image file in the second image file to obtain a second summary value; performs information digest calculation on the second image file to obtain a fourth summary value; obtains the first summary value; tests the deep learning model in the second image file based on the first image file to obtain a test result. The present application gets rid of the defect of relying on the test environment when testing the model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and specifically to a testing method, device, system and storage medium for a deep learning model. Background Art

[0002] Testing and evaluation of deep learning models is a key testing step that all types of deep learning algorithms need to carry out before implementation. This step is used to evaluate and determine whether the deep learning model can achieve the expected set goals.

[0003] Traditional deep learning model testing requires setting up a test environment in the tester's server and connecting the deep learning model to be tested to the tester's server. From a testing perspective, traditional testing methods tightly bind the test object to the test hardware and software environment, resulting in poor isolation of the test process. In addition, if the test environment needs to be changed during the test, repeated work will be required to set up the test environment, which seriously affects test efficiency. Summary of the invention

[0004] Therefore, the present application aims to solve the technical problems in the prior art that the isolation between the test object and the software and hardware environment of the test is poor and the efficiency is low, thereby providing a test method, device, system and storage medium for a deep learning model. The technical solution is as follows:

[0005] On the one hand, the present application provides a deep learning model testing method, which is applied to a testing party server, and the method includes:

[0006] Receive a second image file and a third digest value sent by a test platform server; the second image file includes the first image file and the deep learning model, and the third digest value refers to a digest value obtained by the test platform server performing information digest calculation on the second image file;

[0007] Performing information digest calculation on the first image file in the second image file to obtain a second digest value;

[0008] Performing information digest calculation on the second image file to obtain a fourth digest value;

[0009] Obtaining a first digest value, where the first digest value refers to a digest value pre-stored by the testing party server and obtained by performing information digest calculation on the first image file;

[0010] When the first digest value is the same as the second digest value, and the third digest value is the same as the fourth digest value, testing the deep learning model in the second image file based on the first image file to obtain a test result of the deep learning model;

[0011] Among them, the first image file is formed by encapsulating a test data set and a test environment in an application container engine, the test data set is used to test the deep learning model, and the test environment refers to the software and hardware environment and development environment used when testing the deep learning model.

[0012] Optionally, the method also includes: when the first digest value and the second digest value are different, sending a first error message to the test platform server, the first error message is used to indicate that the first image file has been tampered with by the party under test; when the third digest value and the fourth digest value are different, sending a second error message to the test platform server, the second error message is used to indicate that the second image file has been tampered with by a third party.

[0013] Optionally, before testing the deep learning model in the second image file based on the first image file to obtain the test result of the deep learning model, it also includes: obtaining a configuration file, the configuration file being used to determine the concurrent number of running resources of the testing party server; detecting the running resource occupancy status of the testing party server; testing the deep learning model in the second image file based on the first image file to obtain the test result of the deep learning model, including: when the running resource occupancy status does not reach the concurrent number of running resources, testing the deep learning model in the second image file based on the first image file to obtain the test result of the deep learning model.

[0014] Optionally, the method also includes: receiving a file modification instruction sent by the test platform server; modifying the concurrent number of running resources according to the file modification instruction to obtain a modified concurrent number of running resources, and the modified concurrent number of running resources is used to determine whether to test the deep learning model in the second image file based on the first image file.

[0015] On the other hand, the present application provides a deep learning model testing method, which is applied to a test platform server, and the method includes:

[0016] Receive a first image file sent by a tester server, where the first image file is formed by encapsulating a test data set and a test environment in an application container engine, where the test data set is used to test a deep learning model, and the test environment refers to a software and hardware environment and a development environment used when testing the deep learning model; pre-test the deep learning model based on the first image file;

[0017] If the deep learning model passes the pre-test, encapsulating the first image file and the deep learning model into a second image file; performing information digest calculation on the second image file to obtain a third digest value;

[0018] The second image file and the third digest value are sent to the testing party server, where the third digest value is used by the testing party server to determine whether to test the deep learning model in the second image file based on the first image file.

[0019] Optionally, the method further includes: obtaining a user portrait corresponding to the target user account, the user portrait being used to indicate the test preference of the target user account; and displaying a target test task matching the user portrait when the target user account logs in to the test platform server.

[0020] On the other hand, the present application provides a testing device for a deep learning model, the device comprising: a first receiving module, used to receive a second image file and a third digest value sent by a test platform server; the second image file comprises a first image file and a deep learning model, the third digest value refers to a digest value obtained by the test platform server performing information digest calculation on the second image file; a first calculation module, used to perform information digest calculation on the first image file in the second image file to obtain a second digest value; a second calculation module, used to perform information digest calculation on the second image file to obtain a fourth digest value; a first acquisition module, used to obtain a first digest value, the first digest value refers to a digest value obtained by performing information digest calculation on the first image file pre-stored by the test party server; a first testing module, used to test the deep learning model in the second image file based on the first image file when the first digest value is the same as the second digest value and the third digest value is the same as the fourth digest value, to obtain a test result of the deep learning model; wherein the first image file is formed by encapsulating a test data set and a test environment in an application container engine, the test data set is used to test the deep learning model, and the test environment refers to the software and hardware environment and development environment used when testing the deep learning model.

[0021] On the other hand, the present application provides a testing device for a deep learning model, the device comprising: a third receiving module, used to receive a first image file sent by a testing party server, the first image file is formed by encapsulating a test data set and a test environment in an application container engine, the test data set is used to test the deep learning model, and the test environment refers to a software and hardware environment and a development environment used when testing the deep learning model;

[0022] A second testing module, configured to pre-test the deep learning model based on the first image file; a packaging module, configured to package the first image file and the deep learning model into a second image file if the deep learning model passes the pre-test;

[0023] a third calculation module, used to perform information digest calculation on the second image file to obtain a third digest value; a third sending module, used to send the second image file and the third digest value to the test party server, and the third digest value is used by the test party server to determine whether to test the deep learning model in the second image file based on the first image file.

[0024] On the other hand, the present application provides a deep learning model testing system, including a test platform server and a test party server; the test party server is used to send a first image file to the test platform server, the first image file is formed by encapsulating a test data set and a test environment in an application container engine, the test data set is used to test the deep learning model, and the test environment refers to the software and hardware environment and development environment used when testing the deep learning model; the test platform server is used to pre-test the deep learning model based on the first image file; if the deep learning model passes the pre-test, the first image file and the deep learning model are encapsulated into a second image file; the information digest of the second image file is calculated to obtain a third digest value; the second image file and the third digest value are sent to the test party server, and the third digest value is used by the test party server to determine whether to test the deep learning model in the second image file based on the first image file;

[0025] The testing party server is used to perform information digest calculation on the first image file in the second image file to obtain a second digest value; perform information digest calculation on the second image file to obtain a fourth digest value; obtain a first digest value, where the first digest value refers to a digest value pre-stored by the testing party server and obtained by performing information digest calculation on the first image file; when the first digest value is the same as the second digest value, and the third digest value is the same as the fourth digest value, test the deep learning model in the second image file based on the first image file to obtain a test result of the deep learning model.

[0026] On the other hand, the present application provides a server, comprising a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to execute the test method of the deep learning model on the server side of the above-mentioned test party, or executes the test method of the deep learning model on the server side of the above-mentioned test platform.

[0027] On the other hand, the present application provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute the testing method of the deep learning model on the server side of the above-mentioned testing party, or to execute the testing method of the deep learning model on the server side of the above-mentioned testing platform.

[0028] On the other hand, the present application provides a computer program product, which, when running on a server, is used to implement the testing method of the deep learning model on the server side of the above-mentioned testing party, or to implement the testing method of the deep learning model on the server side of the above-mentioned testing platform.

[0029] The technical solution of this application has the following advantages:

[0030] The test method of the deep learning model provided in this application, by using the application container engine technology to encapsulate the test environment in the application container engine, realizes the isolation of the tested model and the test environment, and gets rid of the defects of the traditional test method that is seriously dependent on the test environment. In addition, by providing an image file template (first image file), the test data set, test environment, test estimation results, etc. are unified to ensure the test fairness of the deep learning model. In addition, the integrity judgment is performed by comparing the first summary value and the second summary value, the third summary value and the fourth summary value to clarify whether the tested party has tampered during the pre-test process and whether the third party has tampered during the transmission process, so as to ensure the stability of the test process and the accuracy of the test results. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0032] Figure 1 This is an interactive flow chart of the deep learning model testing method of Example 1 of the present application;

[0033] Figure 2 This is a structural block diagram of a testing device for a deep learning model according to Example 2 of the present application;

[0034] Figure 3 This is a structural block diagram of a testing device for a deep learning model according to Example 3 of the present application;

[0035] Figure 4 This is a structural block diagram of a test system for a deep learning model according to Example 4 of the present application;

[0036] Figure 5 This is a principle block diagram of the server of Example 5 of the present application. DETAILED DESCRIPTION

[0037] The technical solution of the present application will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0038] In the description of this application, it should be noted that the terms "first", "second", and "third" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance. In addition, the technical features involved in the different embodiments of this application described below can be combined with each other as long as they do not conflict with each other.

[0039] In the following description, the tester uses the tester server, and the tested party uses the test platform server. Among them, the tester refers to the subject that tests and evaluates the deep learning model, provides a mirror file containing the test environment, test data set, etc. for the test of the tested model, runs the tested model and obtains the test results. It can be an individual, organization, group, etc.; the tested party refers to the subject that provides the tested model for a specific test task, which can be an individual, organization, group, etc.; the tested model refers to the deep learning model (AI (Artificial Intelligence, artificial intelligence) algorithm model) for a specific test task. The test platform server is a platform for testing and evaluating the tested model, which is used to publish test tasks, save and display the test results of the deep learning model.

[0040] Example 1

[0041] Please refer to Figure 1 , which shows a method for testing a deep learning model provided by this embodiment, which can be applied to a deep learning model testing system, which includes a testing party server and a testing platform server. Although the process described below includes multiple operations that appear in a specific order, it should be clearly understood that these processes may also include more or fewer operations, which can be executed sequentially or in parallel (for example, using a parallel processor or a multi-threaded environment). Figure 1As shown, the method includes at least some of the following steps.

[0042] S101. The testing party server sends a first image file to the testing platform server.

[0043] The first image file is formed by encapsulating the test data set and the test environment in the application container engine. The test data set is used to test the deep learning model. The test environment refers to the hardware and software environment and the development environment used when testing the deep learning model. Among them, the hardware and software environment includes CPU, operating system and hard disk, the development environment includes platform code, and the application container engine includes but is not limited to Docker, Containerd, Rkt, Kata and Pouch. Among them, Docker is an open source application container engine that allows developers to package applications and dependent packages into portable containers. Its advantages mainly include the following points: First, standardized application delivery. Docker's standardized deliverable is called "image", which contains the application and the operating environment it depends on. It can be used across platforms and hosts, greatly simplifying the application delivery model; second, Docker images can achieve "one build, multiple deliveries", which can better reflect the value of Docker when it comes to multiple copies of application deployment or application migration; third, Docker can isolate the mutual influence between different applications, but with less overhead than virtual machines.

[0044] The test party encapsulates the test data set and the test environment in the application container engine according to the application scenario of the deep learning model to form a first image file. After the first image file is formed, the test party uploads the first image file to the test party server to store it in the test party server. It should be noted that the test data set and test environment required for deep learning models in different application scenarios may be the same or different. Those skilled in the art can make reasonable choices based on actual conditions, which is not limited here.

[0045] The deep learning model can be pre-trained by the tested party before pre-testing. When the trained deep learning model needs to be tested, according to the application scenario of the deep learning model, the test platform server sends a request to the test party server to obtain the corresponding first image file, and the test party server sends the first image file to the test platform server; or, when there is a test task that matches the trained deep learning model, the test party server sends the first image file corresponding to the test task to the test platform server.

[0046] S102. The test platform server pre-tests the deep learning model based on the first image file.

[0047] The test platform server uses the first image file to test the pre-trained deep learning model to obtain the test result of the pre-test. If the test result of the pre-test passes (that is, the deep learning model passes the pre-test), step S103 is executed; if the test result of the pre-test fails (that is, the deep learning model does not pass the pre-test), the tested party can choose to terminate subsequent tests, or strengthen the training of the deep learning model so that the deep learning model passes the pre-test.

[0048] The first image file may also contain test estimation results. When the test results of the pre-test reach the test estimation results, the deep learning model passes the pre-test; when the test results of the pre-test do not reach the test estimation results, the deep learning model fails the pre-test. The test estimation results may be accuracy, recall, precision, etc.

[0049] S103. The test platform server encapsulates the first image file and the deep learning model into a second image file.

[0050] When the deep learning model passes the pre-test, the test platform server creates a second image file containing the first image file and the deep learning model. The first image file used for the pre-test is encapsulated in the second image file, so that the tester server can verify the test data set and test environment used for the pre-test, and prevent the tested party from using a tampered test data set and / or test environment for pre-testing.

[0051] S104. The test platform server calculates the information digest of the second image file to obtain a third digest value.

[0052] The second image file includes the first image file and the deep learning model, and the information digest calculation is performed on the second image file to obtain a third digest value. The third digest value enables the test party server to verify the second image file, thereby preventing the first image file and / or the deep learning model in the second image file from being tampered with by a third party during the transmission process, thereby affecting the subsequent test process and / or results.

[0053] S105. The test platform server sends the second image file and the third digest value to the test party server.

[0054] The test party server receives the second image file and the third digest value sent by the test platform server. Since the test data set and the test environment are encapsulated by the application container engine, the isolation of the tested model and the test environment is achieved. Therefore, in the above implementation steps, the tested party only needs to upload the deep learning model (tested model) to the test party server, which greatly saves the amount of data transmission during the test. At the same time, since the separation of the test environment and the deep learning model (tested model) is achieved, the defect of the traditional test method that is heavily dependent on the test environment is eliminated.

[0055] S106. The testing party server calculates the information digest of the first image file in the second image file to obtain a second digest value.

[0056] The second image file includes the first image file and the deep learning model. In order to verify whether the test data set and / or test environment used by the tested party during the pre-test process have been tampered with, the information summary of the first image file in the second image file is calculated to obtain a second summary value.

[0057] S107. The testing party server calculates the information digest of the second image file to obtain a fourth digest value.

[0058] In order to verify whether the second image file is tampered with by a third party during the transmission process, the testing party server calculates the information digest of the second image file to obtain a fourth digest value.

[0059] S108. The testing party server obtains a first digest value.

[0060] In the above implementation steps, the first digest value is pre-stored in the test party server, and the first digest value is a digest value obtained by performing information digest calculation on the first image file. After the test party encapsulates the test data set and the test environment in the application container engine to form the first image file, the test party performs information digest calculation on the first image file to obtain the first digest value. When the first image file is stored in the test party server, the first digest value is also stored in the test party server. It should be noted that the first digest value, the second digest value, the third digest value and the fourth digest value can be MD5 values, MD4 values, SHA1 values, etc. Of course, their digest value types need to be the same, that is, the first digest value, the second digest value, the third digest value and the fourth digest value need to be MD5 values ​​at the same time, or MD4 values ​​at the same time, or SHA1 values ​​at the same time, etc.

[0061] The testing party server compares the first digest value and the second digest value. If the first digest value and the second digest value are the same, it means that the first image file used by the tested party has not been tampered with during the pre-test process; the testing party server compares the third digest value and the fourth digest value. If the third digest value and the fourth digest value are the same, it means that the second image file has not been tampered with by a third party during the transmission process. When the first digest value and the second digest value are the same, and the third digest value and the fourth digest value are the same, step S109 is executed.

[0062] If the first summary value and the second summary value are not the same, it means that the first image file used by the tested party during the pre-test process has been tampered with, then the testing party server sends a first error message to the testing platform server, and the first error message is used to indicate that the first image file has been tampered with by the tested party; if the third summary value and the fourth summary value are not the same, it means that the first image file and / or the deep learning model in the second image file have been tampered with by a third party, then a second error message is sent to the testing platform server, and the second error message is used to indicate that the second image file has been tampered with by a third party. Among them, tampering includes deletion, addition or modification, etc. After receiving the first error message and / or the second error message, the testing platform server can feed back the error message to the tested party. For example, the error message is fed back to the tested party through text messages, emails, instant messaging software, etc., to remind the tested party to repackage the second image file in time or terminate the test.

[0063] For example, the tester encapsulates the test data set and the test environment in Docker to form a first image file A, and performs information digest calculation on the first image file A to obtain the corresponding first MD5 value a. The tested party downloads the first image file A from the tester server to the test platform server, and the test platform server uses the first image file A to pre-test the trained deep learning model (i.e., the model under test). After the pre-test passes, the first image file A and the deep learning model are encapsulated into a second image file B, and the information digest calculation is performed on the second image file B to obtain the corresponding third MD5 value c, and the second image file B and the third MD5 value c are sent to the tester server. The tester server performs information digest calculation on the first image file in the second image file B to obtain the second MD5 value b, performs information digest calculation on the second image file B to obtain the fourth MD5 value d, and obtains the first MD5 value a from the tester server. Compare the first MD5 value a with the second MD5 value b, the third MD5 value c with the fourth MD5 value d respectively. If there is any difference, send the corresponding error information to the test platform server; if they are all the same, execute step S109.

[0064] S109. The testing server tests the deep learning model in the second image file based on the first image file to obtain a test result of the deep learning model.

[0065] The testing server can use the first image file in the second image file to test the deep learning model, or can use the first image file stored in the testing server to test the deep learning model.

[0066] The testing server runs the first image file to test the deep learning model and obtains the test result of the deep learning model. The operation mode includes normal operation and adversarial operation, wherein the normal operation is to test the deep learning model using the first image file and obtain a single test result; the adversarial operation is to test multiple deep learning models of the same type using the first image file and obtain a ranking list of test results of multiple deep learning models.

[0067] In summary, in the embodiment of the present application, by using the application container engine technology to encapsulate the test environment in the application container engine, the isolation of the tested model and the test environment is achieved, and the defect of the traditional test method that is seriously dependent on the test environment is eliminated. In addition, by providing a mirror file template (first mirror file), the test data set, test environment, test estimation results, etc. are unified to ensure the test fairness of the deep learning model. In addition, the integrity judgment is performed by comparing the first summary value and the second summary value, the third summary value and the fourth summary value to clarify whether the tested party has tampered during the pre-test process and whether the third party has tampered during the transmission process, so as to ensure the stability of the test process and the accuracy of the test results.

[0068] In one or more embodiments, before testing the deep learning model in the second image file based on the first image file to obtain the test result of the deep learning model, the method further includes the following steps (S201-S202).

[0069] S201. The testing party server obtains a configuration file, where the configuration file is used to determine the concurrent number of running resources of the testing party server.

[0070] The number of concurrent running resources can be the number of deep learning models that can be tested simultaneously by the tester's server, or the amount of resources occupied when the deep learning model is tested, such as the amount of CPU and memory occupied when the deep learning model is running.

[0071] S202: The testing party server detects the operating resource occupancy status of the testing party server.

[0072] When the tester server tests the deep learning model, it will occupy the running resources of the tester server. Detecting the running resource occupancy status of the tester server can be to detect the number of deep learning models being tested by the tester server, or it can be to detect the amount of testing party server resources occupied by the neural network model. If the running resource occupancy status does not reach (is less than or equal to) the concurrent number of running resources, execute step S203; if the running resource occupancy status reaches the concurrent number of running resources, enter the waiting state, when the running resources of the tester server are idle, or when the running resource occupancy status of the tester server does not reach the concurrent number of running resources, execute step S203.

[0073] Based on the above steps S201 and S202, Figure 1 The step S109 described in the embodiment can be implemented as the following step S203.

[0074] S203: When the running resource status does not reach the running resource concurrent number, the testing party server tests the deep learning model in the second image file based on the first image file to obtain the test result of the deep learning model. For details, please refer to the relevant description of step S109, which will not be repeated here.

[0075] In one or more embodiments, the test platform server can remotely modify the configuration file of the test party server to update the concurrent number of running resources of the test party server. Based on this, the deep learning model testing method also includes the following steps (S301-S302).

[0076] S301. The test platform server sends a file modification instruction to the test party server.

[0077] The test platform server may include a configuration file, the configuration file of the test platform server includes the IP address of the test party server, based on the IP address of the test party server, determine the corresponding test party server, and send a file modification instruction to the corresponding test party server.

[0078] S302: The testing party server modifies the concurrent number of running resources according to the file modification instruction to obtain the modified concurrent number of running resources.

[0079] The file modification instruction may include the specific value of the modified concurrent number of resources, or it may be a trigger command. After receiving the trigger command, the test party server changes the concurrent number of running resources according to the preset rules. For example, the concurrent number of resources can be increased or decreased by multiples, or it can be increased or decreased by a specific value. The modified concurrent number of running resources is used to determine whether to test the deep learning model in the second image file to obtain the test results of the deep learning model. In other words, the test party server can perform the above steps S201-S203 according to the modified concurrent number of running resources.

[0080] The test platform server can use SSH commands to remotely modify the configuration file of the corresponding test party server to complete the update of the concurrent number of running resources of the test party server. In some embodiments, the test party can also directly modify the configuration file in the test party server to update the concurrent number of running resources of the test party server.

[0081] In one or more embodiments, the deep learning model testing method may further include the following steps (S401-S402).

[0082] S401. The test platform server obtains a user portrait corresponding to a target user account, where the user portrait is used to indicate a test preference of the target user account.

[0083] The tested party has a corresponding user account registered in the test platform server, namely the target user account, and the tested party uses the target user account to log in to the test platform server. The test platform server can store a user profile of the target user account, which is used to indicate the test preference of the target user account. The test preference of the target user account can be a label entered by the tested party, or a label automatically marked by the test platform server based on the historical test records of the tested party.

[0084] S402. When the target user account logs in to the test platform server, the test platform server displays the target test task that matches the user portrait.

[0085] After the tested party logs in to the test platform server using the user account, the test platform server displays the test tasks that match the tested party based on the test preferences corresponding to the user account, so that the tested party can view the corresponding test tasks in time and pre-test the deep learning model through the first mirror file corresponding to the test task. When there is a new test task that meets the test preferences of the target user account, the test platform server can also inform the tested party by SMS, email or instant messaging software.

[0086] One point that needs to be explained is that the above method embodiment only introduces and explains the test method of the deep learning model from the perspective of the interaction between the test platform server and the test party server. In the above embodiment, the steps executed by the test party server can be independently implemented as a test method for the deep learning model on the test party server side; the steps executed by the test platform server can be independently implemented as a test method for the deep learning model on the test platform server side.

[0087] Example 2

[0088] This embodiment provides a deep learning model testing device, which can be the above-mentioned testing party server, or can be set in the above-mentioned testing party server. Figure 2 As shown, it includes: a first receiving module 201, a first calculating module 202, a second calculating module 203, a first acquiring module 204 and a first testing module 205.

[0089] The first receiving module 201 is used to receive the second image file and the third digest value sent by the test platform server; the second image file includes the first image file and the deep learning model, and the third digest value refers to the digest value obtained by the test platform server for calculating the information digest of the second image file; for details, please refer to the relevant description in Example 1, which will not be repeated here.

[0090] The first calculation module 202 is used to calculate the information digest of the first image file in the second image file to obtain a second digest value; for details, please refer to the relevant description in Example 1, which will not be repeated here.

[0091] The second calculation module 203 is used to calculate the information digest of the second image file to obtain a fourth digest value; for details, please refer to the relevant description in Example 1, which will not be repeated here.

[0092] The first acquisition module 204 is used to obtain a first digest value, where the first digest value refers to a digest value pre-stored by the test party server and obtained by performing information digest calculation on the first image file; for details, please refer to the relevant description in Example 1, which will not be repeated here.

[0093] The first testing module 205 is used to test the deep learning model in the second image file based on the first image file when the first digest value is the same as the second digest value and the third digest value is the same as the fourth digest value, so as to obtain a test result of the deep learning model; for details, please refer to the relevant description in Example 1, which will not be repeated here.

[0094] The first image file is formed by encapsulating a test data set and a test environment in an application container engine, wherein the test data set is used to test the deep learning model, and the test environment refers to the hardware and software environment and development environment used when testing the deep learning model. For details, please refer to the relevant description in Example 1, which will not be repeated here.

[0095] In one or more embodiments, the device further includes: a first sending module ( Figure 2 ) and a second sending module ( Figure 2 not shown).

[0096] The first sending module is used to send a first error message to the test platform server when the first digest value and the second digest value are different, and the first error message is used to indicate that the first image file is tampered with by the tested party; for details, please refer to the relevant description in Example 1, which will not be repeated here.

[0097] The second sending module is used to send a second error message to the test platform server when the third digest value and the fourth digest value are different, and the second error message is used to indicate that the second image file has been tampered with by a third party. For details, please refer to the relevant description in Example 1, which will not be repeated here.

[0098] In one or more embodiments, the device further includes: a second acquisition module ( Figure 2 Not shown) and the detection module ( Figure 2 not shown).

[0099] The second acquisition module is used to obtain a configuration file, and the configuration file is used to determine the concurrent number of running resources of the test party server; for details, please refer to the relevant description in Example 1, which will not be repeated here.

[0100] The detection module is used to detect the operating resource occupancy status of the test party server; for details, please refer to the relevant description in Example 1, which will not be repeated here.

[0101] On the basis of the second acquisition module and the detection module, when the running resource occupancy state does not reach the running resource concurrency number, the first test module 205 is used to test the deep learning model in the second image file based on the first image file to obtain the test result of the deep learning model. For details, please refer to the relevant description in Example 1, which will not be repeated here.

[0102] In one or more embodiments, the device further includes: a second receiving module ( Figure 2 ) and the modification module ( Figure 2 not shown).

[0103] The second receiving module is used to receive the file modification instruction sent by the test platform server; for details, please refer to the relevant description in Example 1, which will not be repeated here.

[0104] A modification module is used to modify the concurrent number of running resources according to the file modification instruction to obtain the modified concurrent number of running resources, and the modified concurrent number of running resources is used to determine whether to test the deep learning model in the second image file based on the first image file. For details, please refer to the relevant description in Example 1, which will not be repeated here.

[0105] The effect of the testing device for the deep learning model of this embodiment can be found in the relevant description in Embodiment 1 and will not be elaborated here.

[0106] Example 3

[0107] This embodiment provides a test device for a deep learning model, which can be the above-mentioned test platform server, or can be set in the above-mentioned test platform server. Figure 3 As shown, it includes: a third receiving module 301, a second testing module 302, a packaging module 303, a third calculating module 304 and a third sending module 305.

[0108] The third receiving module 301 is used to receive a first image file sent by the test party server. The first image file is formed by encapsulating a test data set and a test environment in an application container engine. The test data set is used to test the deep learning model. The test environment refers to the software and hardware environment and development environment used when testing the deep learning model. For details, please refer to the relevant description in Example 1, which will not be repeated here.

[0109] The second testing module 302 is used to pre-test the deep learning model based on the first image file; for details, please refer to the relevant description in Example 1, which will not be repeated here.

[0110] The encapsulation module 303 is used to encapsulate the first image file and the deep learning model into a second image file when the deep learning model passes the pre-test; for details, please refer to the relevant description in Example 1, which will not be repeated here.

[0111] The third calculation module 304 is used to calculate the information digest of the second image file to obtain a third digest value; for details, please refer to the relevant description in Example 1, which will not be repeated here.

[0112] The third sending module 305 is used to send the second image file and the third digest value to the test party server, and the third digest value is used by the test party server to determine whether to test the deep learning model in the second image file based on the first image file. For details, please refer to the relevant description in Example 1, which will not be repeated here.

[0113] In one or more embodiments, the device further includes: a third acquisition module ( Figure 3 ) and a display module ( Figure 3 not shown).

[0114] The third acquisition module is used to obtain the user portrait corresponding to the target user account, and the user portrait is used to indicate the test preference of the target user account; for details, please refer to the relevant description in Example 1, which will not be repeated here.

[0115] The display module is used to display the target test task matching the user portrait when the target user account logs in to the test platform server. For details, please refer to the relevant description in Example 1, which will not be repeated here.

[0116] The effect of the testing device for the deep learning model of this embodiment can be found in the relevant description in Embodiment 1 and will not be elaborated here.

[0117] Example 4

[0118] This embodiment provides a deep learning model testing system. Figure 4 As shown, the testing system of the deep learning model includes a testing platform server and a testing party server.

[0119] The test platform server includes a front end 101, a back end 102, a database 103 and a remote management unit 104. The test party server includes an operation unit 105, a storage unit 106, a configuration file 107, a listener 108 and an image warehouse 109. The listener 108 listens to the image warehouse 109 at a fixed frequency, and the operation unit 105 communicates with the listener 108 to receive an instruction to download the second image file.

[0120] The tester creates a new test task on the front end 101 of the test platform server. The front end 101 receives the data of the test task and transmits the data of the test task to the back end 102. The data of the test task can be the start time, end time, task description, submission authority, test standard, test data set download link, submission description, etc. of the test task. The back end 102 reviews the test task, and after the review is passed, the test task is saved to the database 103 and published on the page of the front end 101. Reviewing the test task includes manual review and automatic review. The manual review can be conducted by the administrator logging into the back end and manually reviewing the test task; the automatic review can preset the review template on the test platform server and complete the automatic matching review. The tester uploads the configuration file corresponding to the test task, which contains the IP address of the selected remotely evaluated tester server and the preset value of the concurrent number of resources of its running unit 105. The configuration file is passed in by the front end 101 and saved in the database 103.

[0121] The test party uploads the first image file corresponding to the test task to the image warehouse 109 of the test party server, and uploads the image source code to the Gitlab public address for download. In addition, the test party generates a first summary value based on the first image file corresponding to the test task and saves it in the storage unit 106. On the other hand, according to the registered user preferences and / or historical test task records of the tested party, the test platform server marks the test task category for the tested party, forms the user portrait of the tested party, and saves the user portrait of the tested party in the database 103. When the tested party logs in to the test platform server, the back end 102 calls the user information and user portrait of the database 103, intelligently recommends the test tasks that match the user portrait in the test platform server and displays them on the front end 101. And when the test platform server has a new test task that matches the user portrait of the tested party, the test platform server can notify the tested party by SMS or email.

[0122] The tested party views the test task on the front end 101, chooses to join the test task, and downloads the first image file corresponding to the test task from the image warehouse 109. Based on the first image file of the test task, the tested party uses the first image file to evaluate the trained deep learning model on the test platform server, and after passing the pre-test, produces a second image file containing the deep learning model and the first image file, performs information summary calculation on the second image file to generate a third summary value, and uploads the second image file and the third summary value to the image warehouse 109 to ensure the security and privacy of the tested model.

[0123] The first image file includes a test data set, a test environment, a test estimated result, a remote call interface definition, etc. If the second summary value of the first image file in the second image file uploaded by the tested party is inconsistent with the corresponding first summary value, it means that the tested party has tampered with the test data set, test environment, test estimated result, remote call interface definition, etc. of the first image file when performing pre-testing on the test platform server. The running unit 105 transmits the verification result to the back end 102 of the test platform server through the remote call interface, and the back end 102 notifies the tested party by SMS or email. The tested party can choose to repackage the image file of the tested model or terminate the test task.

[0124] If the second digest value of the first image file in the second image file uploaded by the tested party is consistent with the corresponding first digest value, the information digest of the second image file is calculated to obtain a fourth digest value. If the second digest value and the fourth digest value are inconsistent, it means that the second image file to be run and the second image file uploaded by the user are tampered with. The running unit 105 transmits the verification result to the back end 102 of the test platform server through the remote call interface, and the back end 102 notifies the tested party by SMS or email. The tested party can choose to repackage the image file of the tested model or terminate this test task. Among them, the remote call interface can include a restful interface.

[0125] When the tested party uploads and downloads image files, it can use command line code or client. For example, the Docker client has the functions of retrieving, adding, modifying, and deleting image files. When uploading image files through the client, the parameters of this test task can also be configured synchronously, such as the test task ID and the stage of the test task.

[0126] The tester can run the remote management unit 104, which reads the test task configuration file (ie, the configuration file of the test platform server) from the database 103, and uses the SSH command to remotely modify the configuration file 107 in the tester's server.

[0127] The listener 108 of the test server reads the configuration file 107 and updates the concurrent number of the running resources of the running unit 105 in real time. The listener 108 monitors the current running resource allocation of the running unit 105 in real time. When the running resources of the test server are all occupied, it enters a waiting state; when the running resources of the test server are idle, the listener 108 issues a running instruction to the running unit 105. The running unit 105 downloads and runs the second image file from the image warehouse 109, tests the deep learning model in the second image file, and saves the test results generated by the running unit 105 to the storage unit 106. For example, Figure 4 As shown, a plurality of second image files are run in the running unit 105, and the first tested model 1051, the second tested model 1052, etc. are tested at the same time.

[0128] Optionally, the first image file also includes a remote call interface. During the test process, the running unit 105 of the test party server can send the test process (including one or more of the test progress, test log, etc.) to the back end 102 of the test platform server through the remote call interface in real time, and then the back end 102 notifies the party being tested by SMS or email.

[0129] The storage unit 106 continuously collects and saves the test results of the deep learning model, and the remote management unit 104 monitors the operation unit 105 in real time. If a new test result is found, the operation unit 105 reads the test result and sends the test result to the back end 102 through the remote call interface. The test results can be displayed on the front end 101, and the test results (including but not limited to the test start time, end time, duration, ranking, etc.) can be notified to the tested party by SMS or email.

[0130] For the image classification test task, the test results include accuracy, misclassification rate, precision rate, recall rate, F1 value, ROC (receiver operating characteristic) curve, AUC (Area Under Curve, the area under the ROC curve and the coordinate axis) value, PR (Precision Recall) curve, etc.; for the object detection test task, the test results include mAP (mean Average Precision, class average accuracy), COCO evaluation index, accuracy, misclassification rate, precision rate, recall rate, F1 value, ROC curve, AUC value, PR curve, etc.

[0131] Example 5

[0132] This embodiment provides a server, such as Figure 5 As shown, the server includes a processor 401 and a memory 402, wherein the processor 401 and the memory 402 may be connected via a bus or other means. Figure 5 The example of connecting through bus is taken in the following.

[0133] The processor 401 may be a central processing unit (CPU). The processor 401 may also be other general-purpose processors, digital signal processors (DSP), graphics processors (GPU), embedded neural network processors (NPU), or other dedicated deep learning coprocessors, application specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0134] The memory 402, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as the program instructions / modules corresponding to the test method of the deep learning model on the test party server side or the test method of the deep learning model on the test platform server side in the embodiments of the present application (such as the first receiving module 201, the first calculation module 202, the second calculation module 203, the first acquisition module 204 and the first test module 205 in the above-mentioned embodiments; the first sending module and the second sending module; the second acquisition module and the detection module; the second receiving module and the modification module; or the third receiving module 301, the second test module 302, the encapsulation module 303, the third calculation module 304 and the third sending module 305; the third acquisition module and the display module in the above-mentioned embodiments). The processor 401 executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory 402, that is, implements the test method of the deep learning model on the test party server side or implements the test method of the deep learning model on the test platform server side.

[0135] The memory 402 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created by the processor 401, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 402 may optionally include a memory remotely arranged relative to the processor 401, and these remote memories may be connected to the processor 401 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0136] The embodiment of the present application also provides a computer-readable storage medium, which stores computer-executable instructions, which can execute the test method of the deep learning model on the server side of the above-mentioned test party, or execute the test method of the deep learning model on the server side of the above-mentioned test platform. Wherein, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0137] An embodiment of the present application also provides a computer program product. When the computer program product runs on a server, it implements the testing method of the deep learning model on the server side of the test party in Example 1, or implements the testing method of the deep learning model on the server side of the test platform in Example 1.

[0138] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection created by this application.

Claims

1. A testing method for deep learning models, It is characterized in that Applied to the testing party server, the method includes: Receive a second image file and a third digest value sent by the test platform server; the second image file is obtained by encapsulating the first image file and the trained deep learning model after the test platform server pre-tests the trained deep learning model based on the first image file; the third digest value refers to the digest value obtained by the test platform server performing information digest calculation on the second image file; Performing information digest calculation on the first image file in the second image file to obtain a second digest value; Performing information digest calculation on the second image file to obtain a fourth digest value; Obtaining a first digest value, where the first digest value refers to a digest value pre-stored by the testing party server and obtained by performing information digest calculation on the first image file; When the first digest value is the same as the second digest value, and the third digest value is the same as the fourth digest value, testing the deep learning model in the second image file based on the first image file to obtain a test result of the deep learning model; The first image file is formed by encapsulating a test data set and a test environment in an application container engine, the test data set is used to test the deep learning model, and the test environment refers to the software and hardware environment and development environment used when testing the deep learning model; The first image file also encapsulates a test estimation result. When the test result of the pre-test reaches the test estimation result, the deep learning model passes the pre-test; when the test result of the pre-test does not reach the test estimation result, the deep learning model fails the pre-test; Before testing the deep learning model in the second image file based on the first image file to obtain the test result of the deep learning model, the method further includes: Obtaining a configuration file, wherein the configuration file is used to determine the concurrent number of running resources of the test party server; Detecting the operating resource occupancy status of the testing party server; The testing the deep learning model in the second image file based on the first image file to obtain a test result of the deep learning model includes: When the running resource occupancy status does not reach the running resource concurrency number, the deep learning model in the second image file is tested based on the first image file to obtain a test result of the deep learning model.

2. The method for testing a deep learning model as claimed in claim 1, It is characterized in that The method further comprises: When the first digest value and the second digest value are different, sending first error information to the test platform server, where the first error information is used to indicate that the first image file has been tampered with by the tested party; When the third digest value and the fourth digest value are different, second error information is sent to the test platform server, where the second error information is used to indicate that the second image file is tampered with by a third party.

3. The method for testing a deep learning model as claimed in claim 1, It is characterized in that The method further comprises: Receiving a file modification instruction sent by the test platform server; Modify the concurrent number of running resources according to the file modification instruction to obtain a modified concurrent number of running resources, and the modified concurrent number of running resources is used to determine whether to test the deep learning model in the second image file based on the first image file.

4. A testing method for deep learning models, It is characterized in that Applied in a test platform server, the method comprises: Receive a first image file sent by a test party server, wherein the first image file is formed by encapsulating a test data set and a test environment in an application container engine, wherein the test data set is used to test a deep learning model, and the test environment refers to a software and hardware environment and a development environment used when testing the deep learning model; the first image file also encapsulates a test estimation result, and when the test result of the pretest reaches the test estimation result, the deep learning model passes the pretest; when the test result of the pretest does not reach the test estimation result, the deep learning model fails the pretest; pretest the deep learning model based on the first image file; If the deep learning model passes the pre-test, encapsulating the first image file and the deep learning model into a second image file; Performing information digest calculation on the second image file to obtain a third digest value; Sending the second image file and the third digest value to the test party server, so that the test party server tests the deep learning model in the second image file based on the first image file to obtain the test result of the deep learning model, and the third digest value is used by the test party server to determine whether to test the deep learning model in the second image file based on the first image file; before testing the deep learning model in the second image file based on the first image file to obtain the test result of the deep learning model, the test party server also obtains a configuration file and detects the running resource occupancy status of the test party server; the configuration file is used to determine the running resource concurrency number of the test party server; testing the deep learning model in the second image file based on the first image file to obtain the test result of the deep learning model includes: when the running resource occupancy status does not reach the running resource concurrency number, testing the deep learning model in the second image file based on the first image file to obtain the test result of the deep learning model; The method further comprises: Obtaining a user portrait corresponding to a target user account, wherein the user portrait is used to indicate a test preference of the target user account; When the target user account logs in to the test platform server, the target test task matching the user portrait is displayed.

5. A test device for deep learning models, It is characterized in that The device comprises: A first receiving module is used to receive a second image file and a third digest value sent by a test platform server; the second image file is obtained by encapsulating the first image file and the trained deep learning model after the test platform server pre-tests the deep learning model based on the first image file; the third digest value refers to the digest value obtained by the test platform server performing information digest calculation on the second image file; A first calculation module, configured to perform information digest calculation on the first image file in the second image file to obtain a second digest value; A second calculation module, configured to perform information digest calculation on the second image file to obtain a fourth digest value; A first acquisition module, configured to acquire a first digest value, where the first digest value refers to a digest value pre-stored by the test party server and obtained by performing information digest calculation on the first image file; a first testing module, configured to test the deep learning model in the second image file based on the first image file to obtain a test result of the deep learning model when the first digest value is the same as the second digest value and the third digest value is the same as the fourth digest value; The first image file is formed by encapsulating a test data set and a test environment in an application container engine, wherein the test data set is used to test the deep learning model, and the test environment refers to the software and hardware environment and the development environment used when testing the deep learning model; the first image file also encapsulates a test estimation result, and when the test result of the pre-test reaches the test estimation result, the deep learning model passes the pre-test; when the test result of the pre-test does not reach the test estimation result, the deep learning model fails the pre-test; The device further comprises: a second acquisition module and a detection module; The second acquisition module is used to acquire a configuration file, wherein the configuration file is used to determine the concurrent number of running resources of the test party server; The detection module is used to detect the operating resource occupancy status of the test party server; The first test module is further used for: When the running resource occupancy status does not reach the running resource concurrency number, the deep learning model in the second image file is tested based on the first image file to obtain a test result of the deep learning model.

6. A test device for deep learning models, It is characterized in that The device comprises: A third receiving module is used to receive a first image file sent by the test party server, wherein the first image file is formed by encapsulating a test data set and a test environment in an application container engine, wherein the test data set is used to test a deep learning model, and the test environment refers to a software and hardware environment and a development environment used when testing the deep learning model; the first image file also encapsulates a test estimation result, and when the test result of the pre-test reaches the test estimation result, the deep learning model passes the pre-test; when the test result of the pre-test does not reach the test estimation result, the deep learning model fails the pre-test; A second testing module, used to pre-test the deep learning model based on the first image file; A packaging module, used for packaging the first image file and the deep learning model into a second image file if the deep learning model passes the pre-test; A third calculation module, configured to perform information digest calculation on the second image file to obtain a third digest value; a third sending module, configured to send the second image file and the third digest value to the test party server, so that the test party server tests the deep learning model in the second image file based on the first image file to obtain the test result of the deep learning model, and the third digest value is used by the test party server to determine whether to test the deep learning model in the second image file based on the first image file; before testing the deep learning model in the second image file based on the first image file to obtain the test result of the deep learning model, the test party server is also configured to obtain a configuration file and detect the running resource occupancy status of the test party server; the configuration file is used to determine the running resource concurrency number of the test party server; testing the deep learning model in the second image file based on the first image file to obtain the test result of the deep learning model includes: when the running resource occupancy status does not reach the running resource concurrency number, testing the deep learning model in the second image file based on the first image file to obtain the test result of the deep learning model; The device further comprises: a third acquisition module and a display module; The third acquisition module is used to acquire a user portrait corresponding to the target user account, where the user portrait is used to indicate a test preference of the target user account; The display module is used to display the target test task matching the user portrait when the target user account logs in to the test platform server.

7. A testing system for deep learning models, It is characterized in that Including test platform server and test party server; The test party server is used to send a first image file to the test platform server. The first image file is formed by encapsulating a test data set and a test environment in an application container engine. The test data set is used to test a deep learning model. The test environment refers to the software and hardware environment and the development environment used when testing the deep learning model. The first image file also encapsulates a test estimation result. When the test result of the pre-test reaches the test estimation result, the deep learning model passes the pre-test; when the test result of the pre-test does not reach the test estimation result, the deep learning model fails the pre-test. The test platform server is used to pre-test the deep learning model based on the first image file; If the deep learning model passes the pre-test, encapsulating the first image file and the deep learning model into a second image file; Performing information digest calculation on the second image file to obtain a third digest value; Sending the second image file and the third digest value to the testing party server, where the third digest value is used by the testing party server to determine whether to test the deep learning model in the second image file based on the first image file; The testing party server is used to calculate the information digest of the first image file in the second image file to obtain a second digest value; performing information digest calculation on the second image file to obtain a fourth digest value; obtaining a first digest value, wherein the first digest value refers to a digest value obtained by performing information digest calculation on the first image file and stored in advance by the test party server; when the first digest value is the same as the second digest value, and the third digest value is the same as the fourth digest value, testing the deep learning model in the second image file based on the first image file to obtain a test result of the deep learning model; before testing the deep learning model in the second image file based on the first image file to obtain the test result of the deep learning model, the test party server is further used to obtain a configuration file and detect the running resource occupancy status of the test party server; the configuration file is used to determine the running resource concurrency number of the test party server; testing the deep learning model in the second image file based on the first image file to obtain the test result of the deep learning model includes: when the running resource occupancy status does not reach the running resource concurrency number, testing the deep learning model in the second image file based on the first image file to obtain the test result of the deep learning model; The test platform server is also used to obtain a user portrait corresponding to the target user account, and the user portrait is used to indicate the test preference of the target user account; when the target user account logs in to the test platform server, the target test task matching the user portrait is displayed.

8. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer instructions, which are used to enable a computer to execute the testing method for the deep learning model described in any one of claims 1 to 3, or to execute the testing method for the deep learning model described in claim 4.