An automated testing method and device based on a drag-and-drop machine learning platform

By crawling the request body and the response body to generate test cases, setting the running environment and dependencies, executing the test script regularly and automatically correcting the expected values, it solves the problem of low efficiency and accuracy of task flow testing on the dragged machine learning platform, and achieves high efficiency Accurate automated testing.

CN113360408BActive Publication Date: 2024-12-06INDUSTRIAL AND COMMERCIAL BANK OF CHINA
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202110799046.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-07-15
Publication Date
2024-12-06
Estimated Expiration
2041-07-15

AI Technical Summary

Technical Problem

The automated testing methods of existing drag machine learning platforms are inefficient and have low accuracy, and cannot effectively cover complex task flows and model evaluation verification, especially when faced with a variety of visual interfaces and a large number of components, manual testing is difficult to follow. The pace of development on the platform.

Method used

By crawling the request body and the response body to generate test cases, setting up the running environment and dependencies, executing the test scripts regularly, and automatically correcting the expected values ​​based on the execution results, achieving efficient and accurate automated testing of the task flow of the dragged machine learning platform.

Benefits of technology

It improves the accuracy and efficiency of the test, realizes efficient and accurate automated testing of task flow for dragged machine learning platforms, and solves the problem that traditional manual testing is difficult to keep up with the development of the platform.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113360408B_ABST
    Figure CN113360408B_ABST
Patent Text Reader

Abstract

The present application provides an automated testing method and device based on a drag-and-drop machine learning platform, wherein the method includes: generating a test case for a test object according to the request body and response body of a modeled DGA task flow; setting a running environment, dependencies, and serial and parallel modes for the test case; executing the test script regularly to obtain the execution result of the test case; comparing the execution result of the test case with the expected result; and automatically correcting the expected value of the case according to the comparison result and the automatic correction judgment logic. The above scheme solves the problems of low test efficiency and low accuracy in the existing automated testing of the task flow formed by the drag-and-drop machine learning platform, and achieves the purpose of efficient and accurate automated testing of the task flow formed by the drag-and-drop machine learning platform.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and in particular, relates to an automated testing method and device based on a drag-and-drop machine learning platform. Background Art

[0002] At present, with the continuous development of artificial intelligence technology, in order to lower the threshold of machine learning, the concept of machine learning platform has been proposed, among which the drag-and-drop modeling method is the main machine learning platform mode. The drag-and-drop machine learning platform integrates a wealth of machine learning components (operators) and algorithms, and provides a friendly visual interface. Modeling users can build complex modeling task flows by simply dragging components. A set of modeling task flows built by dragging multiple components can be called a DAG. A complete DAG task flow includes: data set introduction, data set processing (multiple), data set splitting, feature extraction, model training and parameter adjustment, model evaluation, model effect evaluation, model publishing and model reasoning.

[0003] For the testing of task flows based on machine learning platforms, it is necessary to cover the logic testing of each component, the logic testing of multiple components combined, the single-branch and multi-branch testing of task flows, and the parallel execution testing of multiple task flows. In addition to the testing of task flows, it also includes the combined verification of algorithm parameters, the evaluation and verification of the algorithm model effect, the verification of the model publishing process, and the verification of the model reasoning service. For these test contents, if the testers use manual testing, they will inevitably not be able to keep up with the pace of platform development. On the other hand, the drag-and-drop machine learning platform has a visual interface, and the traditional WEB automated testing method is mainly based on the means of page element positioning to truly simulate user operations. However, in the drag-and-drop interface, the position of elements is changeable and unstable, and it is extremely difficult to simulate mouse clicks, especially for the large number of components in the modeling DAG, which is time-consuming and labor-intensive both for the positioning of elements and for the development of automated testing code.

[0004] Currently, there is no effective solution for how to automate the testing of the task flow formed by the drag-and-drop machine learning platform. Summary of the invention

[0005] The purpose of this application is to provide an automated testing method and device based on a drag-and-drop machine learning platform, which can achieve the purpose of automated testing of task flows formed based on the drag-and-drop machine learning platform.

[0006] The present application provides an automated testing method and device based on a drag-and-drop machine learning platform, which is implemented as follows:

[0007] An automated testing method based on a drag-and-drop machine learning platform, the method comprising:

[0008] Generate test cases for the test object based on the request body and response body of the modeled DGA task flow;

[0009] Setting the operating environment, dependencies and serial and parallel modes for the test cases;

[0010] Execute the test script regularly to obtain the execution results of the test case;

[0011] Compare the test case execution results with the expected results;

[0012] According to the comparison results and automatic correction judgment logic, the expected value of the case is automatically corrected.

[0013] In one embodiment, the test object includes the execution of at least one of the following component tasks: data introduction, data processing, data splitting, feature extraction, model training, batch estimation, model publishing, and model reasoning.

[0014] In one embodiment, generating a test case of a test object according to a request body and a response body of a modeled DGA task flow includes:

[0015] Automatically intercept the request and response bodies of the DAG task flow initiated by the user when modeling on the machine learning platform;

[0016] The request body and response body are named by concatenating the request URL with the timestamp.

[0017] The URL, request method and request parameters in the request body, as well as the response body, are used as variable values ​​used in the test script and are filled into each column of EXCEL row by row as a test case.

[0018] In one embodiment, setting the operating environment, dependency, and serial and parallel modes for the test case includes:

[0019] Dividing the parameters of the test case into fixed values ​​and variable values;

[0020] Parameterizing the variable value, wherein the parameterized variable value is obtained in real time by interacting with the database;

[0021] Set the switch value of automatic correction judgment in the test case;

[0022] The values ​​of the core indicators in the response body are taken as the case expected values;

[0023] Set the tasks with dependencies in the DAG task flow to be executed serially, and set the tasks at the same level in the DAG task flow to be executed in parallel.

[0024] In one embodiment, the test script is executed regularly to obtain the execution result of the test case, including:

[0025] Determine whether the preset trigger condition is met;

[0026] When it is determined that the preset trigger conditions are met, the test script is triggered to execute;

[0027] Determine whether there is a dependent script. If there is a dependent script, continue to wait for the dependent script to be successfully executed before initiating execution;

[0028] During the script execution, when the test case is executed serially, a single case is read; when the test case is executed in parallel, multiple cases are read in batches, wherein the number of cases read in parallel is determined according to the preset number of concurrent cases.

[0029] In one embodiment, the script execution process also includes:

[0030] Call the request class in the test code according to the request method of the test object in the test case;

[0031] According to the called request class, the execution end judgment of the test object is performed;

[0032] In the case of synchronous execution, the response body is used as the end mark;

[0033] In the case of asynchronous execution, the component execution status is polled by interacting with the database to determine whether the task execution is completed.

[0034] In one embodiment, the case expected value is automatically corrected according to the comparison result and the automatic correction judgment logic, including:

[0035] Get the autocorrect switch value for the test case;

[0036] When the automatic correction switch value is on, the task is executed successfully and the execution result is inconsistent with the expected result, the expected value of the test case is replaced with the obtained test result;

[0037] When the autocorrection switch value is off, or the task execution fails, the expected value of the test case is not updated.

[0038] An automated testing device based on a drag-and-drop machine learning platform, comprising:

[0039] A generation module is used to generate test cases for the test object according to the request body and response body of the modeled DGA task flow;

[0040] A setting module, used to set the operating environment, dependency and serial and parallel mode for the test case;

[0041] The execution module is used to execute the test script regularly to obtain the execution result of the test case;

[0042] The comparison module is used to compare the execution results of the test cases with the expected results;

[0043] The correction module is used to automatically correct the expected value of the case based on the comparison results and the automatic correction judgment logic.

[0044] An electronic device includes a processor and a memory for storing instructions executable by the processor, wherein when the processor executes the instructions, the steps of the following method are implemented:

[0045] Generate test cases for the test object based on the request body and response body of the modeled DGA task flow;

[0046] Setting the operating environment, dependencies and serial and parallel modes for the test cases;

[0047] Execute the test script regularly to obtain the execution results of the test case;

[0048] Compare the test case execution results with the expected results;

[0049] According to the comparison results and automatic correction judgment logic, the expected value of the case is automatically corrected.

[0050] A computer-readable storage medium having computer instructions stored thereon, wherein the instructions, when executed, implement the steps of the following method:

[0051] Generate test cases for the test object based on the request body and response body of the modeled DGA task flow;

[0052] Setting the operating environment, dependencies and serial and parallel modes for the test cases;

[0053] Execute the test script regularly to obtain the execution results of the test case;

[0054] Compare the test case execution results with the expected results;

[0055] According to the comparison results and automatic correction judgment logic, the expected value of the case is automatically corrected.

[0056] The automated testing method based on the drag-and-drop machine learning platform provided in the present application generates test cases that conform to the script operation by capturing the request body and the response body, without the need for manual packet capture and recording, and establishes the operating environment, dependencies, and serial and parallel methods of the test cases, and triggers the execution of the test scripts at a regular interval to realize the execution of the test cases, and automatically corrects the expected values ​​of the cases based on the execution results, thereby realizing error reporting and adjustment of the script execution and improving the accuracy of the test. Through the above scheme, the problems of low test efficiency and low accuracy in the existing automated testing of the task flow formed by the drag-and-drop machine learning platform are solved, and the purpose of efficient and accurate automated testing of the task flow formed by the drag-and-drop machine learning platform is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0058] Figure 1 It is a method flow chart of an embodiment of an automated testing method based on a drag-and-drop machine learning platform provided by the present application;

[0059] Figure 2 It is a flowchart of a continuous testing method for machine learning modeling based on a DAG graph provided by this application;

[0060] Figure 3 It is a schematic diagram of a module for generating a test object provided by this application;

[0061] Figure 4 It is a test case management diagram provided by this application;

[0062] Figure 5 This is a test case execution optimization combination example diagram of a machine learning modeling DAG task flow provided by this application;

[0063] Figure 6 It is a test script management diagram provided by this application;

[0064] Figure 7 It is a schematic diagram of the test script execution provided by this application;

[0065] Figure 8 It is a schematic diagram for comparing the execution results provided by this application;

[0066] Fig. 9 It is a schematic diagram of automatic correction judgment provided by this application;

[0067] Fig.10 The hardware structure block diagram of an electronic device provided by the present application for an automated testing method based on a drag-and-drop machine learning platform;

[0068] Fig.11 It is a schematic diagram of the module structure of an embodiment of an automated testing device based on a drag-and-drop machine learning platform provided in the present application. DETAILED DESCRIPTION

[0069] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0070] Figure 1 It is a method flow chart of an embodiment of an automated testing method based on a drag-and-drop machine learning platform provided by the present application. Although the present application provides method operation steps or device structures as shown in the following embodiments or drawings, more or fewer operation steps or module units may be included in the method or device based on routine or no creative labor. In the steps or structures where there is no necessary causal relationship logically, the execution order of these steps or the module structure of the device is not limited to the execution order or module structure described in the embodiments of the present application and shown in the drawings. When the method or module structure described is applied to an actual device or terminal product, it can be connected according to the method or module structure shown in the embodiment or drawings for sequential execution or parallel execution (for example, a parallel processor or a multi-threaded processing environment, or even a distributed processing environment).

[0071] Specifically, Figure 1 As shown, the above-mentioned automated testing method based on the drag-and-drop machine learning platform may include the following steps:

[0072] Step 101: Generate a test case for the test object according to the request body and response body of the modeled DGA task flow;

[0073] Among them, the test object may include but is not limited to the execution of at least one of the following component tasks: data introduction, data processing, data splitting, feature extraction, model training, batch estimation, model publishing, and model reasoning.

[0074] When generating test cases, the following steps can be included:

[0075] Step 1: Automatically intercept the request body and response body of the DAG task flow initiated by the user when modeling on the machine learning platform;

[0076] That is, user operations are automatically intercepted, so there is no need for manual packet capture and recording.

[0077] Step 2: Name the request body and response body by concatenating the request URL with the timestamp;

[0078] Step 3: Use the URL, request method and request parameters in the request body, as well as the response body, as variable values ​​used in the test script, and fill them into each column of EXCEL row by row as a test case.

[0079] Step 102: Setting the operating environment, dependency, and serial and parallel modes for the test case;

[0080] Specifically, the parameters of the test case can be divided into fixed values ​​and variable values; the variable values ​​are parameterized, wherein the parameterized variable values ​​are obtained in real time by interacting with a database; the switch value of the automatic correction judgment in the test case is set; the value of the core indicator in the response body is used as the expected value of the case; the tasks with dependencies in the DAG task flow are set to be executed serially, and the tasks at the same level in the DAG task flow are set to be executed in parallel.

[0081] That is, set serial parallel execution, and set the automatic correction judgment switch and case expected value, so that when there are test results, it can automatically trigger the adjustment of the expected value based on the assertion.

[0082] Step 103: Execute the test script regularly to obtain the execution result of the test case;

[0083] Step 104: Compare the execution result of the test case with the expected result;

[0084] Step 105: Automatically correct the case expected value based on the comparison result and the automatic correction judgment logic.

[0085] In the above example, the automated testing method based on the drag-and-drop machine learning platform generates test cases that conform to the script operation by capturing the request body and response body, without the need for manual packet capture and recording, and establishes the test case's operating environment, dependencies, and serial and parallel methods. It triggers the execution of the test script regularly to implement the execution of the test case, and automatically corrects the case's expected value based on the execution result, thereby implementing error reporting and adjustment of the script execution and improving the accuracy of the test. Through the above method, the existing problems of low test efficiency and low accuracy in the automated testing of the task flow formed by the drag-and-drop machine learning platform are solved, and the purpose of efficient and accurate automated testing of the task flow formed by the drag-and-drop machine learning platform is achieved.

[0086] In the specific implementation, the test script is executed regularly to obtain the execution result of the test case, which may include: determining whether the preset trigger condition is met; triggering the execution of the test script when it is determined that the preset trigger condition is met; judging whether there is a dependent script, and when there is a dependent script, continuing to wait for the dependent script to be successfully executed before initiating execution; during the script execution process, when the test case is executed serially, a single case is read, and when the test case is executed in parallel, multiple cases are read in batches, wherein the number of parallel reads is determined according to the preset number of concurrent cases. That is, a trigger condition (for example, whether the time is reached) can be set to trigger the automatic execution of the test script. When executing the test script, it can be determined whether it is executed in parallel or serially according to the actual situation of the system, and different numbers of trigger cases can be selected for different execution modes, so that the test case can be executed in a targeted manner.

[0087] Considering that a termination judgment is required during the execution of the test script, otherwise the test script will continue to run, which is obviously unreasonable. For this reason, during the execution of the script, the request class in the test code can be called according to the request method of the test object in the test case; according to the called request class, the execution end of the test object is judged; in the case of synchronous execution, the response body is obtained as the end mark; in the case of asynchronous execution, the component execution status is polled by interacting with the database to determine whether the task execution is completed.

[0088] When automatically correcting the expected value of the case according to the comparison result and the automatic correction judgment logic, the automatic correction switch value of the test case can be obtained; when the automatic correction switch value is turned on, the task is successfully executed, and the execution result is inconsistent with the expected result, the expected value of the test case is replaced with the obtained test result; when the automatic correction switch value is turned off, or the task fails to execute, the expected value of the test case is not updated. That is, whether to update the expected value of the test case is determined based on the automatic correction switch value, whether the execution is successful, and whether it is consistent with the expected result.

[0089] The above method is described below in conjunction with a specific embodiment. However, it should be noted that this specific embodiment is only for better illustrating the present application and does not constitute an improper limitation on the present application.

[0090] In order to solve the problems of difficult positioning of automated test elements and cumbersome script development in the DAG task flow mode of machine learning modeling, this example provides a continuous testing device implementation solution based on DAG machine learning modeling. Starting from the interface of front-end and back-end interaction, the user cookies are saved through login transactions to truly simulate the process of users building and executing DAG task flows online. The device can automatically collect test data sets, automatically execute DAG task flows, automatically compare test results with expected results, and automatically correct test cases according to the automatic correction switch. Deploy the test script to Jenkins to continuously guard and monitor the quality of the machine learning platform. This method can improve the testing efficiency of the drag-and-drop task flow, and is also suitable for modelers to monitor the model iteration effect.

[0091] Specifically, in this example, an automated verification method based on DAG machine learning modeling is provided, including the following steps:

[0092] Step 1: Automatically obtain the request body and response body of the modeling DAG task flow and generate test objects. The test objects can be: data introduction, data processing, data splitting, feature extraction, model training, batch estimation, model publishing, model reasoning and other component tasks, and the viewing of execution results.

[0093] Step 2: Refine the test cases of the above test objects, make non-fixed parameter values ​​variable, simplify the expected values, and configure whether to enable automatic correction of expected values.

[0094] Step 3: Configure the test running environment, the dependencies of each test object in the DAG task flow test script, and the serial / parallel execution mode.

[0095] Step 4: Deploy the test script to Jenkins and trigger the execution on a scheduled basis. In this mode, by configuring testng.xml, parallel testing of multiple DAG task flows can be supported.

[0096] Step 5: The subscriber views the test execution result report.

[0097] Specifically, the above test method can be as follows Figure 2 As shown, including:

[0098] S1: Generate test object:

[0099] Automatically generate basic test cases for the test object based on the request body and response body of the modeled DAG task flow.

[0100] S2: Manage test cases:

[0101] Refine the basic test cases generated in S1, including: setting the login user, parameterizing some variable parameter values, optimizing the response body as the expected value, setting the automatic correction switch value, etc.

[0102] S3: Manage test scripts:

[0103] Configure the running environment, test script dependencies, test data, and parallel execution of test scripts.

[0104] S4: Execute the test script, which may include two parts: test case driving test script execution and execution completion confirmation.

[0105] S5: Compare execution results:

[0106] Based on S4, compare the test case execution results with the expected results to see if they are consistent.

[0107] S6: Determine whether to automatically correct:

[0108] According to the execution results of S5, combined with the automatic correction judgment logic, automatic correction judgment of the expected results of the case is performed.

[0109] S7: Perform automatic correction:

[0110] If the execution result is inconsistent with the expected result and automatic correction is enabled for the case, the expected value of the case is updated and the case is re-executed until the case is consistent with the expected result.

[0111] S8: Generate test report:

[0112] After the test case is executed, a test report is automatically generated and pushed to the subscriber.

[0113] The specific descriptions of S1 to S8 are as follows:

[0114] like Figure 3 As shown, it is a schematic diagram of generating a test object, which may specifically include:

[0115] S101: Capture DAG task flow interface request;

[0116] After setting the request body and response body matching rules for initiating tasks and viewing results for each component in the task flow, as well as the file name and save path of the landing file in the parameter file of the packet capture tool or packet capture script, use the packet capture tool / packet capture script to automatically intercept the request body and response body of the DAG task flow initiated by the user when modeling on the machine learning platform.

[0117] S102: Implement the task flow subtask interface request body and response body in sequence;

[0118] Each subtask in the task flow intercepted in S101 is named by concatenating the request URL with the timestamp, and the request content and the response content are written into a local file.

[0119] S103: Generate a basic test case for the DAG task flow.

[0120] The URL, request method, request parameters and response body in the request body are used as variable values ​​used in the test script, and are filled into each column of EXCEL row by row, as a basic test case for use in managing test cases.

[0121] like Figure 4 The figure shows a schematic diagram of test case management, which may include:

[0122] S201: parameterize the number of tests of the test object;

[0123] Based on the generated basic test cases, in order to support repeated execution of the cases, the parameters are subdivided into fixed values ​​and variable values. The parameter values ​​that change each time are parameterized and obtained in real time through interaction with the database. The switch value of the automatic correction judgment in the test case is set to determine whether to automatically update the case.

[0124] S202: Simplified assertions or customized assertions;

[0125] Assertions obtain core elements from the response body of S1 as assertion points, or customize assertions. For example, the value of the core indicator in the model evaluation report is used as the expected value. If it is greater than this expected value, the model effect is considered qualified. If it is less than this expected value, the model effect is considered to be degraded.

[0126] S203: Case execution optimal combination configuration.

[0127] The DAG task flow subtasks are divided into serial and parallel execution. Tasks with dependencies in the DAG task flow are set to be executed serially, and tasks at the same level in the DAG task flow are set to be executed in parallel.

[0128] like Figure 5 As shown in the figure, it is an example of test case execution optimization combination of a machine learning modeling DAG task flow. The horizontal direction is parallel execution of components, such as: introduction of data source 1 to data source 4; execution of data processing 1, data processing 2 and data processing 3; execution of feature extraction and feature extraction (batch) components; viewing of model training set evaluation, model validation set evaluation and model test set evaluation; the vertical direction is serial execution of components, such as: data processing 4 must be executed after data processing 1 to 3 are successfully executed. Data splitting depends on the successful execution of data processing 4; model reasoning depends on the release of the model, etc.

[0129] like Figure 6 The figure shows a schematic diagram of managing test scripts, which may include:

[0130] 1) Through configuration management, set up the script running environment to facilitate cross-environment testing.

[0131] 2) Excel data driven, used to separate test data and test scripts. The test data in the Excel table maintained in the management test case drives the execution of the test script.

[0132] 3) The test code in the test script can be divided into: login tool class, request tool class, database interaction class, asynchronous queue polling class and test case class. Among them, the login tool class is used to simulate the login transaction to obtain cookies. The subsequent test cases are based on this login transaction to perform online modeling tests; the request tool class is based on the HttpURLConnection method. According to different request methods, it uses POST, GET, PUT and DELETE as the main branches to encapsulate and pass the cookies saved by the login transaction. Through this tool, it can simulate the user using the browser to send task flow requests to the modeling platform server and return the response results; the database interaction class encapsulates the addition, deletion, modification and query operations related to database table operations; the asynchronous queue polling class polls the execution status of the components in the DAG task flow for the tasks executed asynchronously in the DAG task flow by interacting with the database to obtain the final execution status of the task; the test case class is driven by the test case in the management test case, calls the corresponding method in the request tool class according to the request type in the case and obtains the return result, and compares the return result with the assertion set in the case managed by the management test case for verification. At the same time, the dependencies and serial / parallel execution modes of each test object are set in the case class to be consistent with the optimal combination in managing test cases.

[0133] 4) The test report is based on the TestNG framework. The test case class to be executed is maintained in testng.xml. After execution, an intuitive test report can be automatically generated.

[0134] like Figure 7 As shown, it is a schematic diagram of executing the test script, which may include:

[0135] 1) When the test script is executed, it first determines whether it has dependent scripts. If there are dependent scripts, it will continue to wait for the dependent scripts to be successfully executed before initiating execution.

[0136] 2) When the script is executed, the number of cases in the read management test case is determined based on whether the case is executed serially or in parallel. The serial method is to read a single case, and the parallel method is to obtain multiple cases in batches. The specific number depends on the number of concurrent cases set in the test script management test case class. Then call the request class in the test code according to the test object request method in the case, and make a judgment on the end of execution based on whether the test object is executed asynchronously or synchronously. In the synchronous execution method, the response body is obtained as the end mark; in the asynchronous execution method, the component execution status is polled by interacting with the database to obtain whether the task has been executed.

[0137] like Figure 8 As shown, it is a schematic diagram of comparing execution results, which may include: judging whether the task is executed successfully, and when the execution is successful, comparing the case execution result with the expected result maintained in Excel in the test case management. When the task execution fails, an exception is directly thrown.

[0138] like Fig. 9 As shown, it is a schematic diagram of automatic correction judgment, which may include: according to the automatic correction switch value of the case obtained in the test case management, when the automatic correction switch is turned on, the current task is successfully executed, and when the execution result is inconsistent with the expected result, the latest test result is automatically replaced with the expected value of the current case in S2. If the current task fails to execute, the case is not automatically updated; when the automatic correction switch is turned off, the case is not automatically updated.

[0139] Automatic corrective execution updates the expected value of the test case to the result of this execution and re-executes the case until the case passes.

[0140] Through the method in the above example, the automatic acquisition of test cases can be achieved. By setting the matching rules of the request body and response body for each component in the task flow to initiate tasks and view the results, invalid interfaces can be effectively filtered, request bodies and response bodies can be captured, and test cases that meet the script operation can be automatically generated without manual single-point packet capture and recording. In addition, the test script has a high reuse rate. The script encapsulates four commonly used HTTP request modes, covering DAG task flows from data processing, feature extraction, model training, batch estimation and model evaluation, model on, model reasoning, etc., and is suitable for modeling tests based on all algorithms covered by the platform. The execution order of the test script is consistent with the dependency of the DAG task flow. Different from the traditional single interface test, the method in this example truly conducts verification of multi-component combinations from the perspective of user modeling; further, it can realize parallel verification of multiple algorithms or multiple models. Different DAG task flows can set parallel execution strategies to compress test time and improve test efficiency. In the single-line case of model evaluation, the expected model evaluation index can be used as the assertion threshold, and the effect of the model after automatic training can be compared with it. If it is less than this threshold, the script execution will report an error, and the model effect can be promptly known to the public through email push.

[0141] The method embodiments provided in the above embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on an electronic device as an example, Fig.10 This is a hardware structure diagram of an electronic device provided by the present application for an automated testing method based on a drag-and-drop machine learning platform. Fig.10 As shown, the electronic device 10 may include one or more (only one is shown in the figure) processors 02 (the processor 02 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 04 for storing data, and a transmission module 06 for communication functions. It can be understood by those skilled in the art that Fig.10 The structure shown is only for illustration and does not limit the structure of the above electronic device. Fig.10 More or fewer components as shown, or with Fig.10 Different configurations are shown.

[0142] The memory 04 can be used to store software programs and modules of application software, such as program instructions / modules corresponding to the automated testing method based on the drag-and-drop machine learning platform in the embodiment of the present application. The processor 02 executes various functional applications and data processing by running the software programs and modules stored in the memory 04, that is, the automated testing method based on the drag-and-drop machine learning platform of the above-mentioned application is realized. The memory 04 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 04 may further include a memory remotely arranged relative to the processor 02, and these remote memories can be connected to the electronic device 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0143] The transmission module 06 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the electronic device 10. In one example, the transmission module 06 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 06 can be a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet wirelessly.

[0144] At the software level, the automated testing device based on the drag-and-drop machine learning platform can be Fig.11 As shown, including:

[0145] A generating module 1101 is used to generate a test case of a test object according to a request body and a response body of a modeled DGA task flow;

[0146] A setting module 1102, used to set the operating environment, dependency and serial and parallel mode for the test case;

[0147] An execution module 1103 is used to execute the test script regularly to obtain the execution result of the test case;

[0148] A comparison module 1104 is used to compare the execution result of the test case with the expected result;

[0149] The correction module 1105 is used to automatically correct the case expected value according to the comparison result and the automatic correction judgment logic.

[0150] In one embodiment, the test object may include but is not limited to the execution of at least one of the following component tasks: data introduction, data processing, data splitting, feature extraction, model training, batch estimation, model publishing, and model reasoning.

[0151] In one embodiment, the above-mentioned generation module 1101 can be specifically used to automatically intercept the request body and response body of the DAG task flow initiated by the user when modeling on the machine learning platform; name the request body and response body by concatenating the request URL with the timestamp; use the URL, request method and request parameters in the request body, and the response body as variable values ​​used in the test script, and fill them into each column of EXCEL in sequence by row as a test case.

[0152] In one embodiment, the above-mentioned setting module 1102 can be specifically used to divide the parameters of the test case into fixed values ​​and variable values; parameterize the variable values, wherein the parameterized variable values ​​are obtained in real time by interacting with the database; set the switch value of the automatic correction judgment in the test case; use the value of the core indicator in the response body as the expected value of the case; set the tasks with dependencies in the DAG task flow to serial execution, and set the tasks at the same level in the DAG task flow to parallel execution.

[0153] In one embodiment, the above-mentioned execution module 1103 can be specifically used to determine whether a preset trigger condition is met; when it is determined that the preset trigger condition is met, trigger the execution of the test script; determine whether there is a dependent script, and when there is a dependent script, continue to wait for the dependent script to be successfully executed before initiating execution; during the script execution process, when the test case is executed serially, read a single case, and when the test case is executed in parallel, read multiple cases in batches, where the number of parallel reads is determined according to the preset concurrent number of cases.

[0154] In one embodiment, during the script execution process, the request class in the test code can also be called according to the request method of the test object in the test case; the execution completion of the test object is judged according to the called request class; in the case of synchronous execution, the response body is obtained as the end mark; in the case of asynchronous execution, the execution status of the component is polled by interacting with the database to determine whether the task execution is completed.

[0155] In one embodiment, the correction module 1105 can be specifically used to obtain the automatic correction switch value of the test case; when the automatic correction switch value is on, the task is executed successfully and the execution result is inconsistent with the expected result, the expected value of the test case is replaced by the obtained test result; when the automatic correction switch value is off, or the task execution fails, the expected value of the test case is not updated.

[0156] The embodiments of the present application also provide a specific implementation of an electronic device that can implement all the steps in the automated testing method based on the drag-and-drop machine learning platform in the above embodiments, and the electronic device specifically includes the following contents: a processor, a memory, a communication interface, and a bus; wherein the processor, the memory, and the communication interface communicate with each other through the bus; the processor is used to call the computer program in the memory, and when the processor executes the computer program, all the steps in the automated testing method based on the drag-and-drop machine learning platform in the above embodiments are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0157] Step 1: Generate test cases for the test object based on the request body and response body of the modeled DGA task flow;

[0158] Step 2: Setting the running environment, dependencies and serial and parallel modes for the test case;

[0159] Step 3: Execute the test script regularly to obtain the execution results of the test case;

[0160] Step 4: Compare the test case execution results with the expected results;

[0161] Step 5: Automatically correct the expected value of the case based on the comparison results and the automatic correction judgment logic.

[0162] From the above description, it can be seen that the embodiment of the present application generates a test case that conforms to the script operation by capturing the request body and the response body, without the need for manual packet capture and recording, and establishes the operating environment, dependencies, and serial and parallel methods of the test case, and triggers the execution of the test script regularly to realize the execution of the test case, and automatically corrects the expected value of the case based on the execution result, thereby realizing error reporting and adjustment of the script execution and improving the accuracy of the test. Through the above method, the problems of low test efficiency and low accuracy in the existing automated testing of the task flow formed by the drag-type machine learning platform are solved, and the purpose of efficient and accurate automated testing of the task flow formed by the drag-type machine learning platform is achieved.

[0163] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all the steps in the automated testing method based on the drag-and-drop machine learning platform in the above embodiments. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, all the steps of the automated testing method based on the drag-and-drop machine learning platform in the above embodiments are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0164] Step 1: Generate test cases for the test object based on the request body and response body of the modeled DGA task flow;

[0165] Step 2: Setting the running environment, dependencies and serial and parallel modes for the test case;

[0166] Step 3: Execute the test script regularly to obtain the execution results of the test case;

[0167] Step 4: Compare the test case execution results with the expected results;

[0168] Step 5: Automatically correct the expected value of the case based on the comparison results and the automatic correction judgment logic.

[0169] From the above description, it can be seen that the embodiment of the present application generates a test case that conforms to the script operation by capturing the request body and the response body, without the need for manual packet capture and recording, and establishes the operating environment, dependencies, and serial and parallel methods of the test case, and triggers the execution of the test script regularly to realize the execution of the test case, and automatically corrects the expected value of the case based on the execution result, thereby realizing error reporting and adjustment of the script execution and improving the accuracy of the test. Through the above method, the problems of low test efficiency and low accuracy in the existing automated testing of the task flow formed by the drag-type machine learning platform are solved, and the purpose of efficient and accurate automated testing of the task flow formed by the drag-type machine learning platform is achieved.

[0170] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the hardware + program embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0171] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0172] Although the present application provides method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-creative labor. The order of steps listed in the embodiments is only one way of executing the order of many steps and does not represent the only execution order. When the actual device or client product is executed, it can be executed in the order of the method shown in the embodiments or the drawings or in parallel (for example, in a parallel processor or multi-threaded processing environment).

[0173] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0174] Although the present specification embodiment provides the method operation steps as described in the embodiment or flow chart, more or less operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiment is only one way in the order of execution of many steps, and does not represent a unique execution order. When the device or terminal product in practice is executed, it can be executed in sequence or in parallel (such as a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment) according to the method shown in the embodiment or the accompanying drawings. The term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or equipment including a series of elements not only includes those elements, but also includes other elements not clearly listed, or also includes elements inherent to such process, method, product or equipment. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or equipment including the elements.

[0175] For the convenience of description, the above devices are described in various modules according to their functions. Of course, when implementing the embodiments of this specification, the functions of each module can be implemented in the same or more software and / or hardware, or the module implementing the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0176] Those skilled in the art also know that, in addition to implementing the controller in a purely computer-readable program code, the controller can be made to implement the same function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules for implementing the method and structures within the hardware component.

[0177] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0178] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0179] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0180] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0181] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0182] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0183] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, the embodiments of this specification may take the form of complete hardware embodiments, complete software embodiments or embodiments combining software and hardware. Moreover, the embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0184] The present specification embodiments may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present specification embodiments may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0185] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. In the description of this specification, the description of the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of this specification. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, in the absence of contradiction, a person skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0186] The above is only an example of the embodiment of the present specification and is not intended to limit the embodiment of the present specification. For those skilled in the art, the embodiment of the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the embodiment of the present specification shall be included in the scope of the claims of the embodiment of the present specification.

Claims

1. An automated testing method based on a drag-and-drop machine learning platform. It is characterized in that The method comprises: Generate test cases for the test object based on the request body and response body of the modeled DGA task flow; Setting the operating environment, dependencies and serial and parallel modes for the test cases; Execute the test script regularly to obtain the execution results of the test case; Compare the test case execution results with the expected results; According to the comparison results and automatic correction judgment logic, the expected value of the case is automatically corrected; Among them, according to the request body and response body of the modeled DGA task flow, the test case of the test object is generated, including: Automatically intercept the request and response bodies of the DAG task flow initiated by the user when modeling on the machine learning platform; The request body and response body are named by concatenating the request URL with the timestamp. Use the URL, request method and request parameters in the request body, as well as the response body, as variable values ​​used in the test script, and fill them into each column of EXCEL row by row as a test case; The step of setting the operating environment, dependency, and serial and parallel modes for the test case includes: Dividing the parameters of the test case into fixed values ​​and variable values; Parameterizing the variable value, wherein the parameterized variable value is obtained in real time by interacting with the database; Set the switch value of automatic correction judgment in the test case; The values ​​of the core indicators in the response body are taken as the case expected values; Set the tasks with dependencies in the DAG task flow to be executed serially, and set the tasks at the same level in the DAG task flow to be executed in parallel.

2. The method according to claim 1, It is characterized in that The test object includes the execution of at least one of the following component tasks: data introduction, data processing, data splitting, feature extraction, model training, batch estimation, model publishing, and model reasoning.

3. The method according to claim 1, It is characterized in that Execute test scripts regularly to obtain the execution results of test cases, including: Determine whether the preset trigger condition is met; When it is determined that the preset trigger conditions are met, the test script is triggered to execute; Determine whether there is a dependent script. If there is a dependent script, continue to wait for the dependent script to be successfully executed before initiating execution; During the script execution, when the test case is executed serially, a single case is read; when the test case is executed in parallel, multiple cases are read in batches, wherein the number of cases read in parallel is determined according to the preset number of concurrent cases.

4. The method according to claim 3, It is characterized in that The script execution process also includes: Call the request class in the test code according to the request method of the test object in the test case; According to the called request class, the execution end judgment of the test object is performed; In the case of synchronous execution, the response body is used as the end mark; In the case of asynchronous execution, the component execution status is polled by interacting with the database to determine whether the task execution is completed.

5. The method according to claim 1, It is characterized in that According to the comparison results and automatic correction judgment logic, the expected value of the case is automatically corrected, including: Get the autocorrect switch value for the test case; When the automatic correction switch value is on, the task is executed successfully and the execution result is inconsistent with the expected result, the expected value of the test case is replaced with the obtained test result; When the autocorrection switch value is off, or the task execution fails, the expected value of the test case is not updated.

6. An automated testing device based on a drag-and-drop machine learning platform, It is characterized in that include: A generation module is used to generate test cases for the test object according to the request body and response body of the modeled DGA task flow; A setting module, used to set the operating environment, dependency and serial and parallel mode for the test case; The execution module is used to execute the test script regularly to obtain the execution result of the test case; The comparison module is used to compare the execution results of the test cases with the expected results; The correction module is used to automatically correct the expected value of the case based on the comparison results and the automatic correction judgment logic.

7. An electronic device, comprising a processor and a memory for storing instructions executable by the processor, wherein when the processor executes the instructions, the steps of the following method are implemented: Generate test cases for the test object based on the request body and response body of the modeled DGA task flow; Setting the operating environment, dependencies and serial and parallel modes for the test cases; Execute the test script regularly to obtain the execution results of the test case; Compare the test case execution results with the expected results; According to the comparison results and automatic correction judgment logic, the expected value of the case is automatically corrected; Among them, according to the request body and response body of the modeled DGA task flow, the test case of the test object is generated, including: Automatically intercept the request and response bodies of the DAG task flow initiated by the user when modeling on the machine learning platform; The request body and response body are named by concatenating the request URL with the timestamp. Use the URL, request method and request parameters in the request body, as well as the response body, as variable values ​​used in the test script, and fill them into each column of EXCEL row by row as a test case; The step of setting the operating environment, dependency, and serial and parallel modes for the test case includes: Dividing the parameters of the test case into fixed values ​​and variable values; Parameterizing the variable value, wherein the parameterized variable value is obtained in real time by interacting with the database; Set the switch value of automatic correction judgment in the test case; The values ​​of the core indicators in the response body are taken as the case expected values; Set the tasks with dependencies in the DAG task flow to be executed serially, and set the tasks at the same level in the DAG task flow to be executed in parallel.

8. A computer-readable storage medium having computer instructions stored thereon, wherein the instructions, when executed, implement the steps of the following method: Generate test cases for the test object based on the request body and response body of the modeled DGA task flow; Setting the operating environment, dependencies and serial and parallel modes for the test cases; Execute the test script regularly to obtain the execution results of the test case; Compare the test case execution results with the expected results; According to the comparison results and automatic correction judgment logic, the expected value of the case is automatically corrected; Among them, according to the request body and response body of the modeled DGA task flow, the test case of the test object is generated, including: Automatically intercept the request and response bodies of the DAG task flow initiated by the user when modeling on the machine learning platform; The request body and response body are named by concatenating the request URL with the timestamp. Use the URL, request method and request parameters in the request body, as well as the response body, as variable values ​​used in the test script, and fill them into each column of EXCEL row by row as a test case; The step of setting the operating environment, dependency, and serial and parallel modes for the test case includes: Dividing the parameters of the test case into fixed values ​​and variable values; Parameterizing the variable value, wherein the parameterized variable value is obtained in real time by interacting with the database; Set the switch value of automatic correction judgment in the test case; The values ​​of the core indicators in the response body are taken as the case expected values; Set the tasks with dependencies in the DAG task flow to be executed serially, and set the tasks at the same level in the DAG task flow to be executed in parallel.

Citation Information

Patent Citations

  • Interface automatic test method and device, computer equipment and readable storage medium

    CN111694749A

  • Interface automatic test method and device

    CN111949520A