Test data generation method, device and electronic device for machine learning framework

By acquiring basic data to generate prompt templates and utilizing semantic models, the problem of insufficient randomness and diversity in test data generation in machine learning frameworks is solved, achieving efficient and automatic test data generation and improving testing efficiency and deep code detection capabilities.

CN118194292BActive Publication Date: 2024-11-26BEIJING QIHOOD TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410302818.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-15
Publication Date
2024-11-26
Estimated Expiration
2044-03-15

AI Technical Summary

Technical Problem

Existing machine learning frameworks suffer from insufficient randomness and diversity in their test data generation methods, leading to low testing efficiency and an inability to effectively detect security vulnerabilities deep within the code.

Method used

By acquiring the basic data of the machine learning framework under test, a prompt template that conforms to the data structure attributes is generated, and test data is automatically generated using a semantic model to ensure the legality and diversity of the test data.

Benefits of technology

It improves the diversity and effectiveness of test data, and can automatically and efficiently generate test data that conforms to the data structure attributes of operators, thereby improving testing efficiency and the ability to detect deep code.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118194292B_ABST
    Figure CN118194292B_ABST
Patent Text Reader

Abstract

The embodiment of the present application discloses a test data generation method, device and electronic device for a machine learning framework, and relates to the field of testing technology. The method includes: obtaining basic data related to the test of the machine learning framework to be tested; generating a prompt template that conforms to at least one data structure attribute according to the data structure attributes of the operator in the machine learning framework to be tested and the basic data; inputting the prompt template into a semantic model to generate first test data. The method provided in the present application can provide a variety of data reserves for test data by obtaining basic data. Not only can the prompt template generate test data of a single attribute, but also can generate test data of a composite type of multiple attributes, which can not only ensure the legitimacy of the test data, but also improve the diversity of the test data. Compared with manually generating test data, the prompt template is input into the semantic model to generate the first test data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of testing technology, and in particular to a method, device and electronic device for generating test data for a machine learning framework. Background Art

[0002] At present, in order to test the security of the machine learning framework, it is necessary to use test data to find out the security vulnerabilities of the machine learning framework when developing the machine learning framework.

[0003] There are two problems with the current test data generation method for machine learning frameworks: first, the generated test data is relatively random and does not meet the structural requirements of the machine learning framework for test data, causing the test to stagnate at the legitimacy verification stage and unable to enter the deep code for security testing; second, the diversity of test data is low and relies on the subjective experience of testers. When encountering new test projects, the diversity and effectiveness of the generated test data are difficult to meet the testing requirements, resulting in reduced testing efficiency. Summary of the invention

[0004] The embodiment of the present application provides a test data generation method, device, equipment and storage medium of a machine learning framework, which is used to solve the defect that the diversity and effectiveness of the test data generated in the above-mentioned related technologies are difficult to meet the test requirements. The technical solution provided by the embodiment of the present application is as follows:

[0005] In a first aspect, an embodiment of the present application provides a method for generating test data for a machine learning framework, comprising:

[0006] Obtain basic data related to the testing of the machine learning framework to be tested;

[0007] Generate a prompt template that conforms to at least one data structure attribute according to the data structure attribute of the operator in the machine learning framework to be tested and the basic data;

[0008] The prompt template is input into the semantic model to generate first test data.

[0009] In an optional scheme of the first aspect, the basic data is at least one of the following: historical vulnerability data of the machine learning framework to be tested, each data type in the machine learning framework to be tested and the boundary values ​​corresponding to each data type, and data of a preset type corresponding to the machine learning framework to be tested.

[0010] In an optional solution of the first aspect, after obtaining basic data related to the test of the machine learning framework to be tested, the method further includes:

[0011] The basic data are processed into basic data vectors, and a basic database is constructed based on the basic data vectors.

[0012] In an optional solution of the first aspect, the data structure attributes of the operator include type, shape and value;

[0013] Wherein, the types include basic types and composite types;

[0014] The shape includes the dimension and number of elements of the operator;

[0015] The value is the value of each element in the operator.

[0016] In an optional solution of the first aspect, generating a prompt template that conforms to at least one data structure attribute according to the data structure attribute of the operator in the machine learning framework to be tested and the basic data includes:

[0017] generating a control field based on at least one data structure attribute of the operator;

[0018] Traversing the basic database to obtain basic data related to the control field, and using the traversal result as the data field;

[0019] The prompt template is generated by combining the control field and the data field.

[0020] In an optional solution of the first aspect, inputting the prompt template into a semantic model to generate first test data includes:

[0021] The prompt template is input into a semantic model, a data structure attribute is determined based on a control field of the prompt template, and basic data that conforms to the at least one data structure attribute is determined in a data field of the prompt template as the first test data.

[0022] In an optional solution of the first aspect, inputting the prompt template into a semantic model to generate first test data includes:

[0023] The prompt template is input into the semantic model, the requirements of the data structure attributes are determined based on the control field of the prompt template, basic data that conforms to at least one data structure attribute is determined in the data field of the prompt template, the basic data that conforms to at least one data structure attribute is modified, and the first test data is generated based on the modified data.

[0024] In an optional solution of the first aspect, after generating the first test data, the method further includes:

[0025] The first test data is input into the operator test template for testing, and a first test result is output.

[0026] In an optional solution of the first aspect, after generating the first test data, inputting the first test data into an operator test template for testing, and before outputting the first test result, further includes:

[0027] comparing the first test data with the test data in the test database;

[0028] If there is duplicate test data between the first test data and the test database, deleting the first test data;

[0029] The step of inputting the first test data into an operator test template for testing and outputting a first test result includes:

[0030] If there is no duplicate test data between the first test data and the test database, the first test data is input into the operator test template for testing, and a first test result is output.

[0031] In an optional solution of the first aspect, after outputting the first test result, the method further includes:

[0032] If the first test result does not include error information, the first test data is saved in a test database.

[0033] In an optional solution of the first aspect, the step of inputting the first test data into an operator test template for testing and outputting a first test result further includes:

[0034] If the first test result includes error information, the error parameters corresponding to the error information are corrected and a correction result is output.

[0035] In an optional solution of the first aspect, the correcting the error parameter corresponding to the error information and outputting the correction result includes:

[0036] Traversing the vector database of the machine learning framework to be tested to obtain the code snippet corresponding to the error information;

[0037] A correction prompt template is constructed in combination with the error information, the error parameters and the code snippet, the correction prompt template is input into the semantic model, the error parameters are corrected through the semantic model, and a correction result is output.

[0038] In an optional solution of the first aspect, after outputting the correction result, the method further includes:

[0039] The correction result and the first test data are combined to generate second test data, the second test data is input into the operator test template for testing, and a second test result is output.

[0040] In an optional solution of the first aspect, after combining the correction result with the first test data to generate second test data, inputting the second test data into an operator test template for testing and before outputting the second test result, further comprising:

[0041] comparing the second test data with the test data in the test database;

[0042] If there is duplicate test data between the second test data and the test database, deleting the second test data;

[0043] The step of inputting the second test data into the operator test template for testing and outputting the second test result includes:

[0044] If there is no duplicate test data between the second test data and the test database, the second test data is input into the operator test template for testing, and a second test result is output.

[0045] In an optional solution of the first aspect, the step of inputting the second test data into an operator test template for testing and outputting the second test result includes:

[0046] If the second test result does not include error information, saving the second test data to a test database;

[0047] If the second test result includes error information, the step of correcting the error parameter corresponding to the error information and outputting the correction result is performed again until the second test result does not include error information.

[0048] In a second aspect, an embodiment of the present application further provides a test data generating device for a machine learning framework, comprising:

[0049] The basic data module is used to obtain basic data related to the testing of the machine learning framework to be tested;

[0050] A test data generation module, used to generate a prompt template that conforms to at least one data structure attribute according to the data structure attributes of the operator in the machine learning framework to be tested;

[0051] The test data generation module is further used to input the prompt template into a semantic model to generate first test data.

[0052] In an optional solution of the second aspect, the device further includes a testing module, the test data generating module inputs the first test data into an operator test template of the testing module for testing, and the testing module generates a first test result.

[0053] In an optional scheme of the second aspect, the device also includes a data correction module. After the test data generation module inputs the first test data into the operator test template of the test module for testing, if the first test result generated by the test module includes error information, the data correction module corrects the error parameters corresponding to the error information and outputs the correction result.

[0054] In an optional solution of the second aspect, the data correction module corrects the error parameter corresponding to the error information and outputs a correction result, including:

[0055] The data correction module traverses the vector database of the machine learning framework to be tested to obtain the code snippet corresponding to the error information;

[0056] The data correction module constructs a correction prompt template in combination with the error information, the error parameters and the code snippet, and inputs the correction prompt template into the semantic model. The data correction module corrects the error parameters through the semantic model and outputs the correction result.

[0057] In an optional solution of the second aspect, the data correction module corrects the error parameter corresponding to the error information and outputs the correction result, further comprising:

[0058] The data correction module combines the correction result and the first test data to generate second test data, inputs the second test data into the operator test template of the test module for testing, and the test module outputs a second test result.

[0059] In an optional solution of the second aspect, the data correction module inputs the second test data into the operator test template of the test module for testing, and after the test module outputs the second test result, the method further includes:

[0060] If the second test result does not include error information, the data correction module saves the second test data to a test database;

[0061] If the second test result includes error information, the data correction module again performs the step of correcting the error parameters corresponding to the error information and outputting the correction result until the second test result does not include error information.

[0062] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method provided in the first aspect of the embodiment of the present application or any one of the implementation methods of the first aspect is implemented.

[0063] In a fourth aspect, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method provided by the first aspect of the embodiment of the present application or any one of the implementations of the first aspect.

[0064] The beneficial effects brought about by the technical solutions provided by some embodiments of the present application include at least:

[0065] The test data generation method of a machine learning framework provided in an embodiment of the present application can provide a variety of data reserves for the generation of test data by acquiring basic data related to the test of the machine learning framework to be tested. Further, a prompt template is generated based on the basic data and at least one data structure attribute of the operator, which can be used not only to generate test data of a single attribute, but also to generate test data of a multi-attribute composite type, which can not only ensure the legitimacy of the operator data structure attributes, but also ensure the diversity of the test data. The prompt template is input into the semantic model to generate the first test data. Compared with manually generated test data, the test data can be automatically and efficiently generated based on the prompt template based on the natural language understanding ability of the semantic model. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0067] Figure 1 A schematic diagram of the architecture of a test data generation system for a machine learning framework according to an embodiment of the present application;

[0068] Figure 2 It is a flowchart of a method for generating test data for a machine learning framework according to an embodiment of the present application;

[0069] Figure 3 It is a flowchart of a method for generating test data for a machine learning framework provided in an embodiment of the present application;

[0070] Figure 4 It is a flowchart of a method for generating test data for a machine learning framework provided in an embodiment of the present application;

[0071] Figure 5 It is a flowchart of a method for generating test data for a machine learning framework provided in an embodiment of the present application;

[0072] Figure 6It is a structural schematic diagram of a test data generating device of a machine learning framework provided in an embodiment of the present application;

[0073] Figure 7 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0075] The terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or modules is not limited to the listed steps or modules, but may optionally include steps or modules that are not listed, or may optionally include other steps or modules that are inherent to these processes, methods, products or devices.

[0076] It should be noted that the terms "first\second" involved in the present application are only used to distinguish similar objects, and do not represent a specific order for the objects. It is understandable that "first\second" can be interchanged with a specific order or sequence where permitted. It should be understood that the objects distinguished by "first\second" can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those described or illustrated herein.

[0077] Related technologies mainly focus on the functions, performance and ease of use of machine learning frameworks for developers. For example, Tensorflow focuses more on versatility, Pytorch focuses on ease of use, and Oneflow focuses on high performance, but there is a lack of sufficient consideration for the security of machine learning frameworks.

[0078] Therefore, when developing a machine learning framework, it is necessary to use test data to identify and resolve security vulnerabilities in the machine learning framework, so as to ensure the security of the framework, application, and user data.

[0079] However, there are two problems with the current test data generation method for machine learning frameworks: first, the generated test data is relatively random and does not meet the structural requirements of the machine learning framework for test data, causing the test to stagnate at the legitimacy verification stage and unable to enter the deep code for security testing; second, the diversity of test data is low and relies on the subjective experience of testers. When encountering new test projects, the diversity and effectiveness of the generated test data are difficult to meet the testing requirements, resulting in reduced testing efficiency.

[0080] Based on this, an embodiment of the present application provides a test data generation method for a machine learning framework, which can provide a variety of data reserves for the generation of test data by obtaining basic data related to the test of the machine learning framework to be tested. Further, a prompt template is generated based on the basic data and at least one data structure attribute of the operator, which can be used not only to generate test data of a single attribute, but also to generate test data of a multi-attribute composite type, which can not only ensure the legitimacy of the operator data structure attributes, but also ensure the diversity of the test data. The prompt template is input into the semantic model to generate the first test data. Compared with manually generated test data, the test data can be automatically and efficiently generated based on the prompt template based on the natural language understanding ability of the semantic model.

[0081] Please refer to the following Figure 1 , which is a schematic diagram of the architecture of a test data generation system for a machine learning framework provided by an exemplary embodiment of the present application. Figure 1 As shown, the test data generation system of the machine learning framework includes: a terminal 110 and a server 120.

[0082] The terminal 110 may include one or more terminals corresponding to users. A client version of the software may be installed in the terminal 110, and the software is used to obtain basic data related to the test of the machine learning framework to be tested, generate a prompt template that conforms to at least one data structure attribute according to the data structure attributes of the operator in the machine learning framework to be tested and the basic data, input the prompt template into the semantic model, and generate the first test data. The terminal 110 may also establish a data relationship with the network, and establish a data connection relationship with the server 120 through the network. For example, the terminal 110 may obtain basic data related to the test of the machine learning framework to be tested from the server 120, and may also input the prompt template into the semantic model of the server 120, receive the output result of the semantic model of the server 120, and then generate the first test data.

[0083] The server 120 may receive the prompt template from the terminal 110 through the network, input the prompt template into the semantic model set up on the server 120, and output the output result of the semantic model to the terminal 110. It can be understood that the terminal 110 may be, but is not limited to, a hardware server, a virtual server, a cloud server, etc.

[0084] The test data generation method for the above-mentioned machine learning framework is not limited to execution by the above-mentioned server-side 120, but can also be executed by the terminal 110 alone, or by the terminal 110 and the server-side 120 in combination. The embodiments of the present application do not make specific limitations on this. The following embodiments are all explained by taking the test data generation method for the above-mentioned machine learning framework executed by the terminal 110 as an example.

[0085] It is understandable that the semantic model can be set on a local terminal or a server. The terminal 110 can be connected to peripheral devices to input required basic data through peripheral devices such as a keyboard and a mouse, and can also be connected to an external storage device to directly obtain basic data from the external storage device.

[0086] The network may be a medium that provides a communication link between the server 120 and any terminal 110, or may be the Internet including network devices and transmission media, but is not limited thereto. The transmission medium may be a wired link, such as but not limited to coaxial cable, optical fiber, and digital subscriber line (DSL), or a wireless link, such as but not limited to wireless fidelity (WIFI), Bluetooth, and mobile device network.

[0087] Understandably, Figure 1 The number of terminals 110 and server ends 120 in the test data generation system of the machine learning framework shown is only an example. In a specific implementation, the test data generation system may include any number of terminals and server ends, and the embodiment of the present application does not specifically limit this. For example, but not limited to, the server end 120 may be a server cluster composed of multiple servers, and the terminal 110 may be a terminal cluster composed of multiple terminals.

[0088] Understandably, a machine learning framework is a library, interface, or tool that enables developers to build machine learning models more easily and quickly. By using a library of pre-built and optimized components to define machine learning models, a machine learning framework can avoid programmers starting from scratch when creating specific machine learning applications, making the development process more efficient.

[0089] The present application is described in detail below with reference to specific embodiments.

[0090] Next, combine Figure 1, taking the method for generating test data of a machine learning framework executed by a terminal as an example, the method for generating test data of a machine learning framework provided by the embodiment of the present application is introduced. Figure 2 , Figure 2 A flow chart of a method for generating test data for a machine learning framework provided in an embodiment of the present application is shown, and the method comprises the following steps:

[0091] S201, obtaining basic data related to the test of the machine learning framework to be tested.

[0092] It is understandable that the machine learning framework to be tested includes but is not limited to TensorFlow, PyTorch, MXNet, CNTK, Caffe, etc., and the present application embodiment does not limit this. The complexity of the machine learning framework means that there are many security risks hidden in it, so it is necessary to detect security vulnerabilities in the machine learning framework through testing.

[0093] Specifically, the basic data related to the testing of the machine learning framework to be tested can be understood as reserve data for testing the machine learning framework. The data generated in the historical operation and / or testing of the machine learning framework can be parsed and collected as reserve data. It can also include any data types and values ​​manually selected as reserve data for testing, thereby providing a basis for the generation of test data.

[0094] Optionally, the basic data may be at least one of the following: historical vulnerability data of the machine learning framework to be tested, each data type in the machine learning framework to be tested and the boundary value corresponding to each data type, and data of a preset type corresponding to the machine learning framework to be tested;

[0095] Specifically, historical vulnerability data can be understood as data that has triggered vulnerabilities in the historical use and / or test records of the machine learning framework. It can be obtained directly from the security bulletin issued by the publisher of the machine learning framework, or from the historical use and / or test records of the machine learning framework. Historical vulnerability data can reflect common problems in the machine learning framework. Exemplarily, historical vulnerability data may include heap out-of-bounds read and write and integer overflow vulnerabilities caused by negative or large integer parameters, null pointer dereference vulnerabilities caused by empty arrays, floating point exception vulnerabilities caused by 0, etc. By parsing historical vulnerabilities, key data that triggers the vulnerability can be obtained, and the test data generated by this will likely test similar vulnerabilities.

[0096] Specifically, the machine learning framework to be tested has data of different data types. Different data types have corresponding value ranges. When the value of the data is at the boundary of the corresponding value range, it is easy to cause an out-of-bounds vulnerability. The relevant data types and corresponding boundary values ​​can be analyzed based on the machine learning framework documentation. For example, for a 32-bit signed integer, the value range is [-2 31 ,2 31 -1], when the value is -2 31 When there is a conversion operation with a 64-bit signed integer or a 32-bit unsigned integer, the sign of the value will change and it will become a positive number, thus causing an out-of-bounds vulnerability.

[0097] Specifically, the preset type of data can be understood as: a specific type of data selected based on the type of machine learning framework. Different machine learning frameworks have different data types. Targeted selection is performed based on the special types of data existing in different machine learning frameworks. Data types not covered by historical vulnerability data and boundary value data can also be selected as a further supplement to the basic data. Among them, the above-mentioned historical vulnerability data, data types, and corresponding boundary values ​​only involve simple types of data, such as numbers or strings, while the machine learning framework also includes some special types of data. For example, in TensorFlow, in addition to simple types such as integer data type int and floating-point data type float, there are also special types of data such as resource type DT_RESOURCE and variant type DT_VARIANT.

[0098] It can be understood that the basic data can be any arrangement or combination of the above three types of data, and may include historical vulnerability data of the machine learning framework to be tested, each data type in the machine learning framework to be tested and the boundary values ​​corresponding to each data type, and any two of the preset types of data corresponding to the machine learning framework to be tested. It may include all three types.

[0099] Optionally, the acquired basic data related to the test of the machine learning framework to be tested can be stored locally and / or in the cloud.

[0100] S202, generating a prompt template that conforms to at least one data structure attribute according to the data structure attributes of the operator in the machine learning framework to be tested and the basic data.

[0101] Understandably, by parsing the operator document and description file of the machine learning framework to be tested, the data structure attributes of the operator parameters can be obtained, and the operator parameters may include multiple data structure attributes.

[0102] Specifically, the prompt template is generated according to the data structure attribute of the operator and the basic data, and the prompt template conforms to at least one data structure attribute, which can be understood as that the prompt template defines at least one data structure attribute and also includes the basic data obtained in S201. The data structure attribute in the prompt template can be one of the data structure attributes of the operator, or a combination of more than one data structure attributes of the operator.

[0103] In some embodiments, tensor is a basic data structure for operations in a machine learning framework. A tensor is defined by three data structure attributes: type, shape, and value; wherein the type includes basic types and composite types; the shape includes the dimension and number of elements of the operator; and the value is the value of each element in the operator.

[0104] S203: Input the prompt template into the semantic model to generate first test data.

[0105] It can be understood that the semantic model is a large language model with natural language understanding capabilities. The specific type of the semantic model is not limited in the embodiments of the present application. The large language model can realize the ability to process a large amount of information by analyzing and learning a large amount of data, thereby generating new data based on the existing database.

[0106] Specifically, after the prompt template is input into the semantic model, the semantic model can be used to understand the meaning of the language text in the prompt template, thereby generating first test data that conforms to the data structure attributes in the prompt template. The generated first test data also conforms to the data structure attributes of the operator.

[0107] Specifically, according to different data structure attributes defined in the prompt template, the first test data may not only be test data conforming to one data structure attribute, but may also be test data conforming to multiple attributes.

[0108] The test data generation method of a machine learning framework provided in an embodiment of the present application can provide a variety of data reserves for the generation of test data by acquiring basic data related to the test of the machine learning framework to be tested. Further, a prompt template is generated based on the basic data and at least one data structure attribute of the operator, which can be used not only to generate test data of a single attribute, but also to generate test data of a multi-attribute composite type, which can not only ensure the legitimacy of the operator data structure attributes, but also ensure the diversity of the test data. The prompt template is input into the semantic model to generate the first test data. Compared with manually generated test data, the test data can be automatically and efficiently generated based on the prompt template based on the natural language understanding ability of the semantic model.

[0109] See next Figure 3 , is a flow chart of another method for generating test data for a machine learning framework provided in an embodiment of the present application. Figure 3 As shown, the method may include:

[0110] S301, obtaining basic data related to the test of the machine learning framework to be tested;

[0111] Specifically, for the parts not fully described in this step, please refer to S201, which will not be repeated here.

[0112] S302, processing the basic data into basic data vectors, and constructing a basic database based on the basic data vectors.

[0113] Specifically, after the basic data is acquired, the basic data is processed into a vector form, and the vectorized basic data vector is stored in a basic database.

[0114] Further, a prompt template conforming to at least one data structure attribute is generated according to the data structure attribute of the operator in the machine learning framework to be tested and the basic data, specifically comprising the steps of:

[0115] S303, generating a control field based on the data structure attribute of at least one operator; traversing the basic database to obtain basic data related to the control field, using the traversal result as the data field, and combining the control field and the data field to generate a prompt template.

[0116] Specifically, the prompt template includes a control field and a data field.

[0117] Specifically, the control field defines at least one data structure attribute of the operator, and the data structure attribute in the control field is described in natural language. When generating the prompt template, at least one data structure attribute of the operator parameter described in natural language is constructed as the control field. Exemplarily, for the basic data structure tensor, the control field can be "generate a number with an integer type value less than 10", the required data type is the integer type in the basic type data, the shape (number of elements) is 1, and the element value is 10; the control field can also be "generate a number less than 10", the shape (number of elements) is 1, the element value is 10, and there is no requirement for the data type.

[0118] Specifically, according to the data structure attribute defined in the control field, all stored basic data are traversed in the basic database established in S302, and data that meets the data structure attribute required by the control field is selected as the data field. For example, if the control field is "generate a number of integer type with a value less than 10", then in the basic database, all integer type numbers with a value less than 10 are traversed as data fields.

[0119] Further, the prompt template is input into the semantic model to generate the first test data, which specifically includes the steps of:

[0120] S304: input the prompt template into the semantic model, determine the data structure attribute based on the control field of the prompt template, and determine the basic data that meets at least one data structure attribute in the data field of the prompt template as the first test data.

[0121] Specifically, the natural language understanding capability of the semantic model is used to parse the control field and the data field in the prompt template. The control field provides the requirements for the data structure attributes, such as determining the structure of the test data from three aspects: value, type, and shape, and the data field provides the optional range of the basic data, which is used to determine the specific content of the test data, thereby generating the first test data.

[0122] Exemplarily, if the control field determines that a real number of a positive integer type is required, and the data field includes 1 and 2, the first test data may be generated as: real number 1 of a positive integer type, real number 2 of a positive integer type.

[0123] Optionally, inputting the prompt template into the semantic model to generate the first test data specifically includes the steps of:

[0124] S305, input the prompt template into the semantic model, determine the requirements of the data structure attributes based on the control field of the prompt template, determine the basic data that conforms to at least one data structure attribute in the data field of the prompt template, modify the basic data that conforms to at least one data structure attribute, and generate the first test data based on the modified data.

[0125] Specifically, based on the basic data given in the data field, the given basic data can be modified, and the data can be derived under the premise of meeting the data legality requirements. For example, if the data structure attribute is determined to be a positive integer real number in the control field, and the data field gives data 1 and 2, the data can be derived to 3 or other positive integer real numbers.

[0126] Optionally, modifying the basic data may include changing the data value, data type, data symbol, etc. Part of the basic data may be modified, or all of the basic data may be modified.

[0127] In the embodiment of the present application, by vectorizing the basic database, it is convenient to retrieve the basic data, so that the data can be quickly retrieved and consulted. By generating a prompt template based on the basic data and at least one data structure attribute of the operator, it can be used not only to generate test data of a single attribute, but also to generate test data of a multi-attribute composite type, which can not only ensure the legality of the operator data structure attribute, but also ensure the diversity of the test data. By obtaining or modifying the variant derivative---more data can be generated. The prompt template is input into the semantic model to generate the first test data. Compared with the manually generated test data, the test data can be automatically and efficiently generated based on the prompt template based on the natural language understanding ability of the semantic model.

[0128] See next Figure 4 , which is a flow chart of test data generation of a machine learning framework provided in an embodiment of the present application. Figure 4 As shown, the method may include the following steps:

[0129] S401, inputting first test data into an operator test template for testing, and outputting a first test result.

[0130] It can be understood that the operator test template library refers to the template database of the test operator, which stores different templates of the test operator, as well as contents such as the operator calling method, operator parameter type, and constraints between operator parameters, and can generate test data that conforms to the operator data structure attributes according to the operator data structure attributes. The operator test template library can be used to perform multi-dimensional testing on the operator, thereby effectively improving the security of the operator calculation.

[0131] Specifically, the first test data is input into the operator test template through the function interface corresponding to the machine learning framework to be tested.

[0132] Specifically, the first test result directly reflects the result of the test success or test failure after the test data is input into the operator test template. If the test fails, the first test result will feedback the error message of the test failure. When an error message appears, it indicates that the corresponding test data cannot be used for the test of the machine learning framework to be tested.

[0133] Exemplarily, the error information may include data parsing failure, irregular data processing, etc., which is not limited in the embodiments of the present application.

[0134] Further, depending on whether the first test result includes error information, after S401, if the test result does not include error information, step S402 is executed; if the test result includes error information, step S403 is executed.

[0135] Specifically, if the first test result does not include error information, it specifically includes:

[0136] S402: Save the first test data to a test database.

[0137] Specifically, if there is no error information in the first test result, it indicates that the corresponding first test data can be used to test the corresponding machine learning framework to be tested, and the corresponding first test data can be stored in a test database. By storing all test data that can be used for testing in the test database, the test database can be stored in a local terminal device and / or a cloud server.

[0138] Optionally, a label of a corresponding type may be generated based on the type of the operator test template, and a mapping relationship between the label and the test database may be generated, thereby facilitating calling corresponding test data according to the type of the operator test template.

[0139] Specifically, if the first test result includes error information, it includes:

[0140] S403: If the first test result includes error information, correct the error parameters corresponding to the error information and output the corrected result.

[0141] Specifically, the code snippet corresponding to the error information can be obtained by traversing the vector database of the machine learning framework to be tested;

[0142] Specifically, the error parameter causing the error can be determined through the error information or the code snippet corresponding to the error information, wherein the code snippet includes specific parameter judgment of the relevant parameters, which can be used to determine the corresponding error parameter.

[0143] Specifically, a correction prompt template can be constructed by combining error information, error parameters and code snippets, and the correction prompt template can be input into the semantic model. The natural language understanding ability of the semantic model can be used to understand the error parameters recorded in the correction prompt template, correct the error parameters through the semantic model, and output the correction result.

[0144] For example, when the Tensorflow framework is running, the error message "TransposeOp operand rank5does not match permutation size 4" may appear. This error message indicates that the operand of the operator Transpose is 5-dimensional, which does not match the size 4 of the variable permutation. The Transpose operator parameters include a parameter named permutation. Therefore, the correction prompt template can include information that the dimension of the Transpose operator does not match the dimension of the error parameter permutation. The semantic model can understand the correction prompt template and modify the dimension of the error parameter permutation.

[0145] S404: Combine the correction result and the first test data to generate second test data.

[0146] Specifically, after the first test data is tested, one or more error parameters may exist, and the correction result may be understood as the correction result of all the error parameters.

[0147] Specifically, after the correction result is generated, it is necessary to merge the correction result with the first test data to replace the original erroneous parameters.

[0148] S405: Input the second test data into the operator test template for testing, and output a second test result.

[0149] Furthermore, after obtaining the second test result, it is necessary to determine whether further correction is required to be made to the second test data according to the second test result.

[0150] Specifically, if the second test result does not include error information, step S402 is executed to save the second test data into the test database.

[0151] Specifically, if the second test result includes error information, steps S403, S404 and S405 are performed again until the second test result does not include error information.

[0152] Specifically, the second test data may be iterated multiple times, and if the corrected data still has problems, the test data correction process is repeated.

[0153] Optionally, in the process of performing multiple loop iterations on the second test data, an upper limit of the number of loop iterations may be set. If error information still exists after iterations exceeding the upper limit, error parameters corresponding to the error information are discarded.

[0154] In the embodiment of the present application, after the first test data is tested, if the first test result includes error information, the first test data is corrected, and the second test data is further generated according to the correction result, thereby reducing the cost of generating test data and generating more test data. A correction prompt template is generated by combining code snippets, error parameters and error information, and the correction prompt template is input into the semantic model, thereby intelligently identifying and correcting errors in the test data, reducing the cost of test data correction, improving the accuracy of test data, and improving test efficiency.

[0155] See next Figure 5 , which is a flow chart of test data generation of a machine learning framework provided in an embodiment of the present application. Figure 5 As shown, the method may include the following steps:

[0156] S501: Input a prompt template into a semantic model to generate first test data.

[0157] Specifically, for the parts not fully described in this step, please refer to S203, which will not be repeated here.

[0158] S502: Compare the first test data with the test data in the test database.

[0159] Specifically, the test database is a database for storing test data. Some test data may be selected according to a preset and stored as the test database. Alternatively, test data that does not generate error information may be stored in the test database according to the test result.

[0160] Optionally, the test database may also be a collection of multiple test databases.

[0161] Specifically, the first test data is compared with all existing test data in the test database to avoid duplication of test data. Part of the test data may have the same parameters, and duplicate test data can be understood as all parameters in the two test data being exactly the same.

[0162] Further, based on the comparison result, if there is duplicate test data between the first test data and the test database, the following steps are performed:

[0163] S503, delete the first test data.

[0164] Further, if there is no duplicate test data between the first test data and the test database, the following steps are performed:

[0165] S504: Input the first test data into the operator test template for testing, and output a first test result.

[0166] Specifically, for the parts not fully described in this step, please refer to S401, which will not be repeated here.

[0167] Further, if the first test result does not include error information, it includes:

[0168] S505: Save the first test data to a test database.

[0169] Optionally, the test database and the test database used for comparing the test data are the same test database.

[0170] Specifically, for the parts not fully described in this step, please refer to S402, which will not be repeated here.

[0171] Further, if the first test result includes error information, it includes:

[0172] S506: If the first test result includes error information, correct the error parameters corresponding to the error information and output the corrected result.

[0173] Specifically, for the parts not fully described in this step, please refer to S403, which will not be repeated here.

[0174] S507, combining the correction result and the first test data to generate second test data.

[0175] Specifically, for the parts not fully described in this step, please refer to S404, which will not be repeated here.

[0176] Further, after the second test data is generated, the second test data is input into the operator test template for testing, and before the test result is output, step S502 is executed to compare the second test data with the test data in the test database again.

[0177] Further, based on the comparison result, if there is duplicate test data in the test database, execute step S503 to delete the second test data; if there is no duplicate test data in the test database, execute step S504 to input the second test data into the operator test template for testing and output the test result.

[0178] The method provided in the embodiment of the present application, after each time the first test data and the second test data are generated, checks the first test data and the second test data for duplication based on the test data in the test database, and deletes the test data that is identical to the test data already in the test database, thereby avoiding duplication of test data and preventing repeated testing using the same test data, which is beneficial to improving test efficiency.

[0179] The following are device embodiments of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0180] See next Figure 6 , which is a schematic diagram of the structure of a device for calculating the application usage time provided by an exemplary embodiment of the present application. The device can be implemented as all or part of a terminal through software, hardware, or a combination of both, and can also be integrated on a server as an independent module. The device for calculating the application usage time in the embodiment of the present application can be applied to a terminal or a cloud. The device 6 includes a basic data module 61, a test data generation module 62, a test module 63, and a data correction module 64, specifically including:

[0181] A basic data module 61, used to obtain basic data related to the test of the machine learning framework to be tested;

[0182] A test data generation module 62, for generating a prompt template that conforms to at least one data structure attribute according to the data structure attribute of the operator in the machine learning framework to be tested;

[0183] The test data generating module 62 is further used to input the prompt template into the semantic model to generate the first test data.

[0184] Optionally, the basic data acquired by the basic data module 61 is at least one of the following: historical vulnerability data of the machine learning framework to be tested, each data type in the machine learning framework to be tested and the boundary values ​​corresponding to each data type, and data of a preset type corresponding to the machine learning framework to be tested.

[0185] Optionally, after acquiring basic data related to the test of the machine learning framework to be tested, the basic data module 61 processes the basic data into basic data vectors and constructs a basic database based on the basic data vectors.

[0186] Specifically, the data structure attributes of an operator include type, shape, and value; among them, type includes basic type and composite type; shape includes the dimension and number of elements of the operator; and value is the value of each element in the operator.

[0187] Optionally, the test data generation module 62 inputs the prompt template into the semantic model, determines the data structure attributes based on the control field of the prompt template through the semantic model, and determines the basic data that meets at least one data structure attribute in the data field of the prompt template as the first test data.

[0188] Optionally, the test data generation module 62 inputs the prompt template into the semantic model, the test data generation module 62 determines the requirements of the data structure attributes based on the control field of the prompt template through the semantic model, determines the basic data that conforms to at least one data structure attribute in the data field of the prompt template, modifies the basic data that conforms to at least one data structure attribute, and generates the first test data based on the modified data.

[0189] Optionally, after generating the first test data, the first test data is input into the operator test template for testing. Before outputting the first test result, the test data generation module 62 is also used to compare the first test data with the test data in the test database. If there is duplicate test data between the first test data and the test database, the first test data is deleted; if there is no duplicate test data between the first test data and the test database, the test data generation module 62 inputs the first test data into the operator test template for testing, and outputs the first test result.

[0190] In some embodiments, the apparatus further includes a testing module 63, the test data generating module 62 inputs the first test data into the operator test template of the testing module 63 for testing, and the testing module 63 generates a first test result.

[0191] In some embodiments, the device also includes a data correction module 64. After the test data generation module 62 inputs the first test data into the operator test template of the test module 63 for testing, if the first test result generated by the test module 63 includes error information, the data correction module 64 corrects the error parameters corresponding to the error information and outputs the correction result.

[0192] Specifically, the data correction module 64 corrects the error parameters corresponding to the error information and outputs the correction results, including:

[0193] The data correction module 64 traverses the vector database of the machine learning framework to be tested to obtain the code snippet corresponding to the error information;

[0194] The data correction module 64 constructs a correction prompt template based on the error information, error parameters and code snippets, and inputs the correction prompt template into the semantic model. The data correction module 64 corrects the error parameters through the semantic model and outputs the correction result.

[0195] Specifically, the data correction module 64 corrects the error parameters corresponding to the error information and outputs the correction result, and further includes:

[0196] The data correction module 64 combines the correction result with the first test data to generate second test data, inputs the second test data into the operator test template of the test module 63 for testing, and the test module 63 outputs a second test result.

[0197] Optionally, after the data correction module 64 combines the correction result with the first test data to generate the second test data, the second test data is input into the operator test template of the test module 63 for testing, and before the second test result is output, the method further includes:

[0198] The data correction module 64 compares the second test data with the test data in the test database. If there is duplicate test data between the second test data and the test database, the second test data is deleted; if there is no duplicate test data between the second test data and the test database, the data correction module 64 inputs the second test data into the operator test template of the test module 63 for testing, and outputs the second test result.

[0199] Specifically, the data correction module 64 inputs the second test data into the operator test template of the test module 63 for testing. After the test module 63 outputs the second test result, the following steps are further included:

[0200] If the second test result does not include error information, the data correction module 64 saves the second test data to the test database;

[0201] If the second test result includes error information, the data correction module 64 again performs the steps of correcting the error parameters corresponding to the error information and outputting the correction results until the second test result does not include error information.

[0202] It should be noted that, when the device 6 provided in the above embodiment executes the test data generation method, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device provided in the above embodiment and the test data generation method embodiment belong to the same concept, and the implementation process thereof is detailed in the method embodiment, which will not be repeated here.

[0203] An embodiment of the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above-mentioned method embodiments when executing the program.

[0204] See also Figure 7 , which is a structural block diagram of an electronic device provided in an embodiment of the present application.

[0205] like Figure 7 As shown, the electronic device 700 includes: a processor 701 and a memory 702 .

[0206] In the embodiment of the present application, the processor 701 is the control center of the computer system, which can be a processor of a physical machine or a processor of a virtual machine. The processor 701 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 701 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array).

[0207] The processor 701 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also called a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state.

[0208] The memory 702 may include one or more computer-readable storage media, which may be non-transitory. The memory 702 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments of the present application, the non-transitory computer-readable storage medium in the memory 702 is used to store at least one instruction, which is used to be executed by the processor 701 to implement the method in the embodiment of the present application.

[0209] In some embodiments, the electronic device 700 further includes: a peripheral device interface 703 and at least one peripheral device. The processor 701, the memory 702 and the peripheral device interface 703 can be connected via a bus or a signal line. Each peripheral device can be connected to the peripheral device interface 703 via a bus, a signal line or a circuit board. Specifically, the peripheral devices include: a display screen 704, a camera 705 and an audio circuit 706. The peripheral device interface 703 can be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 701 and the memory 702.

[0210] In some embodiments of the present application, the processor 701, the memory 702, and the peripheral device interface 703 are integrated on the same chip or circuit board; in some other embodiments of the present application, any one or two of the processor 701, the memory 702, and the peripheral device interface 703 can be implemented on a separate chip or circuit board. This embodiment of the present application does not specifically limit this.

[0211] The display screen 704 is used to display a UI. The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 704 is a touch display screen, the display screen 704 also has the ability to collect touch signals on the surface or above the surface of the display screen 704. The touch signal can be input as a control signal to the processor 701 for processing. At this time, the display screen 704 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards.

[0212] In some embodiments of the present application, the display screen 704 may be one, which is arranged on the front panel of the electronic device 700; in other embodiments of the present application, the display screen 704 may be at least two, which are arranged on different surfaces of the electronic device 700 or are folded; in still other embodiments of the present application, the display screen 704 may be a flexible display screen, which is arranged on a curved surface or a folded surface of the electronic device 700. Even more, the display screen 704 may be arranged in a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 704 may be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode) and the like.

[0213] The camera 705 is used to capture images or videos. Optionally, the camera 705 includes a front camera and a rear camera. Usually, the front camera is arranged on the front panel of the electronic device, and the rear camera is arranged on the back of the electronic device. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments of the present application, the camera 705 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0214] The audio circuit 706 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 701 for processing. For the purpose of stereo sound collection or reduction, there may be multiple microphones, which are respectively arranged at different parts of the electronic device 700. The microphone may also be an array microphone or an omnidirectional collection microphone.

[0215] The power supply 707 is used to power various components in the electronic device 700. The power supply 707 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 707 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery charged through a wired line, and a wireless rechargeable battery is a battery charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0216] The electronic device structure block diagram shown in the embodiment of the present application does not constitute a limitation on the electronic device 700. The electronic device 700 may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0217] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of the method of any of the above embodiments are implemented. The computer-readable storage medium may include, but is not limited to, any type of disk, including a floppy disk, an optical disk, a DVD, a CD-ROM, a micro drive, and a magneto-optical disk, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory device, a magnetic card or an optical card, a nanosystem (including a molecular memory IC), or any type of medium or device suitable for storing instructions and / or data.

[0218] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution can be essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0219] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A test data generation method for a machine learning framework, It is characterized in that include: Obtain basic data related to the test of the machine learning framework to be tested, process the basic data into basic data vectors, and build a basic database based on the basic data vectors; Generate a prompt template that conforms to at least one data structure attribute according to the data structure attribute of the operator in the machine learning framework to be tested and the basic data; Inputting the prompt template into a semantic model to generate first test data; Inputting the first test data into an operator test template for testing, and outputting a first test result; If the first test result includes error information, traverse the vector database of the machine learning framework to be tested to obtain the code snippet corresponding to the error information, build a correction prompt template in combination with the error information, error parameters and the code snippet, input the correction prompt template into the semantic model, correct the error parameters through the semantic model, and output the correction result; Combining the correction result with the first test data to generate second test data; The step of generating a prompt template that conforms to at least one data structure attribute according to the data structure attribute of the operator in the machine learning framework to be tested and the basic data comprises: generating a control field based on at least one data structure attribute of the operator; traversing the basic database to obtain basic data related to the control field, and using the traversal result as the data field; and generating the prompt template in combination with the control field and the data field; The step of inputting the prompt template into the semantic model to generate the first test data includes: inputting the prompt template into the semantic model, determining the requirements of the data structure attributes based on the control field of the prompt template, determining basic data that conforms to at least one data structure attribute in the data field of the prompt template, modifying the basic data that conforms to the at least one data structure attribute, and generating the first test data based on the modified data.

2. The method according to claim 1, It is characterized in that The basic data is at least one of the following: historical vulnerability data of the machine learning framework to be tested, various data types in the machine learning framework to be tested and boundary values ​​corresponding to the various data types, and data of preset types corresponding to the machine learning framework to be tested.

3. The method according to claim 1, It is characterized in that The data structure attributes of the operator include type, shape and value; Wherein, the types include basic types and composite types; The shape includes the dimension and number of elements of the operator; The value is the value of each element in the operator.

4. The method according to claim 1, It is characterized in that The step of inputting the prompt template into the semantic model to generate first test data includes: The prompt template is input into a semantic model, a data structure attribute is determined based on a control field of the prompt template, and basic data that conforms to the at least one data structure attribute is determined in a data field of the prompt template as the first test data.

5. The method according to claim 1, It is characterized in that After generating the first test data, inputting the first test data into the operator test template for testing, and before outputting the first test result, further comprising: comparing the first test data with test data in a test database; If there is duplicate test data between the first test data and the test database, deleting the first test data; The step of inputting the first test data into an operator test template for testing and outputting a first test result includes: If there is no duplicate test data between the first test data and the test database, the first test data is input into the operator test template for testing, and a first test result is output.

6. The method according to claim 1, It is characterized in that After outputting the first test result, the method further includes: If the first test result does not include error information, the first test data is saved in a test database.

7. The method according to claim 1, It is characterized in that After inputting the first test data into the operator test template for testing and outputting the first test result, the method further includes: If the first test result includes error information, the error parameters corresponding to the error information are corrected and a correction result is output.

8. The method according to claim 7, It is characterized in that The step of correcting the error parameter corresponding to the error information and outputting the correction result includes: Traversing the vector database of the machine learning framework to be tested to obtain the code snippet corresponding to the error information; A correction prompt template is constructed in combination with the error information, the error parameters and the code snippet, the correction prompt template is input into the semantic model, the error parameters are corrected through the semantic model, and a correction result is output.

9. The method according to claim 7, It is characterized in that After outputting the correction result, the method further includes: The correction result and the first test data are combined to generate second test data, the second test data is input into the operator test template for testing, and a second test result is output.

10. The method according to claim 9, It is characterized in that After combining the correction result with the first test data to generate second test data, inputting the second test data into an operator test template for testing and before outputting the second test result, the method further includes: comparing the second test data with the test data in the test database; If there is duplicate test data between the second test data and the test database, deleting the second test data; The step of inputting the second test data into the operator test template for testing and outputting the second test result includes: If there is no duplicate test data between the second test data and the test database, the second test data is input into the operator test template for testing, and a second test result is output.

11. The method according to claim 9, It is characterized in that The step of inputting the second test data into the operator test template for testing and outputting the second test result comprises: If the second test result does not include error information, saving the second test data to a test database; If the second test result includes error information, the step of correcting the error parameter corresponding to the error information and outputting the correction result is performed again until the second test result does not include error information.

12. A test data generation device for a machine learning framework, It is characterized in that include: A basic data module, used to obtain basic data related to the test of the machine learning framework to be tested, process the basic data into basic data vectors, and build a basic database based on the basic data vectors; A test data generation module, used to generate a prompt template that conforms to at least one data structure attribute according to the data structure attributes of the operator in the machine learning framework to be tested; The test data generation module is further used to input the prompt template into the semantic model to generate first test data, input the first test data into the operator test template for testing, and output a first test result; If the first test result includes error information, traverse the vector database of the machine learning framework to be tested to obtain the code snippet corresponding to the error information, build a correction prompt template in combination with the error information, error parameters and the code snippet, input the correction prompt template into the semantic model, correct the error parameters through the semantic model, and output the correction result; Combining the correction result with the first test data to generate second test data; The test data generation module is specifically used to: generate a control field based on at least one data structure attribute of the operator; traverse the basic database to obtain basic data related to the control field, and use the traversal result as the data field; and generate the prompt template by combining the control field and the data field; The test data generation module is specifically used to: input the prompt template into the semantic model, determine the requirements of the data structure attributes based on the control field of the prompt template, determine the basic data that conforms to the at least one data structure attribute in the data field of the prompt template, modify the basic data that conforms to the at least one data structure attribute, and generate the first test data based on the modified data.

13. The device according to claim 12, It is characterized in that The device also includes a data correction module. After the test data generation module inputs the first test data into the operator test template of the test module for testing, if the first test result generated by the test module includes error information, the data correction module corrects the error parameters corresponding to the error information and outputs the correction result.

14. The device according to claim 13, It is characterized in that The data correction module corrects the error parameters corresponding to the error information and outputs the correction result, including: The data correction module traverses the vector database of the machine learning framework to be tested to obtain the code snippet corresponding to the error information; The data correction module constructs a correction prompt template in combination with the error information, the error parameters and the code snippet, and inputs the correction prompt template into the semantic model. The data correction module corrects the error parameters through the semantic model and outputs the correction result.

15. The device according to claim 13, It is characterized in that The data correction module corrects the error parameters corresponding to the error information and outputs the correction result, further comprising: The data correction module combines the correction result and the first test data to generate second test data, inputs the second test data into the operator test template of the test module for testing, and the test module outputs a second test result.

16. The device according to claim 15, It is characterized in that The data correction module inputs the second test data into the operator test template of the test module for testing, and after the test module outputs the second test result, the method further includes: If the second test result does not include error information, the data correction module saves the second test data to a test database; If the second test result includes error information, the data correction module again performs the step of correcting the error parameters corresponding to the error information and outputting the correction result until the second test result does not include error information.

17. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the program, the steps of the method according to any one of claims 1 to 11 are implemented.

18. A non-transitory computer readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.