Full-process automatic unit testing method and device for AI chip operator and medium

Abstract

The invention discloses a full-process automatic unit testing method, device and equipment for an AI chip operator, a medium and a product. The method comprises the following steps: generating an engineering directory and a test framework design document through an architect agent according to operator interface information and a DSL operator design document knowledge base; the method comprises the following steps of: generating a test code for unit test by a software engineer agent according to a test framework design document and a domain knowledge base, and storing the test code in an engineering directory; multiple rounds of interaction are carried out through a software engineer agent and a test engineer agent, a test code is gradually corrected into a target test code which can be successfully compiled, a generated test case can be successfully executed and the generated test case meets the code coverage rate requirement, and a target operator code passing a unit test is obtained; and submitting the target operator code and the target test code which pass the normativity check to a code warehouse through a code auditing engineer agent so as to automatically realize the whole-process unit test of the AI chip operator.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence (AI) and chip technology, in particular to the field of multi-agent collaboration, and more particularly to an AI chip operator full-process automatic unit test method, device, equipment, medium and product. BACKGROUND

[0002] In the software development of AI chips, a large number of high-quality unit test codes need to be written for AI operators. This work is critical and heavy, and existing technologies often require developers to write manually, which often brings defects such as low efficiency, insufficient test coverage, high maintenance cost, and unstable test quality.

[0003] At present, there are implementation methods on the market that use large language models (LLM) to build test codes based on general programming languages. However, in the field of AI chips, due to the use of domain-specific programming languages (DSL) in the software stack of AI chips, the process of unit testing operators requires frequent calls to special tool chains (such as compilers, debuggers, and coverage test tools) in the software stack of AI chips to cooperate with the test process, and the test process needs to be strictly divided into stages to achieve fine process control. The existing test code automatic generation method cannot be directly applied to the field of unit testing of AI operators. SUMMARY

[0004] The embodiments of the present application provide an AI chip operator full-process automatic unit test method, device, equipment, medium and product, which automatically realizes the full-process unit test of the AI chip operator through the collaborative work between the intelligent agents of different roles in the multi-agent collaboration system, and significantly improves the development efficiency of the AI chip operator test code.

[0005] According to an aspect of the embodiments of the present application, an AI chip operator full-process automatic unit test method is provided, which is executed by a multi-agent collaboration system including an architect agent, a software engineer agent, a test engineer agent and a code review engineer agent, and the method comprises:

[0006] The architect agent generates an engineering directory and a test framework design document according to operator interface information of a target operator and a DSL operator design document knowledge base;

[0007] The test code for unit testing the operator code of the target operator is initialized and stored in an engineering directory according to a test framework design document and a domain knowledge base by the software engineer agent;

[0008] The test code stored in the engineering directory is gradually corrected to target test code that can be successfully compiled, the generated test case can be successfully executed, and the generated test case meets the code coverage requirement by the software engineer agent and the test engineer agent through multiple rounds of interaction, and the target operator code of the target operator that passes the unit test is obtained;

[0009] The target operator code and the target test code that pass the specification check are submitted to a code repository by the code auditing engineer agent.

[0010] According to another aspect of the embodiment of the present application, an AI chip operator full-process automatic unit testing device is also provided, which is configured in a multi-agent cooperation system including an architect agent, a software engineer agent, a test engineer agent and a code auditing engineer agent, and the device comprises:

[0011] A design document generation module is configured to generate an engineering directory and a test framework design document by the architect agent according to operator interface information of a target operator and a DSL operator design document knowledge base;

[0012] A test code initialization generation module is configured to initialize and generate test code for unit testing the operator code of the target operator and store the test code in an engineering directory by the software engineer agent according to a test framework design document and a domain knowledge base;

[0013] A multi-agent interaction module is configured to gradually correct the test code stored in the engineering directory to target test code that can be successfully compiled, the generated test case can be successfully executed, and the generated test case meets the code coverage requirement by the software engineer agent and the test engineer agent through multiple rounds of interaction, and obtain the target operator code of the target operator that passes the unit test;

[0014] An operator code submission module is configured to perform specification checking on the target operator code and the target test code by the code auditing engineer agent, and submit the target operator code and the target test code that pass the specification check to a code repository.

[0015] According to another aspect of the embodiment of the present application, an electronic device is also provided, which comprises:

[0016] At least one processor; and

[0017] a memory in communication with the at least one processor; wherein

[0018] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the AI chip operator full-process automated unit test method of any one of the embodiments of the application.

[0019] According to another aspect of the embodiments of the application, a computer readable storage medium is provided, which stores computer instructions for enabling a processor to implement the AI chip operator full-process automated unit test method of any one of the embodiments of the application when executed by the processor.

[0020] According to another aspect of the embodiments of the application, a computer program product is also provided, which comprises a computer program for enabling a processor to implement the steps of the AI chip operator full-process automated unit test method of any one of the embodiments of the application when executed by the processor.

[0021] The technical scheme of the embodiments of the application constructs an intelligent agent collaboration system in which multiple intelligent agents collaborate, generates an engineering directory and a test framework design document by an architect intelligent agent according to operator interface information of a target operator and a DSL operator design document knowledge base, initializes test code for unit testing of operator code of the target operator and stores the test code in the engineering directory according to a test framework design document and a domain knowledge base by a software engineer intelligent agent, gradually corrects the test code stored in the engineering directory to target test code that can be successfully compiled, the generated test cases can be successfully executed, and the generated test cases meet the code coverage requirements by multiple rounds of interaction between the software engineer intelligent agent and a test engineer intelligent agent, and obtains target operator code of the target operator that passes the unit test, performs standardization checking on the target operator code and the target test code by a code review engineer intelligent agent, and submits the target operator code and the target test code that pass the standardization checking to a code repository, which can significantly improve the development efficiency of AI chip operator test code, automatically generate unit tests for a large number of AI operators, effectively reduce the writing, maintenance cost and complexity of test code, and stably ensure the quality of generated test code, reduce the influence of human factors, effectively overcome the poor adaptability of general LLM in the AI chip DSL field, and ultimately realize a high-reliability and high-efficiency AI operator unit test automation pipeline that generates and verifies at the same time and verifies before entering the repository.

[0022] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the application, nor to limit the scope of the application. Other features of the application will become apparent from the following description. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of a fully automated unit testing method for AI chip operators according to an embodiment of the present invention;

[0025] Figure 2 This is a flowchart of another fully automated unit testing method for AI chip operators provided by an embodiment of the present invention;

[0026] Figure 3 This is a schematic diagram of the structure of the fully automated unit testing method for AI chip operators implemented through multi-agent collaboration, as described in this embodiment of the invention.

[0027] Figure 4 This is a schematic diagram of the structure of a fully automated unit testing device for AI chip operators according to an embodiment of the present invention;

[0028] Figure 5 This is a schematic diagram of the structure of an electronic device that implements the fully automated unit testing method for AI chip operators according to embodiments of the present invention. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0030] It is to be understood that the terminology "first", "second" and the like used in the specification and the claims of the application as well as the preceding drawings is merely intended to distinguish between similar objects and not necessarily to describe a specific sequential or chronological order. It is to be understood that the use of data herein so described makes it possible, in appropriate cases, to adapt the application to diverse embodiments without necessarily departing from the scope of the application. Furthermore, the terms "comprise" and "include" and their conjugates, as used in the specification and the claims of the application, are intended to encompass the inclusions of steps or elements that are not necessarily limiting, for example, a process, method, system, product or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements that are clearly recited, but can include additional steps or elements that are not expressly listed or inherent to such process, method, product or apparatus.

[0031] Figure 1 A flowchart of an AI chip operator full-process automated unit test method provided by an embodiment of the application, the embodiment can be applicable to the collaborative work of intelligent agents of various roles in a multi-agent collaboration system, and the full-process unit test of the AI chip operator is realized automatically. The method can be executed by an AI chip operator full-process automated unit test device configured in the multi-agent collaboration system. The device can be realized in the form of hardware and / or software, and can generally be configured in a terminal device, a server or a server cluster having a data processing function. The multi-agent collaboration system specifically includes an architect intelligent agent, a software engineer intelligent agent, a test engineer intelligent agent and a code review engineer intelligent agent.

[0032] Correspondingly, as shown in Figure 1 the method can include:

[0033] S110, generating an engineering directory and a test framework design document by the architect intelligent agent according to operator interface information of a target operator and a DSL operator design document knowledge base.

[0034] The architect intelligent agent (also referred to as Architect) plays the role of an architect in the manual design and execution of the full-process unit test process of the AI chip operator, and is responsible for analyzing the actual test task requirements.

[0035] The target operator can be understood as a basic functional unit adapted to the AI chip and used to perform a specific computing task, for example, a node in a neural network computation graph, used to implement an operation or transformation, such as convolution, matrix multiplication or activation, etc.

[0036] The operator interface information can be understood as information defining the core specification of the interaction between the operator and the system. In a specific example, the operator interface information is in the following form:

[0037] ConvBiasActivation(Tensor&output, const Tensor&input, const Tensor&weight, const Tensor&bias, const Size_t&stride, const Size_t&padding, const Size_t&dilation, int64_t groups, const ActivationMode_t activation_mode);

[0038] Specifically, the dimension information of the input and output tensors of a composite operator for performing convolution + bias + activation, the dimension information of the convolution kernel weight tensor and the bias term tensor, the convolution kernel sliding step, the input edge padding amount, the dilation rate of the dilated convolution, the group convolution parameter, and the specified activation type are defined.

[0039] It can be understood that after the developer completes the writing of the execution logic of the target operator, the target operator can be packaged in the form of an interface. In this way, the target operator can be called and executed through interface calling, and various specification requirements that the target operator needs to follow when being called can be defined through operator interface information.

[0040] The DSL operator design document knowledge base can be understood as a core database for storing and managing operator design specifications during AI chip development. In the DSL operator design document knowledge base, information corresponding to each operator in the AI chip is stored, including operator interface, operator function description, input and output tensor dimension limit of the operator, and data types supported by the operator. In an optional implementation of the embodiment, the DSL operator design document knowledge base is pre-stored in the internal of the architect agent, and is updated and maintained in real time by the architect agent.

[0041] In software development, the engineering directory is a structured organization of the project file system. Through the division of the engineering directory, the physical isolation of code, resources, and configurations can be achieved. That is, in the engineering directory, the storage location of the test code is defined.

[0042] The test framework design document is a technical document that describes the architecture, functional modules, and implementation logic of the automated test framework, and is used to unify the understanding of the framework design of the development team and ensure the scalability and maintainability of the framework. The document usually contains the component interaction relationship of the framework, data flow design, and extension interface specification. The test framework design document can be understood as the basis for constructing software test cases.

[0043] In an optional implementation of the embodiment, the engineering directory and the test framework design document are generated by the architect agent according to the operator interface information of the target operator and the DSL operator design document knowledge base, which can include:

[0044] S1101, retrieving, by the architect agent, the DSL operator design document knowledge base stored locally according to the operator interface information of the target operator to obtain at least one operator description information matched with the target operator.

[0045] The operator description information includes operator function description information, operator input and output tensor dimension limit information, and data type information supported by the operator.

[0046] S1102, constructing, by the architect agent, a first prompt word text according to each of the operator description information, and inputting the operator interface information of the target operator and the first prompt word text into a pre-constructed large language model to generate the engineering directory and the test framework design document.

[0047] The test framework design document includes operator semantics, file name, function body name, operator list, programming timing diagram, and key constraint conditions.

[0048] The large language model (Large Language Model, LLM) can be understood as a pre-trained model based on a deep neural network architecture (for example, Transformer) with a large number of model parameters (for example, more than 1 billion), which can be a generative AI system, that is, after training with a large amount of data, it can process text, images, or audio and video multi-source input data to achieve natural language understanding, generation and reasoning to obtain new content. Typically, the large language model can be a self-attention language model that supports setting the context size.

[0049] In the embodiment, if the target operator is a single operator, there is only one item in the list content of the operator list, and if the target operator is a fusion operator, the list content of the operator list is all single operators after the fusion operator is decomposed and arranged according to the calculation order. The programming timing diagram can specifically include the timing and calling relationship of multiple function bodies, and the dependency relationship between single operators after the fusion operator is decomposed, and the key constraint conditions can specifically include input and output tensor dimensions, data types, memory layout requirements, and precision verification requirements and other information.

[0050] S120, initializing, by the software engineer agent, the test code storage under the engineering directory for unit testing of the operator code of the target operator according to the test framework design document and the domain knowledge base.

[0051] In the software engineer agent (also referred to as Engineer), the role of a software engineer is played in the process of artificially designing and performing full-process unit testing of AI chip operators. The software engineer agent is responsible for receiving the test framework design document generated by the architect, combining the pre-constructed and locally maintained domain knowledge base, and generating specific test code bodies (including CPU-side code and chip-side code, etc.), error fixes, and specific test cases (also referred to as Test Case), etc.

[0052] In an optional implementation of the embodiment, the domain knowledge base is pre-stored in the software engineer agent and is updated and maintained by the software engineer agent in real time. The domain knowledge base specifically includes a small sample template library and an operator semantic knowledge base.

[0053] The small sample template library stores the mapping relationship between the interface data types of each operator and the code templates, and the code templates include structure code, CPU-side code, AI chip-side code, result verification code, and main function code.

[0054] The operator semantic knowledge base stores the mapping relationship between each operator type and operator type description information, and the operator type description information includes operator mathematical definition and properties, input and output data logical relationship of the operator, precision characteristics of the operator, equivalence class partitioning rules of the operator, and boundary rules of the operator.

[0055] The interface data type is used to define the abstract type of object structure or behavior specification of the operator interface. By declaring method signatures, constants, and other members, it forces the implementation of the class to follow a specific behavior specification. It does not contain specific implementation and only defines the rules of "what can be done". The operator type refers to the category system divided according to the functional characteristics and mathematical properties of the operator, which is used to identify the processing method of the operator for the input data. In mathematics and computer science, operator types are usually represented as mapping rules (such as linear or nonlinear) or computing paradigms (such as arithmetic or logical operations) differences.

[0056] In a specific example, the small sample categories mainly include Elementwise (element-wise operation), Convolution (convolution operation), Reduction (reduction operation), and Random (random operation) and more than ten kinds. A verified code template can be created for each type of operator, and finally a structured small sample template database is built. The template content includes structure code, CPU code, AI chip code, result verification code and main function code. Such as tensor construction code, memory allocation code, input data initialization mode, operator call mode. A small number of basic templates can be extended to cover a large number of related operators, achieving broad-spectrum coverage of operator types. For example, for the add basic summation operator template, related operator templates such as sub, mul, and add_relu can be extended and built accordingly.

[0057] In an optional implementation of the embodiment, one or more interface data types closest to the test framework design document can be retrieved from the small sample template library based on the semantic features of the test framework design document, and one or more operator types closest to the test framework design document can be retrieved from the operator semantic knowledge base. Then, one or more code templates matching the test framework design document and one or more operator type description information can be obtained.

[0058] Then, based on the matched code templates and operator type description information, a prompt word template can be constructed, and the prompt word template and the test framework design document are input into the large language model together. The large language model initializes to generate test code for unit testing of the operator code of the target operator and stores the test code in the engineering directory.

[0059] The large language model can generate complete test code at one time, or the large language model can generate multiple test code slices in multiple times and store them in the matching slice storage location in the engineering directory to organize complete test code in the engineering directory. The embodiment does not limit this. The method of generating multiple test code slices in multiple times by the large language model can effectively solve the information decay problem caused by long code generation.

[0060] S130, through the multi-round interaction between the software engineer agent and the test engineer agent, the test code stored in the engineering directory is gradually corrected to the target test code that can be successfully compiled, the generated test case can be successfully executed, and the generated test case meets the code coverage requirement, and the target operator code of the target operator that passes the unit test is obtained.

[0061] The test engineer agent (also referred to as Tester) plays the role of a test engineer in the process of artificially designing and performing full-process unit testing of the AI chip operator, and deeply integrates the chip software stack tool chain (for example, a compiler, a debugger, a coverage detection tool, a test execution engine, and the like). The test engineer agent is responsible for receiving the test code and test cases generated by the software engineer agent, compiling the test code on a device with a compilation environment, executing the test cases on a device with an AI chip, collecting execution results and logs, analyzing code coverage, and performing result verification (typically, numerical accuracy comparison). During the execution verification, the accuracy error (for example, maximum error, average error, number of points exceeding the threshold, and the like) needs to be reported in detail, and this information is fed back to the software engineer agent. The coverage of the test cases on the equivalence class and the boundary condition is monitored and analyzed, and the insufficiently covered areas are fed back to the software engineer agent to generate supplementary test cases.

[0062] The process of multiple rounds of interaction between the software engineer agent and the test engineer agent can be specifically described as follows:

[0063] Software engineer agent: generate test code -> test engineer agent: compile test code -> if compilation succeeds -> software engineer agent: generate test cases -> test engineer agent: execute test and detect code coverage -> test passes and code coverage improves -> interaction between the two agents ends.

[0064] When the test engineer agent fails to compile, the compilation error is fed back to the software engineer agent, which locates and fixes the error -> sends it to the test engineer agent for recompilation. If the test case fails or the coverage is insufficient, the test engineer agent feeds back the execution failure log and coverage report to the software engineer agent, guiding the software engineer agent to generate new test cases. If the accuracy verification fails, the test engineer agent feeds back the error report -> the software engineer agent analyzes the cause (for example, whether the test input range needs to be adjusted? Whether the comparison logic threshold needs to be modified? Whether the reference calculation needs to be optimized?) -> the software engineer agent fixes and adjusts -> the test engineer agent re-verifies.

[0065] Through the joint efforts of the two agents, a closed-loop iteration of “generation -> verification -> feedback -> repair -> re-verification” is formed, until the target test code that meets the code coverage requirement is successfully generated, and the target operator code of the target operator that passes the unit test is obtained.

[0066] S140, through the code review engineer agent, the target operator code and the target test code are checked for specification, and the target operator code and the target test code that pass the specification check are submitted to the code repository.

[0067] Among them, the code review engineer agent (also known as Reviewer) plays the role of a code review engineer in the whole process of unit testing of AI chip operators designed and executed by humans, is responsible for the normative review of the target operator code and the target test code that finally pass the test (for example, naming specification, comment integrity and code style, etc.), and submits the various types of code that meet the specifications to the code repository.

[0068] The technical scheme of the embodiment of the application constructs an intelligent agent collaboration system in which multiple roles of intelligent agents collaborate, generates an engineering directory and a test framework design document through an architect intelligent agent according to operator interface information of a target operator and a DSL operator design document knowledge base, initializes the generation of test code for unit testing of the operator code of the target operator and stores the test code in the engineering directory through a software engineer intelligent agent according to the test framework design document and the domain knowledge base, gradually corrects the test code stored in the engineering directory to target test code that can be successfully compiled, the generated test cases can be successfully executed, and the generated test cases meet the code coverage requirements through multiple rounds of interaction between the software engineer intelligent agent and the test engineer intelligent agent, and obtains the target operator code of the target operator that passes the unit test; and the implementation mode of performing normative inspection on the target operator code and the target test code through a code review engineer intelligent agent and submitting the target operator code and the target test code that pass the normative inspection to a code repository can significantly improve the development efficiency of AI chip operator test code, automatically generate unit tests for a large number of AI operators, in addition, can effectively reduce the writing, maintenance cost and complexity of test code, stably guarantee the quality of generated test code, reduce the influence of human factors, effectively overcome the poor adaptability of general LLM in the AI chip DSL field, and finally realize a high-reliability and high-efficiency AI operator unit test automation pipeline of generate-verify-verify and enter the repository after verification.

[0069] In an optional implementation of the embodiment, the method can further include:

[0070] The operator semantic knowledge base is constructed by at least one of the following: analyzing and processing each DSL operator design document in the DSL operator design document knowledge base through a pre-constructed large language model, analyzing and processing the header file of each DSL operator through the large language model, and extracting and analyzing a high-level semantic operator in a standard deep learning framework.

[0071] In this optional implementation, the operator semantic knowledge base can be constructed through the following three channels.

[0072] I. Use LLM to analyze the DSL operator design document to obtain the mathematical definition and properties of the operator, the logical relationship between input and output data, and the precision characteristics;

[0073] II. Use LLM to analyze the.h code of the operator (i.e., the header file code) to obtain the test strategy of the operator, such as equivalence class partitioning rules and boundary condition rules;

[0074] III. Extract high-level semantic operators (such as torch.nn.Conv2d, torch.nn.Softmax, etc.) in mainstream frameworks (typically, PyTorch and TensorFlow, etc.), store their interface signatures, functional semantic descriptions, and typical usage examples.

[0075] The final operator semantic knowledge base contains the following content:

[0076] 1. Operator mathematical definition and properties: formally describe the mathematical formula, calculation logic (such as the sliding window multiplication and addition of conv2d, the exponent and normalization of softmax), and algebraic properties (such as linearity, commutativity, and associativity, etc.) of the operator.

[0077] 2. Input and output data logical relationship: describe how input parameters affect output (dimension transformation rules, data type constraints, value range dependencies).

[0078] 3. Precision characteristics: typical numerical sensitivity, types of easy-to-accumulate errors (such as reduce operator sensitivity to accumulation order), recommended verification methods and thresholds.

[0079] 4. Equivalence class partitioning rules: standardized equivalence class partitioning methods for input parameters of the operator (example: for floating-point input tensors, equivalence classes can be divided into: positive numbers, negative numbers, zero, NaN, Inf, -Inf, and subnormal numbers).

[0080] 5. Boundary condition rules: key boundary value definitions for input parameters and calculation logic of the operator (example: for the kernel_size (convolution kernel size) parameter of the convolution operator, boundary values include: 1x1, 3x3 (common minimum non-1 odd kernel), input tensor H / W (height / width ratio) equal to kernel_size, H / W less than kernel_size; for the reduce_sum operator (reduce sum operator), generate test cases for empty input tensors).

[0081] Through the above settings, the problem of sparse domain knowledge can be effectively solved.

[0082] Figure 2Fig. 6 is a flowchart of another AI chip operator full-process automatic unit test method provided by an embodiment of the present application, which is optimized based on the above embodiments. In this embodiment, the operation of "initializing and generating test code for unit testing of the operator code of the target operator by the software engineer agent according to the test framework design document and the domain knowledge base, and storing the test code in the engineering directory", and the operation of "gradually modifying the test code stored in the engineering directory to the target test code that can be successfully compiled, the generated test case can be successfully executed, and the generated test case meets the code coverage requirement, and obtaining the target operator code of the target operator that passes the unit test, by the software engineer agent and the test engineer agent through multiple rounds of interaction" are specifically implemented.

[0083] Correspondingly, as shown in Figure 2 the method comprises:

[0084] S210, generating an engineering directory and a test framework design document by an architect agent according to operator interface information of a target operator and a domain-specific programming language DSL operator design document knowledge base.

[0085] S220, generating multiple test code slices matched with the target operator by a software engineer agent according to the test framework design document, combining a small sample template library in the locally pre-stored domain knowledge base and an operator semantic knowledge base, and slicing.

[0086] In this embodiment, considering that when generating a complete test code at one time, due to the code length of the complete test code being too long, exceeding the context window limit (for example, 2048 tokens) of a large language model, the large language model cannot maintain long-range dependencies, resulting in logical discontinuity or functional missing, and other problems. Based on this, the inventors creatively propose an implementation manner of slicing to generate multiple test code slices matched with the target operator, and organizing the multiple test code slices to obtain a complete test code.

[0087] Correspondingly, in an optional implementation manner of this embodiment, generating multiple test code slices matched with the target operator by the software engineer agent according to the test framework design document, combining the small sample template library in the locally pre-stored domain knowledge base and the operator semantic knowledge base, and slicing can include:

[0088] S2201, converting the test framework design document into a first comparison dense vector feature based on semantics and a first comparison sparse vector feature based on word segmentation by the software engineer agent.

[0089] In this embodiment, in order to improve the accuracy of subsequent retrieval, the test framework design document can be converted into dense vector features (i.e., first comparison dense vector features) and sparse vector features (i.e., first comparison sparse vector features) respectively through different vector conversion methods.

[0090] Optionally, the test framework design document can be converted into first comparison sparse vector features through One-Hot encoding or TF-IDF vectorization, etc. based on word segmentation conversion methods, or converted into first comparison dense vector features through static word embedding (such as Word2Vec) or context-related models (for example, BERT) and other semantic-based conversion methods. The present embodiment does not limit this.

[0091] S2202, through the software engineer agent, the first comparison dense vector features and the first comparison sparse vector features are matched with the standard dense vector features and the standard sparse vector features of the interface data types of each operator in the small sample template library respectively, and the category weights of the dense vector and the sparse vector are combined to select a first number of target code templates in the small sample template library.

[0092] In this embodiment, the interface data types of each operator in the small sample template library are also stored through mixed vectors (sparse vector features and dense vector features). That is, the standard dense vector features based on semantics and the standard sparse vector features based on word segmentation.

[0093] Further, by matching the first comparison dense vector features and the first comparison sparse vector features with the standard dense vector features and the standard sparse vector features of the interface data types of each operator in the small sample template library, the first similarity results between the first comparison dense vector features and the standard dense vector features of the interface data types of each operator, and the second similarity results between the first comparison sparse vector features and the standard sparse vector features of the interface data types of each operator can be obtained.

[0094] Further, the category weights of the dense vector and the sparse vector in the small sample template library can be predetermined in advance, for example, the category weight of the dense vector is 0.7, and the category weight of the sparse vector is 0.3. Further, through weighted summation, the overall similarity scores corresponding to each interface data type and the test framework design document respectively can be obtained.

[0095] For example, the first similarity result between the standard dense vector feature of the interface data type A and the first compared dense vector feature is K1, the second similarity result between the standard sparse vector feature of the interface data type A and the first compared sparse vector feature is K2, and then the overall similarity score of the interface data type A corresponding to the test framework design document can be calculated in the manner of K1*0.7+K2*0.3.

[0096] Finally, by the order of the overall similarity score from high to low, the first number (for example, 3, 4, or 5, etc.) of target interface data types can be selected from the small sample template library, and then the target code template matching the target interface data type can be correspondingly screened out.

[0097] S2203, by the software engineer agent, the first compared dense vector feature and the first compared sparse vector feature are matched with the standard dense vector feature and the standard sparse vector feature of each operator type in the operator semantic knowledge base respectively, and the second number of target operator type description information is selected in the operator semantic knowledge base in combination with the category weight of the dense vector and the sparse vector.

[0098] In the embodiment, each operator type in the operator semantic knowledge base is also stored in the manner of a mixed vector, and then the second number of target operator type description information matching the test framework design document can also be selected in the operator semantic knowledge base by the implementation manner of S2202.

[0099] The first number can be the same as or different from the second number, and the embodiment does not limit this.

[0100] In a specific example, according to the interface data type retrieval of the operator, a template similar in structure can be retrieved in the small sample template library, for example, input ConvBiasActivation, and retrieve the Convolution class template. According to the interface retrieval of the operator, the operator semantic knowledge base is retrieved, for example, if there is a high-level semantic operator implementation, the mainstream framework is used as the CPU test, if there is no high-level semantic operator implementation (for example, a fusion operator), the mathematical formula and calculation logic of the operator mathematics are used to complete the corresponding CPU test, for example, input ConvBiasActivation, and use the mathematical formula and calculation logic of convolution+add+activation three, and the input and output data logic relationship to complete.

[0101] Then, the retrieved template and operator core semantic knowledge can be used as a context to cooperate with the corresponding prompt word engineering to enhance the test code generated by the LLM.

[0102] Specifically, when the software engineer agent needs to generate test cases for a specific operator, the shape transformation can be set according to the input and output data logical relationship retrieved according to the interface of the operator, the composite comparison of atol (absolute tolerance) or rtol (relative tolerance) can be set according to the precision characteristics, the test strategy rule can be set according to the equivalence class division rule, the boundary condition rule and the coverage report, and the test engineering guidance is provided for the LLM.

[0103] S2204, by the software engineer agent, according to the slice type of the test code slice, each target code template, and each target operator type description information, a second prompt word text corresponding to each slice type is constructed.

[0104] As described above, in the present embodiment, instead of generating a complete test code at one time, a plurality of test code slices are generated at multiple times, and an implementation mode of obtaining a complete test code is organized. Therefore, when constructing the prompt word text, the embodiments of the present application also construct a plurality, that is, construct a second prompt word text corresponding to each slice type.

[0105] The slice type includes preambles, structure body code, CPU end code, AI chip end code, result verification code, main function code, and test case code.

[0106] That is, considering that generating a complete AI chip operator test code is easy to cause LLM context truncation, information loss, and logical error problems. The embodiments of the present application creatively propose an implementation mode of using structured code slicing, and dividing the complete operator unit test code into relatively independent slices according to the function logic:

[0107] 1. Preambles: header file inclusion, auxiliary function declaration.

[0108] 2. Config Struct: define test configuration parameters (such as data type, dimension, step size, and bias, etc.).

[0109] 3. CPU: implement the reference calculation logic of the operator on the CPU (for result verification).

[0110] 4. Device: code for calling the target AI chip operator.

[0111] 5. Result Verification: implement the verification logic of the chip calculation result and the CPU reference result (such as precision error).

[0112] 6. Main: test case json file parsing, call test code.

[0113] 7. Test Cases: define specific input data, call the above code, trigger verification.

[0114] Correspondingly, in this embodiment, test code slices corresponding to different slice types need to be generated respectively, and then second prompt text corresponding to different slice types needs to be constructed first.

[0115] Optionally, according to different slice types, local information associated with each target code template and each target operator type description information can be extracted respectively to obtain second prompt text corresponding to each slice type. For example, keyword matching or semantic correlation detection can be used to extract the local information.

[0116] That is, the software engineer agent generates each test code slice in order or as needed. A special prompt strategy (Prompt) is designed for each test code slice, and the relevant information of the test code slice retrieved from the template and operator semantic knowledge base is combined to accurately deliver the domain-specific requirements and constraints of this part.

[0117] S2205, by the software engineer agent, sequentially obtain the current second prompt text corresponding to a current slice type.

[0118] S2206, by the software engineer agent, input the test framework design document and the second prompt text into the pre-constructed large language model to obtain the current test code slice corresponding to the current slice type.

[0119] S2207, by the software engineer agent, return to execute the operation of sequentially obtaining the current second prompt text corresponding to a current slice type until the processing of all slice types is completed to obtain all test code slices.

[0120] S230, by the software engineer agent, store each test code slice to the matching slice storage location under the engineering directory to organize the complete test code under the engineering directory.

[0121] In this embodiment, after the large language model generates test code slices corresponding to different slice types based on different second prompt texts, each test code slice can be stored in the matching slice storage location under the engineering directory, and then the complete test code can be organized under the engineering directory.

[0122] S240, by the test engineer agent, after obtaining the current test code from the engineering directory, calling the pre-integrated chip software stack tool chain to compile the code, and feeding back the successful compilation result to the software engineer agent when the code compilation is successful.

[0123] The chip software stack tool chain can include a compiler, an executor, a coverage detection tool, and a test execution engine.

[0124] In this embodiment, the various test tools in the test engineer agent are pre-integrated into the chip software stack tool chain, and the test engineer agent can implement a complete code testing process.

[0125] S250, by the software engineer agent, when receiving the successful compilation result, generating a matching current test case according to the current test code matching the successful compilation result, and feeding back the current test case to the test engineer agent.

[0126] S260, by the test engineer agent, calling the chip software stack tool chain to test the current operator code of the target operator based on the current test case, and determining the current test code as the target test code and the current operator code as the target operator code when the current test case is executed successfully and the coverage of the current operator code meets the requirements.

[0127] S270, by the test engineer agent, when determining that the code compilation fails, the current test case fails to execute, or the coverage of the current operator code does not meet the requirements, feeding back the compilation failure result, the current test case execution failure result, or the code coverage not meeting the requirements as abnormal result information to the test engineer agent.

[0128] S280, by the software engineer agent, when receiving the abnormal result information, converting the abnormal result information into a second comparison dense vector feature based on semantics and a second comparison sparse vector feature based on word segmentation.

[0129] S290, by the software engineer agent, matching the second comparison dense vector feature and the second comparison sparse vector feature with the standard dense vector feature and the standard sparse vector feature of the standard error information in the fault repair knowledge base in the local pre-stored domain knowledge base respectively, and combining the category weights of the dense vector and the sparse vector to perform a screening operation of the standard repair strategy in the fault repair knowledge base according to the preset screening rule.

[0130] The fault repair knowledge base stores a mapping relationship between the standard error information and the standard repair strategy.

[0131] In the embodiment, a fault repair knowledge base is further stored in the field knowledge base. In the fault repair knowledge base, each standard error information is also stored in a hybrid vector (standard dense vector feature and standard sparse vector feature), and a mapping relationship between each standard error information and a matched standard repair strategy is pre-established. When the fault repair knowledge base is initially constructed, some common errors of DSL are manually added with corresponding repair methods. Subsequently, the fault repair knowledge base is continuously updated. New errors added in the pipeline are recorded, including the LLM repair method, and errors that are not solved after exceeding the iteration number of the pipeline are manually checked after the repair opinion of the LLM is added with the correct repair method.

[0132] Further, when the test engineer agent feeds back the abnormal result information when the code compilation fails, the current test case execution fails, or the coverage of the current operator code does not meet the requirements, the matched standard repair strategy can also be retrieved through the hybrid retrieval method.

[0133] In which, the comprehensive similarity score between each standard error information and the abnormal result information can be calculated based on the method of S2202.

[0134] In an optional implementation of the embodiment, the following screening rule can be set: in all standard error information with a comprehensive similarity score greater than or equal to a preset score value (for example, which can be 80%), a preset number (for example, which can be 3) of standard repair strategies are screened out. Further, in a specific example, after the comprehensive similarity scores of the standard error information are calculated, if the comprehensive similarity scores of 5 standard error information are all greater than or equal to 80%, the first three standard error information can be screened out in the order of the comprehensive similarity score from large to small. If only 1-3 standard error information has a comprehensive similarity score greater than or equal to 80%, all the above standard error information can be screened out. If none of the standard error information has a comprehensive similarity score greater than or equal to 80%, the standard error information cannot be screened out.

[0135] S2100, constructing a third prompt text by the software engineer agent according to the screened standard repair strategy and the error information in the abnormal result information.

[0136] As described above, the number of screened standard repair strategies can be 0, 1 or 2, etc., and the maximum is a preset number. Correspondingly, when the software engineer agent needs to repair errors, the fault repair knowledge base can be retrieved through the failure log content to obtain a repair method with a score exceeding a preset score value, provide LLM learning, and if no repair method is obtained, the LLM can be directly repaired by itself.

[0137] S2110, input the abnormal code content in the abnormal result information and the third prompt word text into the pre-constructed large language model through the software engineer agent, regenerate new current test code and store it in the engineering directory, or generate new test cases, or repair the current operator code until the preset end iteration condition is met.

[0138] The end iteration condition can be that the current operator code of the target operator is tested based on the current test case, and the coverage of the current operator code meets the requirements when the current test case is executed successfully, or it can be a preset iteration round, such as 5 rounds or 6 rounds, etc. That is, even after many rounds of iteration, the target operator code or the target test code cannot be obtained, and the round can be directly ended, and the corresponding execution log is provided to the artificial processing platform, and the target operator code or the target test code is generated by the artificial processing platform. Based on the target operator code or the target test code, the large language model and various knowledge bases are updated.

[0139] The abnormal result information generated by the test engineer agent for the current test code specifically includes: the target test code slice located by the test engineer agent, the error information matched with the target test code slice, and the associated test code slice matched with the target test code slice.

[0140] Correspondingly, the abnormal code content in the abnormal result information and the third prompt word text are input into the pre-constructed large language model through the software engineer agent, and new current test code is regenerated and stored in the engineering directory, which can specifically include:

[0141] S21101, input the target test code slice, the associated test code slice matched with the target test code slice, and the third prompt word text in the abnormal result information into the pre-constructed large language model through the software engineer agent, and obtain the new test code slice matched with the target test code slice regenerated by the large language model.

[0142] S21102, store the generated new test code slice in the matching slice storage location in the engineering directory to reorganize the new current test code in the engineering directory.

[0143] In the optional embodiment, by decomposing the complex long code generation task into multiple shorter subtasks, the LLM burden can be reduced, the generation accuracy and efficiency can be improved, and the DSL information sparsity problem can be effectively addressed. In addition, it is helpful for multi-granularity error localization and targeted repair, and a mapping rule library of compiler or executor error information and specific code slices is established (for example, an "undefined identifier" error is usually associated with precode or structure code; a "kernel startup failure" is associated with AI chip end code; and a "comparison failure" is associated with result verification code or test case code). When the test engineer agent detects an error, it accurately locates the error code slice according to the error information. Only the slice and its related context (such as error information or associated slices) are sent to the software engineer agent for targeted repair. Avoid triggering the regeneration of the entire long code due to small errors, and greatly reduce information loss and inefficiency in the repair process.

[0144] S2120, by the code review engineer agent, performing normative inspection on the target operator code and the target test code, and submitting the target operator code and the target test code after passing the normative inspection to a code repository.

[0145] Figure 3 is a structural schematic diagram of a full-process automatic unit test method of an AI chip operator implemented by cooperation between multiple agents.

[0146] Correspondingly, as shown in Figure 3 , first, the operator interface information (Operation) of the target operator to be tested is input into the architect agent (Architect), and the operator interface information is analyzed and processed (Analyze) by the architect agent, so that a test framework design document (Gen Project) can be finally obtained and sent to the software engineer agent (Engineer). The software engineer agent cooperates with the test engineer agent (Tester) to gradually modify the test code error (Fix bug) and generate the adapted test case (Gen Config) by the software engineer agent, so as to realize the generation of the test (Gen Test), and the test engineer agent cooperates to perform code compilation (Compile), test case execution (Execute), and coverage checking (Converage) and other operations, and finally the target test code and the target operator code of the target operator passing the unit test are obtained. Finally, by the code review engineer agent (Reviewer), after completing the normative inspection (Review), the target test code and the target operator code are continuously integrated and submitted to the code repository (in CI).

[0147] Finally, it needs to be emphasized again that the multi-agent cooperation of the embodiments of the present application has the advantages of:

[0148] Knowledge decoupling and focusing: Each agent is equipped with a dedicated knowledge base (such as the DSL operator design document knowledge base for the architect agent and the coverage rule base for the test engineer agent), avoiding attention dilution when a single model handles multiple targets.

[0149] Improved controllability of details: Specialized hint strategies are set for different roles and task stages to precisely control code generation boundaries, post-processing (such as formatting and static checking), and retrieval pre-filtering (such as filtering templates based on interface types).

[0150] Engineering verification chain: Deep integration of compiler, debugger, and coverage tool to achieve "generation -> compilation -> execution -> coverage analysis -> verification" full-process automation, truly achieving "generation verification".

[0151] Feedback-driven iteration: Automated process of error localization triggered by test failure -> code slice repair -> regression testing.

[0152] Dynamic knowledge injection: Real-time information such as compilation results, test results, and coverage reports is injected as context into the LLM to guide it to generate more accurate results.

[0153] Gradual filtering mechanism: Code needs to pass through multiple agent verifications such as architecture constraints, compilation checks, execution verification, coverage compliance, and code specification review to ensure the high quality and reliability of the final repository code.

[0154] Figure 4 The structural diagram of an AI chip operator full-process automatic unit test device provided by the embodiments of the present application is configured in a multi-agent cooperation system, which includes an architect agent, a software engineer agent, a test engineer agent, and a code review engineer agent, as shown in Figure 4 The device includes a design document generation module 410, a test code initialization generation module 420, a multi-agent interaction module 430, and an operator code submission module 440, wherein:

[0155] The design document generation module 410 is used to generate engineering directory and test framework design documents through the architect agent based on operator interface information of the target operator and the domain-specific programming language DSL operator design document knowledge base.

[0156] The test code initialization generation module 420 is used to initialize the test code storage under the engineering directory for unit testing of the operator code of the target operator through the software engineer agent based on the test framework design document and the domain knowledge base.

[0157] The multi-agent interaction module 430 is configured to perform multi-round interaction between the software engineer agent and the test engineer agent, and gradually modify the test code stored in the engineering directory into target test code which can be successfully compiled, the generated test case can be successfully executed, and the generated test case meets the code coverage requirement, and obtain the target operator code of the target operator which passes the unit test;

[0158] The operator code submission module 440 is configured to perform specification checking on the target operator code and the target test code through the code review engineer agent, and submit the target operator code and the target test code which pass the specification checking to the code repository.

[0159] The technical scheme of the embodiment of the present application constructs an agent collaboration system in which multiple role agents collaborate, generates an engineering directory and a test framework design document according to operator interface information of a target operator and a DSL operator design document knowledge base through an architect agent, initializes the generation of test code for unit testing of the operator code of the target operator and stores the test code in the engineering directory according to the test framework design document and a domain knowledge base through a software engineer agent, performs multi-round interaction between the software engineer agent and the test engineer agent, and gradually modifies the test code stored in the engineering directory into target test code which can be successfully compiled, the generated test case can be successfully executed, and the generated test case meets the code coverage requirement, and obtains the target operator code of the target operator which passes the unit test, and performs specification checking on the target operator code and the target test code through a code review engineer agent, and submits the target operator code and the target test code which pass the specification checking to the code repository. The implementation mode can significantly improve the development efficiency of AI chip operator test code, automatically generate unit tests for a large number of AI operators, effectively reduce the writing, maintenance cost and complexity of test code, and stably ensure the quality of generated test code, reduce the influence of human factors, effectively overcome the poor adaptability of general LLM in the AI chip DSL field, and finally realize a high-reliability and high-efficiency AI operator unit test automation pipeline of generate-verify-verify and enter the repository after verification.

[0160] On the basis of the above embodiments, the design document generation module 410 can be specifically configured to:

[0161] The architect agent is configured to search the DSL operator design document knowledge base stored locally according to the operator interface information of the target operator, and obtain at least one item of operator description information matched with the target operator;

[0162] The operator description information includes operator function description information, dimension limit information of operator input and output tensors, and data type information supported by the operator.

[0163] constructing a first prompt word text according to the description information of each of the operators by the architect agent, and inputting the operator interface information of the target operator and the first prompt word text into a pre-constructed large language model to generate an engineering directory and a test framework design document;

[0164] The test framework design document includes operator semantics, file names, function body names, operator lists, programming timing diagrams, and key constraint conditions.

[0165] On the basis of the above embodiments, the test code initialization generation module 420 can specifically include:

[0166] The test code slice generation unit is configured to generate, by the software engineer agent, a plurality of test code slices matched with the target operator based on the test framework design document, in combination with a small sample template library in a locally pre-stored domain knowledge base and an operator semantic knowledge base.

[0167] The test code slice storage unit is configured to store, by the software engineer agent, each test code slice to a matching slice storage location under the engineering directory, so as to organize a complete test code under the engineering directory.

[0168] The small sample template library stores a mapping relationship between the interface data types of each operator and the code templates, and the code templates include structure code, CPU-side code, AI chip-side code, result verification code, and main function code.

[0169] The operator semantic knowledge base stores a mapping relationship between each operator type and operator type description information, and the operator type description information includes operator mathematical definition and properties, input and output data logical relationship of the operator, precision characteristics of the operator, equivalence class partitioning rules of the operator, and boundary rules of the operator.

[0170] On the basis of the above embodiments, the device can further include:

[0171] The operator semantic knowledge base construction module is configured to construct the operator semantic knowledge base by at least one of the following: analyzing and processing each DSL operator design document in the DSL operator design document knowledge base by a pre-constructed large language model, analyzing and processing a header file of each DSL operator by the large language model, and extracting and analyzing a high-level semantic operator in a standard deep learning framework.

[0172] On the basis of the above embodiments, the test code slice generation unit can be specifically configured to:

[0173] The software engineer agent converts the test framework design document into a first comparison dense vector feature based on semantics and a first comparison sparse vector feature based on word segmentation;

[0174] The software engineer agent matches the first comparison dense vector feature and the first comparison sparse vector feature with the standard dense vector features and the standard sparse vector features of the interface data types of each operator in the small sample template library respectively, and selects a first number of target code templates in the small sample template library by combining the category weights of the dense vector and the sparse vector.

[0175] The software engineer agent matches the first comparison dense vector feature and the first comparison sparse vector feature with the standard dense vector features and the standard sparse vector features of each operator type in the operator semantic knowledge base respectively, and selects a second number of target operator type description information in the operator semantic knowledge base by combining the category weights of the dense vector and the sparse vector.

[0176] The software engineer agent constructs a second prompt word text corresponding to each slice type according to the slice type of the test code slice, each target code template, and each target operator type description information.

[0177] The software engineer agent sequentially obtains a current second prompt word text corresponding to a current slice type.

[0178] The software engineer agent inputs the test framework design document and the second prompt word text into the pre-constructed large language model to obtain a current test code slice corresponding to the current slice type.

[0179] The software engineer agent returns to execute the operation of sequentially obtaining the current second prompt word text corresponding to the current slice type until the processing of all slice types is completed to obtain all test code slices.

[0180] The slice type includes precode, structure code, CPU end code, AI chip end code, result verification code, main function code, and test case code.

[0181] Based on the above embodiments, the multi-agent interaction module 430 can be specifically used for:

[0182] After the test engineer agent obtains the current test code from the engineering directory, the test engineer agent calls the pre-integrated chip software stack tool chain to compile the code, and feeds back the compilation success result to the software engineer agent when the code compilation is successful.

[0183] When the software engineer agent receives the successful compilation result, the software engineer agent generates a matched current test case according to the current test code matched with the successful compilation result, and feeds back the current test case to the test engineer agent;

[0184] The test engineer agent calls the chip software stack tool chain, tests the current operator code of the target operator based on the current test case, and determines the current test code as the target test code and the current operator code as the target operator code when the current test case is executed successfully and the coverage of the current operator code meets the requirements.

[0185] When the test engineer agent determines that the code compilation fails, the current test case fails to execute, or the coverage of the current operator code does not meet the requirements, the test engineer agent feeds back the compilation failure result, the current test case execution failure result, or the code coverage not meeting the requirements as abnormal result information to the test engineer agent.

[0186] When the software engineer agent receives the abnormal result information, the software engineer agent converts the abnormal result information into a second comparison dense vector feature based on semantics and a second comparison sparse vector feature based on word segmentation.

[0187] The software engineer agent matches the second comparison dense vector feature and the second comparison sparse vector feature with the standard dense vector feature and the standard sparse vector feature of the standard error information in the fault repair knowledge base in the local pre-stored domain knowledge base, respectively, and performs a screening operation of the standard repair strategy in the fault repair knowledge base according to a pre-set screening rule, combined with the category weight of the dense vector and the sparse vector.

[0188] The software engineer agent constructs a third prompt word text according to the screened standard repair strategy and the error information in the abnormal result information.

[0189] The software engineer agent inputs the abnormal code content in the abnormal result information and the third prompt word text into the pre-constructed large language model, regenerates new current test code stored in the engineering directory, or generates new test cases, or performs code repair on the current operator code until the pre-set end iteration condition is met.

[0190] The fault repair knowledge base stores a mapping relationship between the standard error information and the standard repair strategy, and the chip software stack tool chain includes a compiler, an executor, a coverage detection tool, and a test execution engine.

[0191] On the basis of each of the above embodiments, the test engineer agent specifically includes in the abnormal result information generated for the current test code: the target test code slice where the abnormality occurs located by the test engineer agent, error information matched with the target test code slice, and associated test code slices matched with the target test code slice;

[0192] Correspondingly, the multi-agent interaction module 430 can be further used for:

[0193] Through the software engineer agent, the target test code slice in the abnormal result information, the associated test code slices matched with the target test code slice, and the third prompt word text are jointly input into a pre-constructed large language model, and a new test code slice matched with the target test code slice is obtained by re-generation of the large language model.

[0194] Through the software engineer agent, the generated new test code slice is stored in the matched slice storage location under the engineering directory, so as to re-organize the new current test code under the engineering directory.

[0195] The AI chip operator full-process automatic unit test device provided in the embodiments of the application can execute the AI chip operator full-process automatic unit test method provided in any embodiment of the application, and has the corresponding function modules and beneficial effects of the execution method.

[0196] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information comply with relevant laws and regulations and do not violate public order and good customs.

[0197] Figure 5 A structural schematic diagram of an electronic device 10 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0198] As Figure 5As shown, the electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11, wherein the memory stores a computer program executable by the at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0199] Various components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc., an output unit 17, such as various types of displays, a loudspeaker, etc., a storage unit 18, such as a magnetic disk, an optical disk, etc., and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0200] The processor 11 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as performing the full-flow automated unit testing method of the AI chip operator as described in any embodiment of the present application.

[0201] That is, through the architect agent, the engineering directory and the test framework design document are generated according to the operator interface information of the target operator and the DSL operator design document knowledge base;

[0202] Through the software engineer agent, the test code storage for unit testing the operator code of the target operator is initialized and stored in the engineering directory according to the test framework design document and the domain knowledge base;

[0203] Through the multi-round interaction between the software engineer agent and the test engineer agent, the test code stored in the engineering directory is gradually corrected to the target test code which can be successfully compiled, the generated test case can be successfully executed, and the generated test case meets the code coverage requirement, and the target operator code of the target operator passing the unit test is obtained;

[0204] The target operator code and the target test code are submitted to a code repository after being checked by the code review engineer agent.

[0205] In some embodiments, the full-process automated unit testing method of the AI chip operator as described in any embodiment of the present application can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the full-process automated unit testing method of the AI chip operator as described in any embodiment of the present application described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the full-process automated unit testing method of the AI chip operator as described in any embodiment of the present application by any other appropriate means, for example, by means of firmware.

[0206] The various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0207] Computer programs used to implement the processes of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program, when executed, implements the functions / acts specified in the flowcharts and / or block diagrams. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0208] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0209] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0210] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0211] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0212] It should be understood that the various forms of flow shown above can be used to reorder, add or delete steps. For example, each step described in the present application can be executed in parallel, sequentially or in a different order, as long as the desired results of the technical solutions of the present application can be achieved, which is not limited herein.

[0213] The above detailed description does not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. An AI chip operator full-process automatic unit test method, characterized in that, The method is executed by a multi-agent collaboration system including an architect agent, a software engineer agent, a test engineer agent, and a code review engineer agent, and the method includes: Through the architect agent, an operator interface information of a target operator and a domain-specific programming language DSL operator design document knowledge base are used to generate an engineering directory and a test framework design document; Through the software engineer agent, a test code storage for unit testing of operator code of the target operator is initialized and stored in the engineering directory according to the test framework design document and the domain knowledge base; Through multi-round interaction between the software engineer agent and the test engineer agent, the test code stored in the engineering directory is gradually corrected to target test code that can be successfully compiled, the generated test case can be successfully executed, and the generated test case meets the code coverage requirement, and target operator code of the target operator that passes the unit test is obtained; Through the code review engineer agent, the target operator code and the target test code are checked for specification, and the target operator code and the target test code that pass the specification check are submitted to a code repository.

2. The method of claim 1, wherein, Through the architect agent, an operator interface information of a target operator and a DSL operator design document knowledge base are used to generate an engineering directory and a test framework design document, including: Through the architect agent, the operator interface information of the target operator is searched in the locally stored DSL operator design document knowledge base to obtain at least one operator description information matched with the target operator; The operator description information includes operator function description information, dimension limit information of operator input and output tensors, and data type information supported by the operator; Through the architect agent, a first prompt word text is constructed according to each of the operator description information, and the operator interface information of the target operator and the first prompt word text are input into a pre-constructed large language model to generate the engineering directory and the test framework design document; The test framework design document includes operator semantics, file name, function body name, operator list, programming timing diagram, and key constraint conditions.

3. The method of claim 1, wherein, Through the software engineer agent, a test code storage for unit testing of operator code of the target operator is initialized and stored in the engineering directory according to the test framework design document and the domain knowledge base, including: Through the software engineer agent, a plurality of test code slices matched with the target operator are generated by fragmenting according to the test framework design document, in combination with a small sample template library in a locally pre-stored domain knowledge base and an operator semantic knowledge base; Through the software engineer agent, each test code slice is stored in a matching slice storage location in the engineering directory to organize a complete test code in the engineering directory; The small sample template library stores a mapping relationship between interface data types of each operator and code templates, and the code templates include structure code, CPU-side code, AI chip-side code, result verification code, and main function code; The operator semantic knowledge base stores a mapping relationship between each operator type and operator type description information, and the operator type description information includes: mathematical definition and properties of the operator, logical relationship between input and output data of the operator, precision characteristics of the operator, equivalence class partitioning rules of the operator, and boundary rules of the operator.

4. The method of claim 3, wherein, The method further comprises: The operator semantic knowledge base is constructed by at least one of the following: analyzing and processing each DSL operator design document in the DSL operator design document knowledge base through a pre-constructed large language model, analyzing and processing the header file of each DSL operator through the large language model, and extracting and analyzing high-level semantic operators in a standard deep learning framework.

5. The method according to claim 3 or 4, characterized in that, Through the software engineer agent, a plurality of test code slices matching the target operator are generated in slices according to the test framework design document, in combination with a small sample template library in the local pre-stored domain knowledge base and the operator semantic knowledge base, including: Through the software engineer agent, the test framework design document is converted into semantic-based first comparison dense vector features and first comparison sparse vector features based on word segmentation; Through the software engineer agent, the first comparison dense vector features and the first comparison sparse vector features are matched with standard dense vector features and standard sparse vector features of the interface data types of each operator in the small sample template library, respectively, and in combination with the category weights of the dense vector and the sparse vector, a first number of target code templates are selected in the small sample template library; Through the software engineer agent, the first comparison dense vector features and the first comparison sparse vector features are matched with standard dense vector features and standard sparse vector features of each operator type in the operator semantic knowledge base, respectively, and in combination with the category weights of the dense vector and the sparse vector, a second number of target operator type description information are selected in the operator semantic knowledge base; Through the software engineer agent, according to the slice type of the test code slice, each target code template, and each target operator type description information, a second prompt word text corresponding to each slice type is constructed; Through the software engineer agent, a current second prompt word text corresponding to a current slice type is obtained in sequence; Through the software engineer agent, the test framework design document and the second prompt word text are input into a pre-constructed large language model to obtain a current test code slice corresponding to the current slice type; Through the software engineer agent, the operation of sequentially obtaining the current second prompt word text corresponding to the current slice type is returned, and the processing of all slice types is completed to obtain all test code slices; The slice type includes: precode, structure code, CPU-side code, AI chip-side code, result verification code, main function code, and test case code.

6. The method of claim 5, wherein, Through the multi-round interaction between the software engineer agent and the test engineer agent, the test code stored in the engineering directory is gradually corrected to the target test code which can be successfully compiled, the generated test case can be successfully executed, and the generated test case meets the code coverage requirement, and the target operator code of the target operator which passes the unit test is obtained, including: Through the test engineer agent, after obtaining the current test code from the engineering directory, the pre-integrated chip software stack tool chain is called for code compilation, and the successful compilation result is fed back to the software engineer agent when the code compilation is successful; Through the software engineer agent, when the successful compilation result is received, the matching current test case is generated according to the current test code matching the successful compilation result, and the current test case is fed back to the test engineer agent; Through the test engineer agent, the chip software stack tool chain is called to test the current operator code of the target operator based on the current test case, and when the current test case is successfully executed and the coverage of the current operator code meets the requirement, the current test code is determined as the target test code, and the current operator code is determined as the target operator code; Through the test engineer agent, when the code compilation fails, the current test case fails to execute, or the coverage of the current operator code does not meet the requirement, the compilation failure result, the current test case execution failure result, or the code coverage not meeting the requirement is fed back to the test engineer agent as the abnormal result information; Through the software engineer agent, when the abnormal result information is received, the abnormal result information is converted into the second comparison dense vector feature based on semantics and the second comparison sparse vector feature based on word segmentation; Through the software engineer agent, the second comparison dense vector feature and the second comparison sparse vector feature are matched with the standard dense vector feature and the standard sparse vector feature of the standard error information in the fault repair knowledge base in the local pre-stored domain knowledge base respectively, and the screening operation of the standard repair strategy in the fault repair knowledge base according to the preset screening rule is performed combined with the category weight of the dense vector and the sparse vector; Through the software engineer agent, the third prompt word text is constructed according to the screened standard repair strategy and the error information in the abnormal result information; Through the software engineer agent, the abnormal code content in the abnormal result information and the third prompt word text are input into the pre-constructed large language model to generate new current test code stored in the engineering directory, or generate new test case, or repair the current operator code until the preset end iteration condition is met. Wherein, the fault repair knowledge base stores the mapping relationship between the standard error information and the standard repair strategy, and the chip software stack tool chain includes a compiler, an executor, a coverage detection tool, and a test execution engine.

7. The method of claim 6, wherein, The abnormal result information generated by the test engineer agent for the current test code specifically includes: the target test code slice where the abnormality occurs located by the test engineer agent, error information matched with the target test code slice, and associated test code slices matched with the target test code slice; Correspondingly, the abnormal code content in the abnormal result information and the third prompt word text are input into the pre-constructed large language model by the software engineer agent, and new current test code is regenerated and stored in the engineering directory, specifically including: The target test code slice, the associated test code slice matched with the target test code slice, and the third prompt word text in the abnormal result information are input into the pre-constructed large language model by the software engineer agent, and a new test code slice matched with the target test code slice is obtained by the large language model; The generated new test code slice is stored in the matching slice storage location in the engineering directory by the software engineer agent, so as to reorganize the new current test code in the engineering directory.

8. An AI chip operator full-process automatic unit testing device, characterized in that, The device is configured in a multi-agent collaboration system, and the multi-agent collaboration system includes an architect agent, a software engineer agent, a test engineer agent, and a code review engineer agent, and the device includes: A design document generation module for generating an engineering directory and a test framework design document by the architect agent according to operator interface information of a target operator and a domain-specific programming language DSL operator design document knowledge base; A test code initialization generation module for initializing test code for unit testing of operator code of a target operator stored in an engineering directory by the software engineer agent according to a test framework design document and a domain knowledge base; A multi-agent interaction module for performing multi-round interaction between the software engineer agent and the test engineer agent, gradually correcting the test code stored in the engineering directory into target test code that can be successfully compiled, the generated test case can be successfully executed, and the generated test case meets the code coverage requirement, and obtaining target operator code of the target operator that passes the unit test; An operator code submission module for performing specification checking on the target operator code and the target test code by the code review engineer agent, and submitting the target operator code and the target test code that pass the specification checking to a code repository.

9. An electronic device, comprising: The electronic device includes: At least one processor; and The memory is in communication connection with the at least one processor; wherein The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the full-process automated unit testing method of the AI chip operator in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for enabling the processor to implement the full-process automated unit testing method of the AI chip operator in any one of claims 1-7 when executed.

11. A computer program product, characterised in that, The computer program product comprises a computer program which, when executed by a processor, implements the full-process automatic unit test method of the AI chip operator according to any one of claims 1-7.

Images