Test method and device, electronic equipment and storage medium

Abstract

The invention provides a testing method and device, a storage medium, electronic equipment and a program product, and relates to the field of chip research and development, in particular to the technical field of chip testing and operator updating. According to the specific implementation scheme, based on chip parameters of a to-be-researched and developed chip, an operation environment used for operating a first operator is configured on a simulator used for simulating the to-be-researched and developed chip, and a first simulation chip is obtained; based on the hardware parameters of the reference hardware, configuring a running environment for running a second operator on the reference hardware to obtain a reference chip; determining a first test result and a reference result; and based on the first test result and the reference result, performing test evaluation on the to-be-researched and developed chip, and determining a test evaluation result of the to-be-researched and developed chip.

Description

Technical Field

[0001] This disclosure relates to the field of chip research and development technology, and in particular to the fields of chip testing and operator updating. Specifically, it relates to testing methods, apparatus, electronic devices, storage media, and software products. Background Technology

[0002] Chip testing is a critical step in ensuring that semiconductor products meet performance, functional, and quality standards. During chip design and manufacturing, testing not only confirms whether the product meets design specifications but also enables the timely detection and correction of defects, thereby reducing rework costs and enhancing the product's market competitiveness. As chip complexity increases, testing becomes increasingly complex, and the requirements for testing also rise accordingly. Summary of the Invention

[0003] This disclosure provides a test method, apparatus, electronic device, storage medium, and program product.

[0004] According to one aspect of this disclosure, a testing method is provided, comprising: configuring a runtime environment for running a first operator on a simulator for simulating the chip under development based on chip parameters of the chip under development, thereby obtaining a first simulated chip; configuring a runtime environment for running a second operator on the reference hardware based on hardware parameters of the reference hardware, thereby obtaining a reference chip, wherein the execution logic of the first operator and the second operator is the same, the compilation method of the first operator matches the simulator, and the compilation method of the second operator matches the reference hardware; determining a first test result and a reference result, wherein the first test result is obtained by calling the first operator to process test data on the first simulated chip, and the reference result is obtained by calling the second operator to process the test data on the reference chip; and performing a test evaluation on the chip under development based on the first test result and the reference result, thereby determining a test evaluation result for the chip under development.

[0005] According to another aspect of this disclosure, a first configuration module is provided for configuring a runtime environment for running a first operator on a simulator for simulating the chip under development based on chip parameters of the chip under development, thereby obtaining a first simulated chip; a second configuration module is provided for configuring a runtime environment for running a second operator on the reference hardware based on hardware parameters of the reference hardware, thereby obtaining a reference chip, wherein the execution logic of the first operator and the second operator is the same, the compilation method of the first operator matches the simulator, and the compilation method of the second operator matches the reference hardware; a result determination module is provided for determining a first test result and a reference result, wherein the first test result is obtained by calling the first operator to process test data on the first simulated chip, and the reference result is obtained by calling the second operator to process the test data on the reference chip; and a test evaluation module is provided for performing a test evaluation on the chip under development based on the first test result and the reference result, thereby determining the test evaluation result of the chip under development.

[0006] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0007] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the method described above.

[0008] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method described above.

[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0011] Figure 1A This illustration schematically shows an exemplary system architecture to which testing methods and apparatus can be applied according to embodiments of the present disclosure;

[0012] Figure 1B This schematically illustrates a test platform architecture according to an embodiment of the present disclosure;

[0013] Figure 2 A flowchart illustrating a test method according to an embodiment of the present disclosure is shown schematically;

[0014] Figure 3 This illustration schematically depicts a test diagram for determining operator adaptability according to an embodiment of the present disclosure;

[0015] Figure 4A A schematic diagram illustrating the determination of a first analog chip according to an embodiment of the present disclosure is shown;

[0016] Figure 4B A schematic diagram illustrating a configuration assistance script according to an embodiment of the present disclosure is shown.

[0017] Figure 4C A schematic diagram illustrating configuration test data according to an embodiment of this disclosure is shown.

[0018] Figure 5 A schematic diagram illustrating a test of a hardware simulation chip according to an embodiment of the present disclosure is shown.

[0019] Figure 6 A block diagram of a test apparatus according to an embodiment of the present disclosure is schematically shown; and

[0020] Figure 7 A block diagram of an electronic device suitable for implementing a test method according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0021] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0022] When a chip is in the research and development stage, deep learning models can be used to predict the performance of the chip under development, using chip parameters as a reference, thus obtaining chip performance evaluation results. Simulators can also be used to run operators to simulate the chip under development and obtain test results. However, both test results and chip performance evaluation results are compared with manually defined targets to determine whether the chip under development needs optimization.

[0023] Using artificially set goals as a reference is subjective, and the evaluation dimensions are coarse-grained, making it difficult to make comprehensive comparisons from multiple dimensions to refine and optimize the granularity.

[0024] This disclosure provides a testing method, comprising: configuring a runtime environment for running a first operator on a simulator for simulating the chip under development based on chip parameters of the chip under development, thereby obtaining a first simulated chip; configuring a runtime environment for running a second operator on reference hardware based on hardware parameters of reference hardware, thereby obtaining a reference chip, wherein the execution logic of the first operator and the second operator is the same, the compilation method of the first operator matches the simulator, and the compilation method of the second operator matches the reference hardware; determining a first test result and a reference result, wherein the first test result is obtained by calling the first operator to process test data on the first simulated chip, and the reference result is obtained by calling the second operator to process test data on the reference chip; and performing a test evaluation on the chip under development based on the first test result and the reference result, thereby determining the test evaluation result of the chip under development.

[0025] Using the testing method provided in this embodiment, a first operator can be run on a first simulated chip obtained from a simulator to simulate the chip under development. While obtaining the test results, the reference results obtained by executing a second operator on a reference chip are used as reference data to test and evaluate the chip under development. This makes the test evaluation results not subjective judgments, but comparisons made with the performance results of developed chips as a reference, thereby improving R&D efficiency, reducing subjective judgments, and improving the objectivity, effectiveness, and fine granularity of chip optimization.

[0026] Figure 1A The illustration schematically depicts an exemplary system architecture to which testing methods and apparatus can be applied according to embodiments of the present disclosure.

[0027] It is important to note that Figure 1A The examples shown are merely illustrative of system architectures applicable to embodiments of this disclosure, intended to help those skilled in the art understand the technical content of this disclosure. They do not imply that embodiments of this disclosure cannot be used in other devices, systems, environments, or scenarios. For instance, in another embodiment, an exemplary system architecture to which testing methods and apparatus can be applied may include a terminal device. However, the terminal device can implement the testing methods and apparatus provided in embodiments of this disclosure without interacting with a server.

[0028] like Figure 1A As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, and 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the terminal devices 101, 102, and 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0029] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (for example only).

[0030] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0031] Server 105 can be a server that provides various services, such as a backend management server that supports the content browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0032] It should be noted that the testing methods provided in the embodiments of this disclosure can generally be executed by terminal devices 101, 102, or 103. Accordingly, the testing apparatus provided in the embodiments of this disclosure can also be disposed in terminal devices 101, 102, or 103.

[0033] Alternatively, the testing methods provided in this disclosure can generally be executed by server 105. Correspondingly, the testing apparatus provided in this disclosure can generally be located in server 105. The testing methods provided in this disclosure can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the testing apparatus provided in this disclosure can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105.

[0034] For example, a user sends a test command to server 105 via terminal devices 101, 102, and 103. Server 105 responds to the test command by determining the chip parameters of the chip to be developed, configuring a runtime environment for running a first operator on a simulator used to simulate the chip, thus obtaining a first simulated chip; configuring a runtime environment for running a second operator on the reference hardware based on the hardware parameters of the reference hardware, thus obtaining a reference chip; determining a first test result and a reference result; and, based on the first test result and the reference result, performing a test evaluation on the chip to be developed, determining the test evaluation result of the chip, and sending the test evaluation result to terminal devices 101, 102, and 103. Alternatively, a server or server cluster capable of communicating with terminal devices 101, 102, and 103 and / or server 105 responds to the test command to perform a test evaluation on the new product to be developed, thus obtaining a test evaluation result.

[0035] It should be understood that Figure 1A The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0036] Figure 1B The test platform architecture according to an embodiment of this disclosure is illustrated schematically.

[0037] like Figure 1B As shown, a test platform can be configured on the server to perform tests on different types and functions of chips.

[0038] like Figure 1B As shown, the test platform can include four test platform layers. The first test platform layer can be configured with Artificial Intelligence (AI) hardware, such as the first hardware, ..., the Nth hardware, and environment configuration tools, such as the first environment configuration tool, ..., the Nth environment configuration tool; simulation drivers; and the first operator compiler, ..., the Nth operator compiler. The first test platform layer is mainly responsible for managing the hardware, the environment configuration tools for different hardware, and the environment variable parameters of the environment configuration tools. However, it is not limited to this. It can also be used to manage simulators and simulation environment configuration tools, such as simulation drivers.

[0039] The second test platform layer stores different types of operators, such as the first operator, ..., the Nth operator. This layer is primarily responsible for managing operators, different optimized versions of the same operator, and the test data used for testing.

[0040] The third test platform layer stores functional parameters such as operator call parameters, script test parameters, and test data parameters, which are used to configure auxiliary scripts for functionality.

[0041] The fourth test platform layer is configured with auxiliary scripts such as instruction recording scripts, log recording scripts, and parsing scripts. These auxiliary scripts are used to record the test status during the test process. For example, the instruction recording script can be used to obtain the instruction log, the log recording script can be used to obtain the status log, and then the parsing script can be used to determine the test result based on one or more of the instruction log and the status log.

[0042] By configuring the testing platform, it achieves "one-time maintenance, multiple uses," meaning that once the testing platform is built, subsequent testing and performance analysis can be performed with a single click. This improves testing efficiency and expands the testing scope.

[0043] It should be noted that the sequence numbers of the operations in the following methods are for descriptive purposes only and should not be considered as indicating the execution order of the operations. Unless explicitly stated otherwise, the method does not need to be executed in the exact order shown.

[0044] Figure 2 A flowchart illustrating a test method according to an embodiment of the present disclosure is shown schematically.

[0045] like Figure 2 As shown, the method includes operations S210~S240.

[0046] In operation S210, based on the chip parameters of the chip to be developed, a runtime environment for running the first operator is configured on the simulator used to simulate the chip to be developed, thereby obtaining the first simulated chip.

[0047] In operation S220, based on the hardware parameters of the reference hardware, the runtime environment for running the second operator is configured on the reference hardware to obtain the reference chip.

[0048] In operation S230, determine the first test result and the reference result.

[0049] In operation S240, based on the first test result and the reference result, the chip to be developed is tested and evaluated to determine the test evaluation result of the chip to be developed.

[0050] During the chip development process, the functions and performance of newly developed operators or operators already in use are compared between reference hardware and pre-research chips. This helps to improve the chip under development and design more competitive products.

[0051] Reference hardware may include, but is not limited to, a central processing unit (CPU), a graphics processing unit (GPU), a neural network processing unit (NPU), a data processing unit (Data Processing Unit), an inter-processor unit (IPU), and a tensor processing unit (TPU).

[0052] The chip parameters of the chip under development can include configuration parameters describing the chip. These include hardware parameters and functional parameters. For example, chip parameters can include chip type, on-chip memory type, on-chip memory access configuration type, and chip bandwidth. Chip type can include, but is not limited to, Graphics Processing Unit (GPU), Field Programmable Gate Array (FPGA), and Application Specific Integrated Circuit (ASIC). On-chip memory type can include read-only memory and random access memory. On-chip memory access configuration type can include allowed memory access and disabled memory access.

[0053] Simulators, also known as digital simulators, are used in electronic design automation (EDA) systems.

[0054] It is an important tool in the field of design automation (EDA). It is used to simulate the chip under development for testing and obtain test results that characterize the function and performance of the chip under development.

[0055] Compared to fabricating physical hardware based on chip parameters for simulation testing, simulating the chip under development using software based on chip parameters during the research and design phase can reduce testing costs and improve research and development efficiency.

[0056] In the field of computer science, the components used to perform computational tasks are called operators. For example, computational tasks can be performed through functions, and each function can be called an operator. The execution logic of the first and second operators is the same. The compilation method of the first operator is matched to the simulator, while the compilation method of the second operator is matched to the reference hardware.

[0057] The execution logic of the first operator and the second operator is the same, which can be understood as the first operator and the second operator having the same computational tasks, such as representing the same functions.

[0058] Compilation refers to converting a source program written in a source language into an object program using a compiler. Different compilation methods refer to different compilers used. This mainly depends on the type of source language used in the operator's source program. When the configuration parameters of the simulator and reference hardware are different, the source language types of the operator's source program will differ. Therefore, the compilation method for the first operator can be set to match the simulator, and the compilation method for the second operator can be set to match the reference hardware, thereby ensuring the successful execution of the operator.

[0059] For example, a first operator using a first source language type runs on a first analog chip. A second operator using a second source language type runs on a second analog chip. The computational task performed by both the first and second operators is the function M=A+B.

[0060] Different operator execution platforms may require different runtime environments. For example, the simulator uses a first environment configuration tool to configure its runtime environment. Using the GPU as the reference hardware, a second environment configuration tool is used to configure the runtime environment. Adaptive runtime environments can be configured separately on the simulator and the reference hardware, thereby ensuring the smooth execution of the operators. The first test result is obtained by calling the first operator to process the test data on the first simulation chip, and the reference result is obtained by calling the second operator to process the test data on the reference chip.

[0061] The first operator can be run on the first simulated chip obtained by the simulator to simulate the chip under development. At the same time, the reference result obtained by executing the second operator on the reference chip can be used as reference data to test and evaluate the chip under development. This makes the test evaluation result not a subjective judgment, but a comparison with the performance result of the developed chip as a reference. This improves the R&D efficiency, reduces subjective judgment, and improves the objectivity, effectiveness and fine granularity of chip optimization.

[0062] According to embodiments of this disclosure, when performing such Figure 2 Following operation S240, the testing method may further include: if the test evaluation results indicate that the performance of the chip under development is superior to that of the reference chip, configuring a runtime environment for running the third operator on a simulator to obtain a second simulated chip. Calling the third operator on the second simulated chip to process test data to determine a second test result. Based on the second test result and the first test result, evaluating the operator adaptation performance of the chip under development to obtain operator test results characterizing operator adaptability.

[0063] The third operator has the same function as the first operator but different execution logic; it can be understood as different versions of the first and third operators. For example, the first and third operators perform the same computational task, but the computation methods differ. For instance, the functions represented by the first operator include... The functions represented by the third operator include The first and third operators produce the same result, but the instructions invoked and the processing logic differ.

[0064] Operators with the same function but different versions can be run on the same operator running platform. By comparing the first test result and the second test result, the operator test result that characterizes the operator's adaptability can be determined.

[0065] For example, if the first test result of the first operator and the second test result of the third operator have the same functional result value, it indicates that both the first and third operators can achieve the same computational function. However, if the first test result of the first operator and the second test result of the third operator have different functional results, such as processing time, it is determined that the operator with shorter processing time has higher compatibility with the chip under development.

[0066] Therefore, by running operators with the same function but different execution logic on the same operator runtime platform, multiple operators can be tested, thereby determining the operator test results that characterize the operator adaptability of the chip under development. This improves the richness and granularity of the testing, further enhancing the pertinence and effectiveness of determining the optimization direction of the chip under development.

[0067] Figure 3 The illustration shows a schematic diagram of a test to determine operator adaptability according to an embodiment of the present disclosure.

[0068] like Figure 3 As shown, a first simulation chip is obtained by configuring a runtime environment on the simulator to run the first operator. A reference chip is obtained by configuring a runtime environment on the reference hardware to run the second operator. A first test result is determined by calling the first operator to process test data on the first simulation chip. A reference result is determined by calling the second operator to process test data on the reference chip.

[0069] like Figure 3 As shown, when the test evaluation results indicate that the performance of the chip under development is superior to that of the reference chip, a runtime environment for running the third operator is configured on the simulator to obtain a second simulated chip. Based on the second test results and the first test results, the operator adaptation performance of the chip under development is evaluated, and operator test results characterizing the operator adaptation are obtained.

[0070] Optionally, if the test evaluation results indicate that the performance of the chip under development is lower than that of the reference information, the chip parameters of the chip under development may be optimized and adjusted.

[0071] The performance relationship between the chip under development and existing chips is compared by comparing a reference chip and a first simulated chip. The compatibility between operators with different execution logics and the chip under development is compared by running them on a simulator. Therefore, the chip under development is tested from different dimensions according to different stages of the testing process, achieving multi-indicator, fine-grained, and highly efficient testing and evaluation, thereby improving the efficiency and effectiveness of chip optimization.

[0072] Optionally, a runtime environment for running the fourth operator can be configured on the second simulator to obtain a third simulated chip. The fourth operator is then called on the third simulated chip to process test data, yielding a third test result. Based on the third test result and the first test result, the operator adaptation performance of the chip under development is evaluated, obtaining operator test results characterizing operator adaptability. The first and fourth operators can have the same execution logic but different compilation methods. The simulator running the first operator can be used as the first simulator. The first simulator and the second simulator can use different chip types for the chip under development, or any chip parameters of the chip under development can be different. This improves the ability and effectiveness of evaluating the impact of the chip parameters of the chip under development on optimization indicators.

[0073] The testing method provided by the embodiments of this disclosure can cope with multi-dimensional comparative testing tasks with different operating environments, operator versions, and reference hardware. While ensuring that the operator can be executed smoothly, it also improves the objectivity and comprehensiveness of the comparison, thereby improving chip development efficiency.

[0074] The above section provided an overall overview of the testing process for the chip under development. The following section will explain how to configure the operating environment and obtain the test results.

[0075] According to embodiments of this disclosure, for example, Figure 2 The illustrated operation S210, based on the chip parameters of the chip under development, configures a runtime environment for running the first operator on a simulator used to simulate the chip under development, thereby obtaining a first simulated chip. This may include: configuring the runtime environment on the simulator using a first environment configuration tool that matches the simulator parameters; and compiling the first operator using an operator compiler that matches the simulator parameters to obtain the first simulated chip.

[0076] Simulator parameters are determined based on chip parameters. For example, chip parameters can be used as simulator parameters, but this is not a limitation; driver parameters of the simulation driver used to drive the simulator can also be included. Any information that can describe the simulator is acceptable.

[0077] During the research and development process, it was discovered that different simulator parameters require different environment configuration tools. Furthermore, different source language types for different operators require different operator compilers.

[0078] Configure the runtime environment on the emulator using a first-environment configuration tool that matches the emulator parameters. This first-environment configuration tool can include containers (Docker), which can be a set of platform-as-a-service products. Based on operating system-level virtualization technology, software and its dependencies are packaged into containers; the software hosting these containers can be called the Docker engine.

[0079] Multiple environment configuration tools can be pre-set, and the first environment configuration tool can be determined from among the multiple environment configuration tools based on the simulator parameters.

[0080] Multiple operator compilers can be pre-configured. Based on simulator parameters or the compilation method of the first operator, the first operator compiler is selected from among the multiple compilers. The first operator compiler is then used to compile the first operator to obtain the first simulation chip.

[0081] By configuring the runtime environment from multiple aspects, including operator and container dimensions, the stability and reliability of the testing environment are improved, providing a foundation for testing.

[0082] According to embodiments of this disclosure, for example, Figure 2 The illustrated operation S220, based on the hardware parameters of the reference hardware, configures a runtime environment for running the second operator on the reference hardware to obtain a reference chip. This may include: configuring the runtime environment on the reference hardware using a second environment configuration tool matched to the hardware parameters; and compiling the second operator using an operator compiler matched to the hardware parameters to obtain the reference chip.

[0083] Different hardware-based operator execution platforms require different environment configuration tools and operator compilers. A first mapping relationship between hardware parameters and the environment configuration tool, and a second mapping relationship between hardware parameters and the operator compiler can be established in advance. Based on the hardware parameters of the reference hardware indicated in the test task, the first mapping relationship, and the second mapping relationship, a second environment configuration tool and operator compiler are determined, and a reference chip is obtained using the second environment configuration tool and operator compiler.

[0084] The method for configuring the simulator's runtime environment is similar to that for configuring the reference hardware's runtime environment, and will not be repeated here.

[0085] By configuring the runtime environment from multiple aspects, such as operator and container dimensions, the stability and reliability of the testing environment are improved, providing a foundation for testing.

[0086] According to embodiments of this disclosure, when performing such Figure 2 Before operation S210 or S220, the test method may further include: verifying the locally stored environment configuration tool and obtaining a verification result. If the verification result indicates that the environment configuration tool is not the latest version, update the environment configuration tool.

[0087] The environment configuration tool may include a first environment configuration tool for configuring the runtime environment of a first operator or a second environment configuration tool for configuring the runtime environment of a second operator.

[0088] Environment configuration tools can be stored locally, such as in a pre-allocated storage space, for quick response upon invocation. However, with the continuous development of artificial intelligence, the development speed of environment configuration tools is constantly increasing. Before invoking the first or second environment configuration tool, it can be verified whether it is the latest version. If the verification result indicates it is the latest version, it can be invoked directly. If the verification result indicates it is not the latest version, the environment configuration tool should be updated. Alternatively, an invocation command can be sent to a third party, such as the environment configuration tool manufacturer, to obtain the latest version of the environment configuration tool.

[0089] Optionally, the version identification information of the environment configuration tool can be marked when storing the local storage environment configuration tool. Based on the version identification information and the latest version identification information of the environment configuration tool released by the third party, a verification result is determined to determine whether it is the latest version.

[0090] The testing method provided by this disclosure can improve test stability and reliability by adapting the environment configuration tool to the simulator or reference hardware, and further ensure test stability and reliability by verifying whether the environment configuration tool has been updated.

[0091] According to optional embodiments of this disclosure, when performing such Figure 2 Before operation S210 or S220, the test method may further include: verifying the operator compiler stored locally and obtaining a verification result. If the verification result indicates that the operator compiler is not the latest version, the operator compiler is updated.

[0092] There are no restrictions on the verification and update methods of the operator compiler. The verification and update methods are similar to those of the environment configuration tool.

[0093] Therefore, by adapting the environment configuration tool to the simulator or reference hardware to improve test stability and reliability, the stability and reliability of the test can be further guaranteed by verifying whether the versions of the environment configuration tool and the operator compiler are updated.

[0094] Figure 4A A schematic diagram illustrating the determination of a first analog chip according to an embodiment of the present disclosure is shown.

[0095] like Figure 4A As shown, based on chip parameters, the simulator, the first environment configuration tool, and the first operator compiler are determined from the first test platform layer of the test platform. The first operator is determined from the second test platform layer of the test platform.

[0096] like Figure 4A As shown, the environment configuration tool and operator compiler are verified to be the latest versions. If the verification result indicates that they are the latest versions, the first simulation chip is obtained based on the simulator, the first environment configuration tool, the first operator compiler, and the first operator.

[0097] like Figure 4A The method for determining the first analog chip shown is similar to that for determining the reference chip. This is only used as an example, and the method for determining the reference chip will not be described in detail.

[0098] According to embodiments of this disclosure, for example, Figure 2 The operation S230 shown, determining the first test result, may include: acquiring status information; parsing the status information to obtain field information representing the running status; and determining the first test result based on the field information.

[0099] The status information can be obtained by recording the status of the first analog chip running the first operator.

[0100] A logging script can be used to record the running status, resulting in a status log that includes status information. The content of the status information is not limited and can include data processing results, data processing time, number of operator executions, etc.

[0101] Log scripts with varying levels of information complexity can be configured. Based on the metrics to be tested, a target log script is selected from multiple scripts to record different types or complexities of status information.

[0102] For example, logging script 1 is used to record the data processing time, and logging script 2 is used to record the data processing time, data processing results, and hardware power consumption.

[0103] You can set up a parsing script to parse the status information, such as field recognition or keyword matching, to obtain field information that represents the running status.

[0104] Determining the first test result based on field information can include directly using the field information as the first test result. However, it is not limited to this. It can also include rearranging multiple field information according to a predetermined order to obtain the first test result.

[0105] The method for determining the first test result is similar to that for determining the reference result. This will only be used as an example, and the method for determining the reference result will not be elaborated upon.

[0106] Figure 4B A schematic diagram of a configuration assistance script according to an embodiment of the present disclosure is shown.

[0107] like Figure 4B As shown, auxiliary scripts can be downloaded from the fourth test platform layer based on the metrics to be tested. For example, a logging script can be downloaded to record the running status. However, this is not the only option. Parsing scripts for data processing to obtain the first test results can also be downloaded.

[0108] According to embodiments of this disclosure, an auxiliary script is used to record the running status and obtain status information. A parsing script is then used to parse the status information to obtain a first test result. This achieves fully automated test analysis, refining the granularity of analysis while improving analysis efficiency.

[0109] According to embodiments of this disclosure, when performing such Figure 2 Prior to operation S230, the test method may further include: determining the data parameters and operator call parameters of the test data based on the indicator to be tested; generating test data based on the data parameters so that the first operator or the second operator can be run to process the test data using the operator call parameters.

[0110] Operator call parameters characterize how the first or second operator is invoked. For example, the number of times the first or second operator is executed.

[0111] Data parameters can refer to the data type, data scale, such as data dimensions, and other information used to describe the data.

[0112] For example, the size of the matrix of test data in matrix form can be determined based on the data dimension. The value of each element in the matrix can be determined based on the initialization type. Furthermore, it can be determined whether the test data is a floating-point number or an integer based on the data type.

[0113] The metrics to be tested can be functional or performance metrics. A mapping relationship can be established between the metrics and data parameters, and between the metrics and operator call parameters. Based on these mapping relationships and the metrics to be tested, the data parameters and operator call parameters for the test data are determined.

[0114] Figure 4C A schematic diagram illustrating configuration test data according to an embodiment of the present disclosure is shown.

[0115] like Figure 4CAs shown, data parameters and operator call parameters can be called from the third test platform layer according to the metrics to be tested, so as to generate test data through the data parameters, and then use the operator call parameters to run the first or second operator to process the test data.

[0116] Using the metrics to be tested to determine the test data and operator call parameters can ensure that the test is executed as expected, thereby improving the effectiveness and accuracy of the test evaluation results corresponding to the metrics to be tested.

[0117] The testing process has been explained above; the following section will explain how to conduct the test evaluation.

[0118] According to embodiments of this disclosure, when performing such Figure 2 The illustrated operation S240, based on the first test result and the reference result, performs a test evaluation on the chip under development, and determines the test evaluation result of the chip under development, including: determining the first test indicator result and the reference indicator result that match the test indicator from the first test result and the reference result, respectively; obtaining the indicator evaluation result based on the first test indicator result and the reference indicator result; and obtaining the test evaluation result based on the indicator evaluation results of each of the multiple test indicators.

[0119] The metrics to be tested can include functional metrics or performance metrics. Functional metrics can include a first functional metric indicating whether the operator can function normally, and a second functional metric indicating whether the operator can obtain correct results. Performance metrics can include duration metrics, power consumption metrics, storage space capacity metrics, etc.

[0120] For different types of indicators to be tested, indicator results can be determined, such as the result of a first test indicator and a reference indicator result. The indicator evaluation result can include scores representing the degree of difference between the first test indicator result and the reference indicator result, for example, by determining the weights used to represent the degree of difference between the first test indicator result and the reference indicator result according to mapping rules. However, it is not limited to this. The indicator evaluation result can also be a quantified value obtained by normalizing the difference between the first test indicator result and the reference indicator result. Any indicator evaluation result that can be combined using a unified dimension is acceptable.

[0121] The test evaluation results are obtained by weighted summation of the evaluation results of multiple indicators.

[0122] By utilizing the method for determining test evaluation results provided in this embodiment, each test indicator can be analyzed, thereby aligning the analysis results according to the indicator dimensions, improving the scientific rigor of the analysis, and refining the granularity of the analysis.

[0123] The testing and evaluation process has been explained above. The following section will describe the subsequent procedures for chips under development that meet the testing standards.

[0124] According to embodiments of this disclosure, for example, Figure 2 After the operation S240 shown, the test method may further include: recording the instructions executed by running the first operator to obtain the instruction set.

[0125] Instruction recording scripts can be used to record the executed instructions during the first operator's operation, thus obtaining an instruction set. This instruction set, obtained through software simulation testing, can then be used for hardware simulation testing, thereby improving the utilization rate of software simulation testing.

[0126] According to embodiments of this disclosure, after executing the operation to obtain the instruction set, the testing method may further include: if the operator test results indicate that the performance of the chip under development is better than that of the reference chip, writing the instruction set onto the circuit board to obtain a hardware simulation chip. Testing the hardware simulation chip to obtain the simulation test results of the chip under development.

[0127] The instruction set is used to characterize the process of the first operator processing the test data.

[0128] Figure 5 A schematic diagram illustrating a test of a hardware emulation chip according to an embodiment of the present disclosure is shown.

[0129] like Figure 5 The procedure for determining the operator test results shown is as follows: Figure 3 The procedure for determining the operator test results is the same as shown. The difference lies in that, when the operator test results indicate that the operator adaptability of the third operator is higher than that of the first operator, the circuit board can be determined using chip parameters. The instruction set of the third operator is written onto the circuit board to obtain a hardware simulation chip. The hardware simulation chip is then tested to determine the adaptability between the third operator and the hardware simulation chip, thus obtaining the simulation test results.

[0130] The testing method provided by this disclosure can perform testing from multiple dimensions, including software simulation testing and hardware simulation testing, with software simulation testing preceding hardware simulation testing. This improves chip optimization efficiency, reduces chip R&D costs, and enhances the comprehensiveness and effectiveness of testing.

[0131] Figure 6 A block diagram of a test apparatus according to an embodiment of the present disclosure is shown schematically.

[0132] like Figure 6 As shown, the testing device 600 may include a first configuration module 610, a second configuration module 620, a result determination module 630, and a test evaluation module 640.

[0133] The first configuration module 610 is used to configure the runtime environment for running the first operator on a simulator used to simulate the chip under development based on the chip parameters of the chip under development, so as to obtain the first simulated chip.

[0134] The second configuration module 620 is used to configure a runtime environment for running the second operator on the reference hardware based on the hardware parameters of the reference hardware, thereby obtaining a reference chip. The execution logic of the first operator and the second operator is the same. The compilation method of the first operator is matched with the simulator, and the compilation method of the second operator is matched with the reference hardware.

[0135] The result determination module 630 is used to determine a first test result and a reference result, wherein the first test result is obtained by calling a first operator to process test data on a first simulation chip, and the reference result is obtained by calling a second operator to process test data on a reference chip.

[0136] The test evaluation module 640 is used to perform test evaluation on the chip under development based on the first test result and the reference result, and to determine the test evaluation result of the chip under development.

[0137] According to embodiments of this disclosure, the testing apparatus further includes: a third configuration module, a testing module, and an adaptation evaluation module.

[0138] The third configuration module is used to configure the runtime environment for running the third operator on the simulator when the test evaluation results indicate that the performance of the chip under development is better than that of the reference chip, so as to obtain the second simulated chip. The third operator has the same function as the first operator but different execution logic.

[0139] The test module is used to call the third operator on the second simulation chip to process the test data and determine the second test result.

[0140] The adaptation evaluation module is used to evaluate the operator adaptation performance of the chip under development based on the second test results and the first test results, and obtain the operator test results characterizing the operator adaptation.

[0141] According to embodiments of this disclosure, the testing apparatus further includes a writing module and a simulation module.

[0142] The writing module is used to write the instruction set onto the circuit board to obtain a hardware simulation chip when the operator test results indicate that the performance of the chip under development is better than that of the reference chip. The instruction set is used to characterize the running process of the first operator processing test data.

[0143] The simulation module is used to test the hardware simulation chip and obtain the simulation test results of the chip to be developed.

[0144] According to embodiments of this disclosure, the testing apparatus further includes an instruction recording module.

[0145] The instruction recording module is used to record the instructions executed by the first operator to obtain the instruction set.

[0146] According to embodiments of this disclosure, the result determination module includes: a status acquisition submodule, a parsing submodule, and a determination submodule.

[0147] The status acquisition submodule is used to acquire status information, which is obtained by recording the status of the first analog chip running the first operator.

[0148] The parsing submodule is used to parse the status information to obtain field information that represents the running status.

[0149] The determination submodule is used to determine the first test result based on the field information.

[0150] According to embodiments of this disclosure, the first configuration module includes a first configuration submodule and a first compilation submodule.

[0151] The first configuration submodule is used to configure the runtime environment on the simulator using a first environment configuration tool that matches the simulator parameters, wherein the simulator parameters are determined based on the chip parameters.

[0152] The first compilation submodule is used to compile the first operator using an operator compiler that matches the simulator parameters, thereby obtaining the first simulation chip.

[0153] According to embodiments of this disclosure, the second configuration module includes: a second configuration submodule and a second compilation submodule.

[0154] The second configuration submodule is used to configure the runtime environment on the reference hardware using a second environment configuration tool that matches the hardware parameters.

[0155] The second compilation submodule is used to compile the second operator using an operator compiler that matches the hardware parameters, to obtain a reference chip.

[0156] According to embodiments of this disclosure, the testing apparatus further includes a version verification module and a tool update module.

[0157] The version verification module is used to verify the environment configuration tools stored locally and obtain the verification results. The environment configuration tools include a first environment configuration tool for configuring the runtime environment of the first operator or a second environment configuration tool for configuring the runtime environment of the second operator.

[0158] The tool update module is used to update the environment configuration tool when the verification results indicate that the environment configuration tool is not the latest version.

[0159] According to embodiments of this disclosure, the testing apparatus further includes a parameter determination module and a data generation module.

[0160] The parameter determination module is used to determine the data parameters and operator call parameters of the test data based on the indicators to be tested. The operator call parameters represent the calling method of the first or second operator.

[0161] The data generation module is used to generate test data based on data parameters, so that the first or second operator can be run using the operator call parameters to process the test data.

[0162] According to embodiments of this disclosure, the test evaluation module includes: a first evaluation submodule, a second evaluation submodule, and a third evaluation submodule.

[0163] The first evaluation submodule is used to determine the first test indicator result and the reference indicator result that match the indicator to be tested from the first test result and the reference result, respectively.

[0164] The second evaluation submodule is used to obtain the indicator evaluation results based on the results of the first test indicator and the reference indicator results.

[0165] The third evaluation submodule is used to obtain the test evaluation result based on the evaluation results of each of the multiple indicators to be tested.

[0166] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of user personal information comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and there is no violation of public order and good morals.

[0167] In the technical solution disclosed herein, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0168] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0169] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0170] According to embodiments of the present disclosure, a non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions are used to cause a computer to perform the method described above.

[0171] According to an embodiment of this disclosure, a computer program product includes a computer program that, when executed by a processor, implements the method described above.

[0172] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0173] like Figure 7 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded into random access memory (RAM) 703 from storage unit 708. The RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0174] Multiple components in device 700 are connected to input / output (I / O) interface 705, including: input unit 706, such as a keyboard, mouse, etc.; output unit 707, such as various types of displays, speakers, etc.; storage unit 708, such as a disk, optical disk, etc.; and communication unit 709, such as a network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0175] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as test methods. For example, in some embodiments, the test method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the test method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform test methods by any other suitable means (e.g., by means of firmware).

[0176] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0177] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0178] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0179] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0180] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0181] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, distributed system servers, or servers incorporating blockchain technology.

[0182] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0183] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A test method, comprising: configuring, based on chip parameters of a chip to be developed, a running environment for running a first algorithm on a simulator for simulating the chip to be developed, to obtain a first simulated chip; configuring, based on hardware parameters of a reference hardware, a running environment for running a second algorithm on the reference hardware, to obtain a reference chip, wherein the first algorithm and the second algorithm have the same execution logic, the first algorithm is compiled in a manner matched with the simulator, and the second algorithm is compiled in a manner matched with the reference hardware; determining a first test result and a reference result, wherein the first test result is obtained by processing test data by calling the first algorithm on the first simulated chip, and the reference result is obtained by processing the test data by calling the second algorithm on the reference chip; and based on the first test result and the reference result, performing test evaluation on the chip to be developed, to determine a test evaluation result of the chip to be developed.

2. The method of claim 1, further comprising: in a case where the test evaluation result indicates that the performance of the chip to be developed is better than that of the reference chip, configuring, on the simulator, a running environment for running a third algorithm, to obtain a second simulated chip, wherein the third algorithm has the same function as the first algorithm but different execution logic; processing the test data by calling the third algorithm on the second simulated chip, to determine a second test result; and based on the second test result and the first test result, evaluating the operator adaptation performance of the chip to be developed, to obtain an operator test result indicating operator adaptation.

3. The method of claim 1 or 2, further comprising: in a case where the operator test result indicates that the performance of the chip to be developed is better than that of the reference chip, writing an instruction set onto a circuit board, to obtain a hardware simulation chip, wherein the instruction set is used to indicate a running process of the first algorithm processing the test data; and performing test on the hardware simulation chip, to obtain a simulation test result of the chip to be developed.

4. The method of claim 3, further comprising: recording instructions executed by the first algorithm, to obtain the instruction set.

5. The method of any one of claims 1 to 4, wherein, determining the first test result, comprising: obtaining state information, wherein the state information is obtained by recording a state of the first algorithm running on the first simulated chip; analyzing the state information, to obtain field information indicating a running state; and based on the field information, determining the first test result.

6. The method of any one of claims 1 to 5, wherein, the configuring, based on chip parameters of a chip to be developed, a running environment for running a first algorithm on a simulator for simulating the chip to be developed, to obtain a first simulated chip, comprising: configuring a running environment on the simulator by using a first environment configuration tool matched with simulator parameters of the simulator, wherein the simulator parameters are determined based on the chip parameters; and compiling the first algorithm by using an algorithm compiler matched with the simulator parameters, to obtain the first simulated chip.

7. The method of any one of claims 1 to 6, wherein, The hardware parameter based on the reference hardware is used to configure a running environment for running a second operator on the reference hardware, to obtain a reference chip, including: configuring a running environment on the reference hardware by using a second environment configuration tool matched with the hardware parameter; and compiling the second operator by using an operator compiler matched with the hardware parameter, to obtain the reference chip.

8. The method of any one of claims 1 to 7, further comprising: verifying a locally stored environment configuration tool to obtain a verification result, wherein the environment configuration tool includes a first environment configuration tool for configuring a running environment of the first operator or a second environment configuration tool for configuring a running environment of the second operator; and updating the environment configuration tool in a case where the verification result indicates that the environment configuration tool is not the latest version.

9. The method of claim 1, further comprising: determining data parameters and operator calling parameters of test data based on to-be-tested indicators, wherein the operator calling parameters represent calling manners of the first operator or the second operator; and generating the test data based on the data parameters, so as to process the test data by running the first operator or the second operator by using the operator calling parameters.

10. The method of claim 1, wherein, The testing and evaluating the to-be-developed chip based on the first test result and the reference result, to determine a test evaluation result of the to-be-developed chip, including: determining a first test indicator result and a reference indicator result matched with to-be-tested indicators from the first test result and the reference result, respectively; obtaining an indicator evaluation result based on the first test indicator result and the reference indicator result; and obtaining the test evaluation result based on indicator evaluation results of the to-be-tested indicators respectively.

11. A testing apparatus, comprising: a first configuration module configured to configure a running environment for running a first operator on a simulator for simulating a to-be-developed chip based on chip parameters of the to-be-developed chip, to obtain a first simulation chip; a second configuration module configured to configure a running environment for running a second operator on reference hardware based on hardware parameters of the reference hardware, to obtain a reference chip, wherein execution logics of the first operator and the second operator are the same, a compiling manner of the first operator is matched with the simulator, and a compiling manner of the second operator is matched with the reference hardware; a result determination module configured to determine a first test result and a reference result, wherein the first test result is obtained by calling the first operator to process test data on the first simulation chip, and the reference result is obtained by calling the second operator to process the test data on the reference chip; and a test evaluation module configured to testing and evaluate the to-be-developed chip based on the first test result and the reference result, to determine a test evaluation result of the to-be-developed chip.

12. An electronic device, comprising: at least one processor; and a memory connected with the at least one processor in communication; wherein, The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-10.

13. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are for causing the computer to perform the method of any one of claims 1-10.

14. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-10.

Images