A chip evaluation method, system, electronic device and storage medium

Abstract

The application discloses a chip evaluation method and device, electronic equipment and a storage medium, and the method comprises the following steps: inputting a to-be-tested chip for testing according to an image test set, wherein the image test set comprises a test image and a credibility threshold, determining a load condition and a test index corresponding to the image test set of the to-be-tested chip, and counting the test index under different load conditions as an evaluation result of the to-be-tested chip. According to the application, the to-be-tested chip is tested according to the image test set, the corresponding load condition and test index are obtained, the corresponding test index is counted as the evaluation result of the to-be-tested chip under different load conditions, the consistency of the test scene of chip evaluation is ensured, the reliability of the evaluation result of the chip is higher, and the development and testing of the chip are facilitated.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a chip evaluation method, apparatus, electronic device, and storage medium. Background Technology

[0002] In the field of Artificial Intelligence (AI), AI chips, also known as AI accelerators, are chips specifically designed to handle the massive computational tasks in AI applications. Performance evaluation is a crucial step in the development of AI chips, verifying whether their performance meets the demands of real-world scenarios.

[0003] In the research and development of AI chips, the introduction of various types of neural networks requires chips to have stronger adaptability and robustness. However, existing chip evaluation methods mainly focus on chip power consumption and frame rate testing, without considering factors such as chip load, model input conditions, and accuracy. This makes it impossible to guarantee the consistency of test scenarios, resulting in unreliable chip evaluation results. Summary of the Invention

[0004] This invention provides a chip evaluation method, apparatus, electronic device, and storage medium to test the chip under test based on an image test set to obtain the corresponding load conditions and test indicators. Then, the corresponding test indicators under different load conditions are statistically analyzed as the evaluation results of the chip under test, ensuring the consistency of the test scenarios for chip evaluation, making the chip evaluation results more reliable, and facilitating the guidance of chip development and testing.

[0005] According to one aspect of the present invention, a chip evaluation method is provided, the method comprising:

[0006] The chip under test is tested by inputting an image test set, which includes test images and a confidence threshold.

[0007] Determine the load conditions and test indicators of the image test set corresponding to the chip under test;

[0008] The statistical test indicators under different load conditions are used to evaluate the chip under test.

[0009] According to another aspect of the present invention, a chip evaluation apparatus is provided, the apparatus comprising:

[0010] The chip testing module is used to test the chip under test based on the image test set input, wherein the image test set includes test images and confidence thresholds;

[0011] The load and test index determination module is used to determine the load and test index of the corresponding image test set of the chip under test;

[0012] The evaluation result generation module is used to statistically analyze the evaluation results of the chip under test under different load conditions.

[0013] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0014] At least one processor; and

[0015] A memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the chip evaluation method according to any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the chip evaluation method according to any embodiment of the present invention.

[0018] The technical solution of this invention involves testing the chip under test (DUT) using an image test set. The image test set includes test images and a confidence threshold. The load conditions and test metrics corresponding to the image test set for the DUT are determined, and the test metrics under different load conditions are statistically analyzed to obtain the evaluation results of the DUT. This invention ensures the consistency of the testing scenarios for chip evaluation by testing the DUT using an image test set and obtaining the corresponding load conditions and test metrics. Furthermore, it statistically analyzes the corresponding test metrics under different load conditions to obtain the evaluation results of the DUT, thus improving the reliability of the chip evaluation results and facilitating guidance for chip development and testing.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart of a chip evaluation method provided in Embodiment 1 of the present invention;

[0022] Figure 2 This is a flowchart of a chip evaluation method provided according to Embodiment 2 of the present invention;

[0023] Figure 3 This is a flowchart of a chip evaluation method provided in Embodiment 3 of the present invention;

[0024] Figure 4 This is an example diagram of a chip interface driver function test according to Embodiment 3 of the present invention;

[0025] Figure 5 This is an example diagram of a chip interface rate bandwidth test provided in Embodiment 3 of the present invention;

[0026] Figure 6 This is an example diagram of a chip load width test provided according to Embodiment 3 of the present invention;

[0027] Figure 7 This is an example diagram of a chip power consumption test provided in Embodiment 3 of the present invention;

[0028] Figure 8 This is a flowchart illustrating an optimized neural network model and algorithm according to Embodiment 3 of the present invention.

[0029] Figure 9 This is a schematic diagram of the structure of a chip evaluation device according to Embodiment 4 of the present invention;

[0030] Figure 10 This is a schematic diagram of the structure of an electronic device that implements the chip evaluation method of this invention. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] Example 1

[0034] Figure 1 This is a flowchart illustrating a chip evaluation method provided in Embodiment 1 of the present invention. This embodiment is applicable to chip evaluation. The method can be executed by a chip evaluation device, which can be implemented in hardware and / or software. This chip evaluation device can be configured in an electronic device, such as a computer or server. Figure 1 As shown in the figure, the chip evaluation method provided in this embodiment includes the following steps:

[0035] S110. Test the chip under test according to the image test set, wherein the image test set includes test images and confidence thresholds.

[0036] The chip under test (DUT) can refer to the target chip used for performance evaluation. The DUT can be various types of AI chips, including but not limited to: System-on-Chips (SoC), Application-Specific Integrated Circuit (ASIC) chips, and Field-Programmable Gate Array (FPGA) chips. The image test set can refer to an image dataset used to test the neural network model or AI algorithm in the DUT. The image test set can include test images and a confidence threshold. Test images can use existing open-source datasets, such as the KITTI dataset, Apollo Scape dataset, and Waymo dataset, or can be acquired in a real-world scene using a camera connected to the DUT. This embodiment of the invention does not impose any limitations on this. The confidence threshold can refer to a pre-set confidence threshold for the neural network model or AI algorithm in the DUT. The confidence threshold can be used to measure the recognition accuracy of the neural network model or AI algorithm in the DUT, and can be set to 85% or 90%, etc.

[0037] In this embodiment of the invention, when it is necessary to evaluate the chip under test, a pre-configured image test set can be input into the chip under test for performance evaluation. The model test results output by the neural network model that meet the confidence threshold are obtained. The image test set may include test images and a confidence threshold. Test images can be open-source datasets such as the KITTI dataset, Apollo Scape dataset, and Waymo dataset, or images captured in a real-world scene by a camera connected to the chip under test. This embodiment of the invention does not impose any limitations on this. In one specific embodiment, a test command and an image test set sent by a host computer can be received. The neural network model number in the chip under test can then be parsed from the test command. Subsequently, image recognition tests are performed on the corresponding neural network model in the chip under test according to the image test set to obtain the model recognition result corresponding to the image test set.

[0038] S120. Determine the load conditions and test indicators of the image test set corresponding to the chip under test.

[0039] The load condition can be understood as the utilization of Central Processing Unit (CPU) and bus resources by various functional modules in the chip under test during the test. Load conditions can include: CPU utilization, Graphics Processing Unit (GPU) utilization, Image Signal Processing (ISP) module utilization, Digital Signal Processing (DSP) module utilization, and bus resource usage. Test metrics can be understood as performance evaluation indicators for the chip under test. Test metrics can include: recognition accuracy of neural network models or AI algorithms, recognition time per frame of test image, operating temperature, and power consumption.

[0040] In this embodiment of the invention, the corresponding load conditions and test indicators can be obtained during the testing process of the chip under test. The load conditions may include CPU utilization, GPU utilization, ISP module utilization, DSP module utilization, bus resource utilization, etc., while the test indicators may include the recognition accuracy of the neural network model or AI algorithm, the recognition time per frame of test image, operating temperature, power consumption, etc. In one specific embodiment, during the testing process of the chip under test, the resource occupancy of various functional modules in the chip under test can be collected at a preset detection frequency as the load conditions of the chip under test. Data such as the operating temperature, power consumption, recognition accuracy of the neural network model, and recognition time per frame of test image of the chip under test can be obtained as test indicators of the chip under test.

[0041] S130: The statistical test indicators under different load conditions are the evaluation results of the chip under test.

[0042] The evaluation results can be understood as the performance evaluation results of the chip under test. The evaluation results can be the evaluation results of a single chip under test or the comprehensive evaluation results of multiple chips under test. This embodiment of the invention does not limit this. The evaluation results can include the grouping or classification information of the test indicators of the chip under test under different load conditions.

[0043] In this embodiment of the invention, the corresponding test indicators can be statistically grouped according to the different load conditions of the chip under test, and the grouped results can be used as the evaluation results of the chip under test. The evaluation results can be for a single chip under test or a comprehensive evaluation result for multiple chips under test; this embodiment of the invention does not impose any limitations on this. Furthermore, the evaluation results for the chip under test can be visualized on a host computer or other testing equipment using charts or other formats, so that testers can quickly and conveniently understand the performance of the chip under test.

[0044] The technical solution of this invention involves testing the chip under test (DUT) using an image test set. The image test set includes test images and a confidence threshold. The load conditions and test metrics corresponding to the image test set for the DUT are determined, and the test metrics under different load conditions are statistically analyzed to obtain the evaluation results of the DUT. This invention ensures the consistency of the testing scenarios for chip evaluation by testing the DUT using an image test set and obtaining the corresponding load conditions and test metrics. Furthermore, it statistically analyzes the corresponding test metrics under different load conditions to obtain the evaluation results of the DUT, thus improving the reliability of the chip evaluation results and facilitating guidance for chip development and testing.

[0045] Example 2

[0046] Figure 2 This is a flowchart of a chip evaluation method provided in Embodiment 2 of the present invention. It is further optimized and extended based on the above embodiments and can be combined with various optional technical solutions in the above embodiments. For example... Figure 2 As shown in the figure, the chip evaluation method provided in this embodiment includes the following steps:

[0047] S210: Receive test commands sent by the host computer and obtain the image test set transmitted by the host computer or camera.

[0048] In this context, the host computer can be understood as the device used to send test commands to the chip under test (DUT). The host computer can be a computer, mobile phone, tablet, panel, or touchscreen, among other devices. Test commands refer to control commands sent by the host computer to evaluate the performance of the DUT. Test commands can include information such as test interfaces, test functions, and test objects. Test interfaces can include some driver interfaces of the DUT, such as camera input interfaces, video output interfaces, bus interfaces, storage interfaces, and peripheral interfaces. Test functions can include chip interface driver function testing, chip interface bandwidth testing, chip recognition frame rate testing, chip load testing, and chip power consumption testing. Test objects can include various pre-configured neural network models and operators within the DUT.

[0049] In this embodiment of the invention, when it is necessary to evaluate the chip under test, a test command transmitted by a host computer can be received, and an image test set for testing the chip under test can be obtained through the host computer or a camera connected to the chip under test.

[0050] S220. Input the test images from the image test set into the neural network model in the chip under test, and receive the model test results output by the neural network model that meet the confidence threshold.

[0051] The neural network model can be a neural network used for image recognition, such as a convolutional neural network or a deep neural network. One or more neural network models can be pre-stored in the chip under test. The model test results can include at least the recognition results of the test image and the recognition time for each frame of the test image.

[0052] In this embodiment of the invention, each frame of test image in the acquired image test set can be sequentially input into the neural network model in the chip under test. The pre-stored neural network model is used to perform recognition tests on the target to be detected in the test image. When the recognition accuracy of the model meets the confidence threshold, the model test result output by the neural network model is received. The model test result may include at least the recognition result of the test image and the recognition time of each frame of test image.

[0053] S230. Compare the model test results with the model reference results to obtain the recognition accuracy of the neural network model.

[0054] The model reference result can be understood as the reference result for the neural network model, which may include the reference recognition result corresponding to the test image, etc.

[0055] In this embodiment of the invention, after obtaining the model test result corresponding to the neural network model in the chip under test, the model test result can be compared with the pre-configured model reference result to obtain the recognition accuracy of the corresponding neural network model. The model reference result can be the reference recognition result of the test image.

[0056] S240. Obtain the load status of the chip under test during the test process. The load status includes at least one of the following: CPU utilization, GPU utilization, image signal processing module utilization, and digital signal processing module utilization.

[0057] In this embodiment of the invention, during the testing of the chip under test, the real-time load status of the chip under test can be obtained, for example, according to a preset detection frequency. The load status includes at least one of the following: CPU utilization, GPU utilization, ISP module utilization, and DSP module utilization.

[0058] S250. Obtain the recognition time in the model test results corresponding to each frame of test image as the recognition rate, and use the recognition accuracy and recognition rate as test indicators.

[0059] In this embodiment of the invention, the recognition time of each test image frame can be determined as the recognition rate from the obtained model test results, and the recognition accuracy of the neural network model and the recognition rate of each test image frame can be used as test indicators of the chip under test.

[0060] S260. Group the recognition accuracy and recognition rate in the test indicators according to different load conditions, and use the grouped results as the evaluation results.

[0061] In this embodiment of the invention, the recognition accuracy and recognition rate in the test indicators can be grouped according to different load conditions, and the grouping results can be used as the evaluation results of the chip under test in the form of the table below.

[0062]

[0063] Furthermore, in this embodiment of the invention, multiple chips under test can be tested according to the above steps, and the evaluation results of each chip can be compared under the same input conditions (e.g., the same image test set) and the same load. By ensuring that each chip has the same test environment, the consistency and reliability of the chip evaluation results can be improved, making them more convincing.

[0064] Furthermore, based on the above embodiments, the chip evaluation method provided in Embodiment 2 of the present invention further includes:

[0065] When the accuracy of the test indicator is lower than the confidence threshold, the neural network model in the chip under test is optimized according to the preset model optimization method. The preset model optimization method includes at least one of the following: reducing the size of the test image, changing the storage location of the data, and modifying the model parameters of the neural network model.

[0066] In this embodiment of the invention, if the recognition accuracy of the test index of the chip under test is lower than the preset confidence threshold, the neural network model in the chip under test can be optimized according to the preset model optimization method so that the recognition accuracy corresponding to the neural network model in the chip under test meets the confidence threshold. The preset model optimization method may include at least one of the following: reducing the size of the test image, changing the storage location of the data, and modifying the model parameters of the neural network model.

[0067] Furthermore, based on the above embodiments, the chip evaluation method provided in Embodiment 2 of the present invention further includes:

[0068] According to the test command sent by the host computer, the speed and bandwidth of the driver interface under test of the chip under test are tested. The driver interface under test includes at least one of the following: camera input interface, video output interface, bus interface, storage interface, and peripheral interface.

[0069] In this embodiment of the invention, a test command sent by a host computer can be received. The test command can be used to parse the driver interface of the chip under test (DUT) and a test script for testing the chip interface speed and bandwidth. Then, the speed and bandwidth of the corresponding driver interface of the DUT can be tested according to the test script. The driver interface under test may include at least one of the following: a camera input interface, a video output interface, a bus interface, a storage interface, or a peripheral interface. In a specific embodiment, taking the Double-Data-Rate Fourth Generation (DDR4) interface in the storage interface as an example, the number of neural network models can be sequentially increased within the DUT according to the test script corresponding to the chip speed and bandwidth test. During this process, the read / write speed and bandwidth of the DDR4 interface of the DUT can be obtained. When the number of neural network models increases to the point where the read / write speed and bandwidth of the DDR4 interface no longer change, the maximum speed and bandwidth of the DDR4 interface can be obtained.

[0070] Furthermore, based on the above embodiments, the chip evaluation method provided in Embodiment 2 of the present invention further includes:

[0071] The power supply voltage and current of the chip under test are obtained according to the preset detection frequency;

[0072] The product of the supply voltage and the supply current is used as the power consumption value of the chip under test.

[0073] In this embodiment of the invention, during the evaluation of the chip under test (DUT), the real-time supply voltage and supply current of the DUT can be obtained at a preset detection frequency (e.g., not limited to 10 milliseconds), and the product of the supply voltage and supply current is used as the real-time power consumption value of the DUT. It is understood that the DUT may be powered by multiple power supplies; in this case, the sum of the products of the supply voltage and supply current of all power supplies can be used as the real-time power consumption value of the DUT.

[0074] The technical solution of this invention involves receiving a test command sent by a host computer and acquiring an image test set transmitted by the host computer or a camera. The test images in the image test set are input into the neural network model in the chip under test (TBD). The model test results that meet the confidence threshold are received from the neural network model. The model test results are compared with the model reference results to obtain the recognition accuracy of the neural network model. The load of the TBD during the test is obtained, including at least one of the following: CPU utilization, GPU utilization, image signal processing module utilization, and digital signal processing module utilization. The recognition time in the model test results corresponding to each frame of test image is obtained as the recognition rate. Recognition accuracy and recognition rate are used as test indicators. The recognition accuracy and recognition rate in the test indicators are grouped according to different load conditions, and the grouped results are used as the evaluation results. This invention provides an embodiment of the invention that performs image recognition tests on the neural network model in the chip under test based on an image test set, obtaining the corresponding recognition accuracy and recognition rate as test indicators. Then, the recognition accuracy and recognition rate in the test indicators are grouped according to different load conditions, and the grouped results are used as the evaluation results. This ensures the consistency of the test scenarios for chip evaluation, making the chip evaluation results more reliable and facilitating the guidance of chip development and testing.

[0075] Example 3

[0076] Figure 3 This is a flowchart of a chip evaluation method provided in Embodiment 3 of the present invention. Based on the above embodiments, this embodiment provides an implementation of a chip evaluation method capable of evaluating the performance of the AI ​​chip under test. Figure 3 As shown, the chip evaluation method provided in Embodiment 3 of the present invention specifically includes the following steps:

[0077] S310. Perform chip interface driver function test on the driver interface of the chip under test according to the test command sent by the host computer.

[0078] In embodiments of the present invention, such as Figure 4As shown, it can receive test commands sent by the host computer, parse the driver interface under test of the chip under test and the test script for testing the chip interface driver function from the test commands, and then perform chip interface driver function testing according to the test script for the corresponding driver interface of the chip under test. The driver interface under test may include: camera input interface, video output interface, Peripheral Component Interconnect Express (PCIe) bus interface, Reduced Gigabit Media Independent Interface (RGMII), DDR4 storage interface, Nor Flash storage interface, embedded Multi Media Card (eMMC) storage interface, Controller Area Network (CAN) interface, audio interface, USB interface, etc. Taking the driver function test of the Nor Flash memory interface as an example, the interface performance of its Octal Serial Peripheral Interface (OSPI) can be tested. At the same time, in order to ensure the load rate, it can be stressed, such as requiring a load rate of 70%. In addition, Flash read and write tests are performed. According to the chip manual requirements, the Nor Flash memory interface is tested for the total number of read and write operations throughout its entire life cycle.

[0079] S320: Perform chip interface rate and bandwidth tests on the driver interface of the chip under test according to the test commands sent by the host computer.

[0080] In this embodiment of the invention, a test command sent by a host computer can be received. The test command can be parsed to extract the driver interface of the chip under test and a test script for testing the chip interface speed and bandwidth. Then, the chip interface speed and bandwidth test is performed on the corresponding driver interface of the chip under test according to the test script. Taking the DDR4 interface in a storage interface as an example, as... Figure 5 As shown, the number of neural network models can be sequentially increased in the chip under test according to the test script corresponding to the chip speed and bandwidth test. During this process, the waveform of the DDR4 interface of the chip under test is detected to determine the rise time and fall time of the waveform. Then, the read and write speed of the DDR4 interface is detected. Finally, the clock frequency and bandwidth are tested. When the number of neural network models increases to the point that the read and write speed and bandwidth of the DDR4 interface no longer change, the maximum speed and bandwidth of the DDR4 interface can be obtained.

[0081] S330 receives the test command sent by the host computer, and obtains the image test set transmitted by the host computer or camera. It inputs the test images in the image test set into the neural network model in the chip under test, and receives the model test results output by the neural network model that meet the confidence threshold.

[0082] S340. Compare the model test results with the model reference results to obtain the recognition accuracy of the neural network model, and obtain the recognition time in the model test results corresponding to each frame of test image as the recognition rate, and use the recognition accuracy and recognition rate as test indicators.

[0083] S350: Obtain the load status of the chip under test during the testing process.

[0084] In this embodiment of the invention, the real-time load status of the chip under test can be acquired during the testing process, for example, according to a preset detection frequency. Figure 6 As shown, test images can be stored in different random access memory (RAM) to obtain the load conditions of the chip under test under different test environments. The load conditions can include: CPU utilization, GPU utilization, ISP module utilization, DSP module utilization, etc.

[0085] S360: Obtain the power consumption of the chip under test during the test process according to the preset detection frequency.

[0086] In embodiments of the present invention, such as Figure 7 As shown, during the testing of the chip under test (DUT), all supply voltages and currents of the DUT can be collected in real time at a preset detection frequency, such as 10 milliseconds, and saved to a power log as the power consumption performance output of the DUT. Typically, the DUT is powered by four or more power supplies; the sum of the products of the supply voltages and currents of all power supplies can be used as the real-time power consumption value of the DUT.

[0087] S370. When the accuracy of the test indicator identification is lower than the confidence threshold, optimize the neural network model and algorithm in the chip under test according to the preset model optimization method.

[0088] In embodiments of the present invention, such as Figure 8As shown, in order to achieve the optimal evaluation results while considering the reliability of the results, a reliability evaluation method can be set. Specifically, the neural network model in the chip under test will recognize the test image into multiple recognition results, each with a different probability. These results can be sorted according to their probability, with the highest being TOP1, and so on. The final recognition accuracy of the neural network model should meet the following requirements: the accuracy of TOP1 should be higher than 85% and the accuracy of TOP5 should be higher than 95%. If the recognition accuracy of the neural network model is lower than the above reliability threshold, the neural network model and algorithm in the chip under test need to be optimized. Preset model optimization methods may include: reducing the size of the test image, changing the data storage location, and modifying the model parameters of the neural network model.

[0089] S380. Group the recognition accuracy and recognition rate in the test indicators according to different load conditions, and use the grouped results as the evaluation results.

[0090] In this embodiment of the invention, as shown in the table below, multiple chips under test can be tested according to the above steps, and the evaluation results of each chip can be compared under the same input conditions (e.g., the same image test set) and the same load. By ensuring that each chip has the same test environment, the consistency and reliability of the chip evaluation results can be improved, making them more convincing.

[0091]

[0092]

[0093] The technical solution of this invention involves testing the chip interface driver function of the chip under test according to test commands sent by the host computer, testing the chip interface rate and bandwidth of the chip under test according to test commands sent by the host computer, receiving test commands sent by the host computer, acquiring image test sets transmitted by the host computer or camera, inputting the test images in the image test set into the neural network model in the chip under test, receiving the model test results that meet the confidence threshold output by the neural network model, comparing the model test results with the model reference results to obtain the recognition accuracy of the neural network model, and obtaining the recognition time in the model test results corresponding to each frame of test image as the recognition rate, and using the recognition accuracy and recognition rate as test indicators, obtaining the load of the chip under test during the test, obtaining the power consumption of the chip under test during the test according to a preset detection frequency, optimizing the neural network model and algorithm in the chip under test according to a preset model optimization method when the recognition accuracy of the test indicators is lower than the confidence threshold, grouping the recognition accuracy and recognition rate in the test indicators according to different load conditions, and using the grouped results as the evaluation results. The embodiments of the present invention realize the interface driver function test, interface rate bandwidth test, load test, power consumption test, and image recognition test of the neural network model in the chip under test. In this way, the input conditions, accuracy and load of the chip under test are constrained in the application scenario, ensuring the consistency of the test scenario for chip evaluation, making the chip evaluation results more reliable, and facilitating the guidance of chip development and testing.

[0094] Example 4

[0095] Figure 9 This is a schematic diagram of a chip evaluation device provided in Embodiment 4 of the present invention. Figure 9 As shown, the device includes:

[0096] The chip testing module 41 is used to test the chip under test according to the image test set input, wherein the image test set includes test images and confidence thresholds.

[0097] The load and test index determination module 42 is used to determine the load and test index of the image test set corresponding to the chip under test.

[0098] The evaluation result generation module 43 is used to statistically analyze the evaluation results of the chip under test under different load conditions.

[0099] The technical solution of this invention involves a chip testing module that tests the chip under test based on an image test set. The image test set includes test images and a confidence threshold. A load and test index determination module determines the load conditions and test indexes corresponding to the image test set for the chip under test. An evaluation result generation module statistically analyzes the test indexes under different load conditions to obtain the evaluation results for the chip under test. This invention ensures the consistency of the testing scenarios for chip evaluation by testing the chip under test based on the image test set and obtaining the corresponding load conditions and test indexes. Furthermore, it statistically analyzes the corresponding test indexes under different load conditions to obtain the evaluation results for the chip under test. This results in higher reliability of the chip evaluation results and facilitates guidance for chip development and testing.

[0100] Furthermore, based on the above embodiments of the invention, the chip testing module 41 includes:

[0101] The test command and image test set acquisition unit is used to receive test commands sent by the host computer and acquire image test sets transmitted by the host computer or camera.

[0102] The model test result output unit is used to input the test images from the image test set into the neural network model in the chip under test, and to receive the model test results output by the neural network model that meet the confidence threshold.

[0103] Furthermore, based on the above embodiments of the invention, the load and test index determination module 42 includes:

[0104] The recognition accuracy determination unit is used to compare the model test results with the model reference results to obtain the recognition accuracy of the neural network model.

[0105] The load acquisition unit is used to acquire the load status of the chip under test during the test process. The load status includes at least one of the following: CPU utilization, GPU utilization, image signal processing module utilization, and digital signal processing module utilization.

[0106] The test index determination unit is used to obtain the recognition time in the model test results corresponding to each frame of test image as the recognition rate, and to use the recognition accuracy and recognition rate as test indexes.

[0107] Furthermore, based on the above embodiments of the invention, the evaluation result generation module 43 includes:

[0108] The evaluation result generation unit is used to group the recognition accuracy and recognition rate in the test indicators according to different load conditions, and use the grouped results as the evaluation results.

[0109] Furthermore, based on the above embodiments of the invention, the chip evaluation apparatus further includes:

[0110] The model optimization module is used to optimize the neural network model in the chip under test according to a preset model optimization method when the recognition accuracy of the test index is lower than the confidence threshold. The preset model optimization method includes at least one of the following: reducing the size of the test image, changing the storage location of the data, and modifying the model parameters of the neural network model.

[0111] Furthermore, based on the above embodiments of the invention, the chip evaluation apparatus further includes:

[0112] The rate and bandwidth test module is used to perform rate and bandwidth tests on the driver interface under test of the chip under test according to the test commands sent by the host computer. The driver interface under test includes at least one of the following: camera input interface, video output interface, bus interface, storage interface, and peripheral interface.

[0113] Furthermore, based on the above embodiments of the invention, the chip evaluation apparatus further includes:

[0114] The power supply voltage and current acquisition module is used to acquire the power supply voltage and current of the chip under test according to a preset detection frequency.

[0115] The power consumption value determination module is used to take the product of the supply voltage and the supply current as the power consumption value of the chip under test.

[0116] The chip evaluation apparatus provided in this embodiment of the invention can execute the chip evaluation method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0117] Example 5

[0118] Figure 10 A schematic diagram of an electronic device 50 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0119] like Figure 10As shown, the electronic device 50 includes at least one processor 51 and a memory, such as a read-only memory (ROM) 52 and a random access memory (RAM) 53, communicatively connected to the at least one processor 51. The memory stores computer programs executable by the at least one processor. The processor 51 can perform various appropriate actions and processes based on the computer program stored in the ROM 52 or loaded into the RAM 53 from storage unit 58. The RAM 53 can also store various programs and data required for the operation of the electronic device 50. The processor 51, ROM 52, and RAM 53 are interconnected via a bus 54. An input / output (I / O) interface 55 is also connected to the bus 54.

[0120] Multiple components in electronic device 50 are connected to I / O interface 55, including: input unit 56, such as keyboard, mouse, etc.; output unit 57, such as various types of monitors, speakers, etc.; storage unit 58, such as disk, optical disk, etc.; and communication unit 59, such as network card, modem, wireless transceiver, etc. Communication unit 59 allows electronic device 50 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0121] Processor 51 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 51 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 51 performs the various methods and processes described above, such as chip evaluation methods.

[0122] In some embodiments, the chip evaluation method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 58. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 50 via ROM 52 and / or communication unit 59. When the computer program is loaded into RAM 53 and executed by processor 51, one or more steps of the chip evaluation method described above may be performed. Alternatively, in other embodiments, processor 51 may be configured to perform the chip evaluation method by any other suitable means (e.g., by means of firmware).

[0123] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0124] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0125] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0126] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0127] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0128] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0129] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0130] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A chip evaluation method, characterized in that, The method includes: The chip under test is tested by inputting an image test set, wherein the image test set includes test images and a confidence threshold. Determine the load and test indicators of the chip under test corresponding to the image test set; The test indicators are statistically analyzed under different load conditions to obtain the evaluation results of the chip under test; The step of testing the chip under test based on the image test set includes: Receive test commands sent by the host computer and obtain the image test set transmitted by the host computer or camera; The test images in the image test set are input into the neural network model in the chip under test, and the model test results that meet the confidence threshold are received from the neural network model. Determining the load and test metrics of the chip under test corresponding to the image test set includes: The recognition accuracy of the neural network model is obtained by comparing the model test results with the model reference results. The load status of the chip under test during the test process is obtained, and the load status includes at least one of the following: CPU utilization, GPU utilization, image signal processing module utilization, and digital signal processing module utilization. The recognition time in the model test result corresponding to each frame of the test image is obtained as the recognition rate, and the recognition accuracy and the recognition rate are used as the test indicators.

2. The method according to claim 1, characterized in that, The evaluation results of the chip under test, which are obtained by statistically analyzing the test indicators under different load conditions, include: The recognition accuracy and recognition rate in the test indicators are grouped according to different load conditions, and the grouped results are used as the evaluation results.

3. The method according to claim 1, characterized in that, Also includes: When the recognition accuracy of the test index is lower than the confidence threshold, the neural network model in the chip under test is optimized according to a preset model optimization method, wherein the preset model optimization method includes at least one of the following: reducing the size of the test image, changing the storage location of the data, or modifying the model parameters of the neural network model.

4. The method according to claim 1, characterized in that, Also includes: The speed and bandwidth of the driver interface under test of the chip under test are tested according to the test command sent by the host computer. The driver interface under test includes at least one of the following: camera input interface, video output interface, bus interface, storage interface, and peripheral interface.

5. The method according to claim 1, characterized in that, Also includes: The power supply voltage and power supply current of the chip under test are obtained according to a preset detection frequency; The product of the supply voltage and the supply current is used as the power consumption value of the chip under test.

6. A chip evaluation device, characterized in that, The device includes: A chip testing module is used to test the chip under test according to an image test set, wherein the image test set includes test images and a confidence threshold. The load and test index determination module is used to determine the load and test index of the chip under test corresponding to the image test set; The evaluation result generation module is used to statistically analyze the test indicators under different load conditions to obtain the evaluation results of the chip under test; The chip testing module includes: The test command and image test set acquisition unit is used to receive test commands sent by the host computer and acquire the image test set transmitted by the host computer or camera. The model test result output unit is used to input the test image in the image test set into the neural network model in the chip under test, and to receive the model test result output by the neural network model that meets the confidence threshold. The load and test index determination module includes: The recognition accuracy determination unit is used to compare the model test results with the model reference results to obtain the recognition accuracy of the neural network model; A load condition acquisition unit is used to acquire the load condition of the chip under test during the test process, wherein the load condition includes at least one of the following: CPU utilization, GPU utilization, image signal processing module utilization, and digital signal processing module utilization. The test index determination unit is used to obtain the recognition time in the model test result corresponding to each frame of the test image as the recognition rate, and to use the recognition accuracy and the recognition rate as the test index.

7. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the chip evaluation method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that are used to cause a processor to execute the chip evaluation method according to any one of claims 1-5.

Images