NPU testing methods, related devices, equipment, and storage media

Abstract

This application discloses an NPU testing method and related apparatus, equipment, and storage medium. The NPU testing method includes: selecting a target NPU and several computing modules of the target NPU as a test object; constructing multiple test structures based on the input layer, output layer, and the test object; obtaining the measured execution time of each test structure operating independently; and obtaining a first time for data processing of the test object and a second time for data transmission between the test object and its adjacent objects in the test structures based on the measured execution time of each test structure. This scheme can accurately and meticulously test the operating efficiency of the NPU.

Description

Technical Field

[0001] This application relates to the field of chip testing technology, and in particular to an NPU testing method and related apparatus, equipment and storage medium. Background Technology

[0002] NPU (Neural-Network Processing Unit) chips, designed specifically for edge-side neural network inference, are widely used in edge-side inference due to their advantages such as high efficiency and low power consumption.

[0003] However, different NPUs are limited by their hardware architecture, resulting in differences in their computational power utilization for different network structures and data transmission between hardware modules. Therefore, in order to guide algorithm deployment personnel in deploying networks reasonably before network application to maximize the performance of NPUs and ultimately improve the inference efficiency of neural networks, it is necessary to accurately and meticulously test the operating efficiency of NPUs. Summary of the Invention

[0004] The main technical problem addressed by this application is to provide an NPU testing method and related apparatus, equipment, and storage medium that can accurately and meticulously test the operating efficiency of the NPU.

[0005] To address the aforementioned technical problems, the first aspect of this application provides an NPU testing method, comprising: selecting a test object from a target NPU and several computing modules of the target NPU; constructing multiple test structures based on the input layer, output layer, and the test object; obtaining the measured execution time of each test structure running independently; and obtaining a first time for data processing of the test object and a second time for data transmission between the test object and adjacent objects of the test object in the test structures based on the measured execution time of each test structure.

[0006] To address the aforementioned technical problems, a second aspect of this application provides an NPU testing device, comprising: a selection module, a construction module, an acquisition module, and a solution module. The selection module is used to select a test object from a target NPU and several computing modules of the target NPU. The construction module is used to construct multiple test structures based on the input layer, the output layer, and the test object. The acquisition module is used to acquire the measured execution time of each test structure operating independently. The solution module is used to obtain, based on the measured execution time of each test structure, the first execution time of data processing for the test object and the second execution time of data transmission between the test object and adjacent objects of the test object in the test structure.

[0007] To address the aforementioned technical problems, a third aspect of this application provides an electronic device including a memory and a processor coupled to each other. The memory stores program instructions, and the processor executes the program instructions to implement the NPU testing method described in the first aspect.

[0008] To address the aforementioned technical problems, a fourth aspect of this application provides a computer-readable storage medium storing program instructions executable by a processor, the program instructions being used to implement the NPU testing method of the first aspect described above.

[0009] The above scheme selects the object under test (AUT) from the target NPU and its various computing modules, and constructs multiple test structures based on the input layer, output layer, and AUT. Then, it obtains the measured execution time of each test structure running independently. Based on the real-time execution time of each test structure, it obtains the first time for data processing of the AUT and the second time for data transmission between the AUT and its neighboring objects within the test structures. On the one hand, by constructing multiple test structures related to the AUT and obtaining the measured execution time of each structure independently, and using these measured times to calculate the target NPU's operating efficiency, the accuracy of NPU testing can be improved. On the other hand, after obtaining the measured execution time of each test structure, the first time for data processing of the AUT itself is calculated, and simultaneously, the second time for data transmission between the AUT and its neighboring objects is also calculated. This allows for the comprehensive measurement of key factors affecting NPU operating efficiency, contributing to the detailed nature of NPU testing. Therefore, it can accurately and meticulously test the operating efficiency of the NPU. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating an embodiment of the NPU testing method of this application;

[0011] Figure 2a This is a schematic diagram of an embodiment of the test structure when the cumulative module is selected as the object under test;

[0012] Figure 2b This is a schematic diagram of another embodiment of the test structure when the cumulative module is selected as the object under test;

[0013] Figure 3a This is a schematic diagram of an embodiment of a test structure when a nonlinear module is selected as the test object and an accumulator module is selected as the reference object;

[0014] Figure 3b This is a schematic diagram of another embodiment of the test structure when a nonlinear module is selected as the test object and an accumulator module is selected as the reference object;

[0015] Figure 4aThis is a schematic diagram of an embodiment of the test structure when the target NPU is selected as the test object;

[0016] Figure 4b This is a schematic diagram of another embodiment of the test structure when the target NPU is selected as the test object;

[0017] Figure 5 This is a schematic diagram of the framework of an embodiment of the NPU testing device of this application;

[0018] Figure 6 This is a schematic diagram of the framework of an embodiment of the electronic device of this application;

[0019] Figure 7 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0020] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0021] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0022] In this paper, the terms "system" and "network" are often used interchangeably. The term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the slash " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this paper indicates two or more objects.

[0023] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the NPU testing method of this application. Specifically, it may include the following steps:

[0024] Step S11: Select the object to be tested from the target NPU and its various computing modules.

[0025] It should be noted that the various computational modules may include, but are not limited to, the Multiply Accumulate (MAC) module, the Tensor Processing (TP) module, and the Non-linear (None Linear) module, etc., without limitation here. Among these, the Multiply Accumulate (MAC) module is typically the main computational module in the NPU, usually accounting for a major portion of the NPU's computing power, and is primarily used for matrix multiplication operations; the Tensor Processing (TP) module is typically used for processing tensors; and the Non-linear module is typically used for non-linear calculations in neural networks. Of course, the above examples are merely specific examples of computational modules contained in an NPU in practical applications and do not limit the specific computational modules contained in an NPU.

[0026] In one implementation scenario, when the primary focus is on the overall operational efficiency of the target NPU, the target NPU can be selected as the object under test to test its overall operational efficiency. In this case, the CPU can be further introduced as an adjacency of the target NPU when constructing the subsequent test structure. This allows for further calculation of the transmission time between the target NPU and the CPU, such as the transmission time from the target NPU to the CPU and the transmission time from the CPU to the target NPU. For details, please refer to the relevant descriptions below, which will not be elaborated upon here.

[0027] In another implementation scenario, when the primary focus is on the operational efficiency of the computing modules within the target NPU, these modules can be selected as the test objects to evaluate their efficiency. In this case, other computing modules can be further considered as reference objects. A test structure can then be constructed based on both the test object and the reference objects to further measure the transmission time between them. For example, the transmission time from computing module A (e.g., an accumulation module) to computing module B (e.g., a tensor computation module) and vice versa. For details, please refer to the relevant descriptions below, which will not be elaborated upon here.

[0028] In another implementation scenario, each computing module and the target NPU itself can be selected as the test objects to conduct a comprehensive test on the operating efficiency of each computing module in the target NPU and the overall operating efficiency of the target NPU.

[0029] Step S12: Construct multiple test structures based on the input layer, output layer, and the object under test.

[0030] Specifically, as mentioned earlier, the target NPU, the computation module, or both can be selected as the test objects. Based on this, when the test object is the target NPU, multiple test structures can be constructed based on the input layer, output layer, and target NPU. When the test object is the computation module, a network layer adapted to the computation object of the computation module can be selected as the target layer, and multiple test structures can be constructed based on the input layer, output layer, and target layer. This approach, when the test object is the target NPU, constructs multiple test structures based on the input layer, output layer, and target NPU; when the test object is the computation module, it first obtains a network layer adapted to the computation object of the computation module as the target layer, and then constructs multiple test structures based on the input layer, output layer, and target layer. This allows for the construction of test structures for different situations, thereby improving the accuracy of NPU testing.

[0031] It should be noted that the specific construction of the test structure is not limited in the embodiments disclosed in this application, and multiple test structures may have different constructions. Several construction forms are given below as examples to illustrate the construction methods of the test structure.

[0032] In an implementation scenario, when selecting a computation module as the test object, as mentioned earlier, a network layer that matches the computational object of the computation module can be selected as the target layer. Taking a target NPU containing computation modules such as an accumulation module, a tensor processing module, and a non-linear module as an example, if the computation module is an accumulation module, a convolutional layer can be selected as the target layer; or, if the computation module is a tensor processing module, any one of a normalized exponent (softmax) layer, a dimension reshaping layer, or an add layer can be selected as the target layer; or, if the computation module is a non-linear module, an activation layer (such as a sigmoid layer) can be selected as the target layer. This principle applies when the target NPU contains other computation modules, and will not be elaborated further here. The above method selects a convolutional layer as the target layer when the computation module is an accumulation module, a normalized exponential layer, a dimension renormalization layer, or an addition layer when the computation module is a tensor processing module, and an activation layer when the computation module is a nonlinear module. Therefore, it can select a network layer that is compatible with the computation object as the target layer for different computation modules, which helps to improve the accuracy of testing the running efficiency of the computation modules in the target NPU.

[0033] In one implementation scenario, after selecting a network layer adapted to the computational object of the computing module as the target layer, different first substructures containing the target layer can be obtained. This allows different first substructures to be inserted between the input layer and the output layer, resulting in multiple test structures when the computing module is selected as the object under test. Of course, depending on the different expected metrics for runtime efficiency testing, it is possible to add or omit networks adapted to the computational objects of other computing modules to construct the first substructure based on the target layer. Please refer to the following description for details.

[0034] In a specific implementation scenario, in response to the expectation of obtaining the transmission time of at least one of the input layer and output layer, at least a series structure and a parallel structure formed by several target layers can be obtained as different first substructures. In the parallel structure, each target layer is connected to a different output layer, and the input layer and output layer are adjacent objects. Taking the selection of an accumulator module as the test object as an example, as mentioned earlier, a network layer adapted to the computational object of the accumulator module, i.e., a convolutional layer, can be selected as the target layer. Based on this, a series structure and a parallel structure formed by several convolutional layers can be obtained as different first substructures and inserted between the input layer and the output layer respectively, to provide multiple test structures when selecting the accumulator module as the test object. For example, please refer to [reference needed]. Figure 2a and Figure 2b , Figure 2a This is a schematic diagram of an embodiment of the test structure when the cumulative module is selected as the object under test. Figure 2b This is a schematic diagram of another embodiment of the test structure when the accumulation module is selected as the object under test. For example... Figure 2a The number of convolutional layers in the cascaded structure can be N, such as 2, 3, 4, etc. Figure 2b As shown, the number of convolutional layers in a parallel structure can, for example, be set to two, etc. There is no limitation on the number of convolutional layers in serial and parallel structures. Please refer to further details. Figure 2a and Figure 2b The test structures shown respectively, when run independently, can be represented as follows:

[0035]

[0036]

[0037] It should be noted that the above formula (1) represents Figure 2a The time consumption expression for the test structure running independently, as shown in formula (2) above, represents... Figure 2b The given test structure is shown as having a time consumption expression when run independently. Wherein, Indicates the time taken for data input. T represents the time taken for data output. convThis represents the data processing time of the convolutional layer. Furthermore, in this example and in the following description of this application, the transmission time between adjacent and similar computations (e.g., ...) will be ignored. Figure 2a and Figure 2b (The test structure shown ignores the transmission time between adjacent convolutional layers). Based on this, different N values ​​can be designed, and the measured latency of different test structures can be substituted into T on the left side of the equation to solve for the first data processing time of the accumulator module (the object under test) in the target NPU, the second data input time of the input layer (the adjacent object), and the second data output time of the output layer (the adjacent object). For the specific calculation process, please refer to the relevant description below, which will not be elaborated here. Of course, Figure 2a and Figure 2b The diagram shown is merely one possible construction of the test structure when selecting the accumulation module as the test object in practical applications, but it is not limited to this. Furthermore, the construction method of the test structure can be deduced similarly when selecting other computation modules as the test object, which will not be listed here. For example, when selecting the tensor computation module as the test object, the following can be used... Figure 2a and Figure 2b Replace the convolutional layers with dimension renormalization layers; or, when selecting non-linear modules as the test object, you can... Figure 2a and Figure 2b The convolutional layers are replaced with activation layers. In response to the expectation of obtaining the transmission time of at least one of the input or output layers, the above method obtains at least a series and parallel structure formed by several target layers as different first substructures. In the parallel structure, each target layer is connected to a different output layer, and the input and output layers are adjacent objects. Therefore, the construction complexity of the test structure can be reduced, while simultaneously improving the accuracy of subsequent time consumption calculations.

[0038] In another specific implementation scenario, unlike the aforementioned implementations, there is also the possibility of wanting to obtain the transmission time between different computing modules in the target NPU. In this case, in response to the desire to obtain the transmission time between the object under test and the reference object, a network layer adapted to the computing object of the reference object can be selected as the reference layer. Different first substructures formed by the target layer connecting to different numbers of reference layers can be obtained, where the reference object is a computing module in the target NPU, and its computing object is different from the object under test; the reference object is an adjacent object. For example, a first substructure formed by the target layer connecting to two reference layers, a first substructure formed by the target layer connecting to three reference layers, etc., can be obtained, and these will not be listed here. Taking the object under test as a nonlinear module and the reference object as an accumulator module as an example, the network layer adapted to the computing object of the nonlinear layer of the object under test can be, for example, set as a sigmoid layer. Furthermore, as mentioned above, a network layer adapted to the computing object of the accumulator module of the reference object, such as a convolutional layer, can be selected as the reference layer. Based on this, a first substructure formed by a sigmoid layer connecting two convolutional layers can be obtained, as well as a first substructure formed by a sigmoid layer connecting three convolutional layers. Of course, practical applications are not limited to this. Furthermore, the aforementioned first substructure can be inserted between the input layer and the output layer to obtain the test structure. It should be noted that when the target layer is connected to three or more reference layers, one of the reference layers is also connected to the input layer, and the remaining reference layers are also connected to the output layer. For an example, please refer to the relevant documentation. Figure 3a and Figure 3b , Figure 3a This is a schematic diagram of an embodiment of the test structure when a nonlinear module is selected as the test object and an accumulator module is selected as the reference object. Figure 3b This is a schematic diagram of another embodiment of the test structure when a nonlinear module is selected as the test object and an accumulator module is selected as the reference object. For example... Figure 3a As shown, the sigmoid layer, serving as the target layer, is connected between two convolutional layers serving as reference layers; as Figure 3b As shown, the sigmoid layer, serving as the target layer, is connected to three convolutional layers, which serve as reference layers. One of these convolutional layers is also connected to the input layer, while the others are connected to the output layer. Please refer to further details. Figure 2a and Figure 2b The test structures shown respectively, when run independently, can be represented as follows:

[0039]

[0040]

[0041] It should be noted that the above formula (3) represents Figure 3a The time consumption expression for the test structure running independently, as shown in formula (4) above, represents... Figure 3b The expression for the time consumption of the test structure running independently is shown. The specific meanings of the parameters identical to those in formulas (1) and (2) can be found in the relevant descriptions of formulas (1) and (2), and will not be repeated here. Furthermore, This indicates the time taken to process data in the sigmoid layer. This represents the transmission time from the convolutional layer to the sigmoid layer. This represents the transmission time from the sigmoid layer to the convolutional layer. Based on this, different N values ​​can be designed, and the measured transmission times of different test structures can be substituted into T on the left side of the equation to solve for the first data processing time of the nonlinear module (the object under test) in the target NPU, and the second data transmission time between it and the adjacent accumulator module. For the specific calculation process, please refer to the relevant descriptions below, which will not be elaborated here. Of course, Figure 3a and Figure 3b This is merely one possible construction of the test structure when a nonlinear module is selected as the test object and an accumulator module as the reference object in practical applications, but it is not limited to this. Furthermore, the construction method of the test structure can be deduced similarly when other computational modules are selected as the test object and reference object respectively, which will not be listed here. For example, when selecting a tensor processing module as the test object and an accumulator module as the reference object, the construction method can be... Figure 3a and Figure 3b Replace the sigmoid layer with a dimension renormalization layer; or, when selecting the nonlinear module as the test object and the tensor processing module as the reference object, you can... Figure 3a and Figure 3b The convolutional layers are replaced with dimension remodeling layers. In response to the desire to obtain the transmission time between the test object and the reference object, the above method selects a network layer adapted to the computation of the reference object as the reference layer, and obtains different first substructures formed by the target layer connecting different numbers of reference layers. Since the reference object is a computation module in the target NPU, and its computational object differs from the test object, and the reference object is an adjacent object, the construction complexity of the test structure can be reduced, while simultaneously improving the accuracy of subsequent time consumption measurements.

[0042] In another implementation scenario, unlike the case where a computing module is selected as the test object, when a target NPU is selected as the test object, different second substructures containing the target NPU can be obtained. It should be noted that each of these different second substructures can contain a CPU connected between multiple target NPUs. Multiple target NPUs can include, but are not limited to, 2, 3, or 4 target NPUs, etc., and are not limited here. Based on this, different second substructures can be inserted between the input and output layers to obtain multiple test structures when the target NPU is selected as the test object. In this case, the CPU is the adjacent object, and through subsequent testing processes, the computational efficiency of the target NPU itself (such as data processing time) can be obtained, as well as the data transfer time between the target NPU and the CPU.

[0043] In a specific implementation scenario, it can be understood that, when the time spent on data transfer between the target NPU and the CPU is not a concern, different second substructures may not include a CPU; for example, they may only include different numbers of target NPUs. For instance, it can be... Figure 2a and Figure 2b Simply replace the intermediate convolutional layers with the target NPU.

[0044] In another specific implementation scenario, please refer to [the relevant documentation]. Figure 4a and Figure 4b , Figure 4a This is a schematic diagram of an embodiment of the test structure when the target NPU is selected as the object under test. Figure 4b This is a schematic diagram of another embodiment of the test structure when the target NPU is selected as the object under test. For example... Figure 4a As shown, a second substructure formed by the CPU connected between two target NPUs can be obtained. This second substructure can then be inserted between the input and output layers to obtain one of the test structures when selecting a target NPU as the object under test. Please refer to further details. Figure 4b This allows us to obtain a second substructure formed by a star connection between the CPU and more than two target NPUs. This second substructure can then be inserted between the input and output layers, resulting in another test structure when the target NPU is the object under test. It should be noted that for the latter, such as... Figure 4a As shown, more than two target NPUs can specifically include three, four, etc., and this is not limited here. Furthermore, one target NPU is connected to the input layer, and the remaining target NPUs are connected to the output layer. Please refer to further details. Figure 4a and Figure 4b The test structures shown respectively, when run independently, can be represented as follows:

[0045]

[0046]

[0047] It should be noted that the above formula (5) represents Figure 4a The time consumption expression for the test structure running independently, as shown in formula (6) above, represents... Figure 4b The expression for the time consumption of the test structure running independently is shown. The specific meanings of the same parameters as in formulas (1) and (2) can be found in the relevant descriptions of formulas (1) and (2), and will not be repeated here. Furthermore, T... npu This indicates the data processing time of the target NPU. This indicates the CPU's data processing time. This indicates the time taken for the data transfer from the target NPU to the CPU. This indicates the time taken for data transfer from the CPU to the NPU. Based on this, we can... Figure 4a The test structure incorporates different numbers of CPUs and substitutes the measured latency of the test structure into the left-hand side T of the formula to calculate the first data processing time of the target NPU as the test object, as well as the transmission time from the target NPU to the CPU and the transmission time from the CPU to the target NPU. For the specific calculation process, please refer to the relevant description below, which will not be elaborated here. This method obtains a second substructure formed by the CPU connected between two target NPUs, and a second substructure formed by the CPU and more than two target NPUs connected in a star topology, with one target NPU connected to the ingestion layer and the others connected to the output layer. Therefore, it reduces the construction complexity of the test structure and improves the accuracy of subsequent latency calculations.

[0048] Step S13: Obtain the measured time of each test structure running independently.

[0049] Specifically, the multiple test structures constructed above can be run independently to obtain the measured time consumption of each test structure. For example, when selecting a computation module as the test object, if it is desired to obtain the transmission time of at least one of the input layer and output layer, a structure similar to... Figure 2a and Figure 2b The test structure shown allows for the independent execution of each test structure on the target NPU to obtain the measured time of each test structure; alternatively, when selecting a computing module as the object under test, if it is desired to obtain the transmission time between the object under test and the reference object, a similar structure can be constructed. Figure 3a and Figure 3b The test structure shown allows for the independent execution of each test structure on the target NPU to obtain the measured execution time for each test structure; alternatively, when selecting the target NPU as the test object, a test structure such as... Figure 4a and Figure 4bThe test structure shown can be used to run each test structure separately to obtain the actual time taken for each test structure.

[0050] Step S14: Based on the measured time consumption of each test structure, obtain the first time consumption of data processing of the object under test and the second time consumption of data transmission between the object under test and the adjacent objects of the object under test in the test structure.

[0051] Specifically, based on the measured processing time of the test structures, objective equations can be constructed with the processing time of the object under test and the transmission time between the object under test and its adjacent objects as unknowns. These objective equations, constructed separately for each test structure, are then solved to obtain the first and second processing times. In other words, a system of equations can be formed by simultaneously solving the objective equations constructed for each test structure, and the first and second processing times can be obtained. This method, by simultaneously solving the objective equations constructed from the measured processing times of each test structure to obtain the first and second processing times, can minimize the complexity of efficiency testing.

[0052] In one implementation scenario, when selecting a computing module as the object to be tested, if it is desired to obtain the transmission time of at least one of the input layer and output layer, a structure such as... Figure 2a and Figure 2b For the test structure shown, please refer to [link / reference] for details. Figure 2a and Figure 2b And related descriptions, will not be repeated here. Figure 2a and Figure 2b Taking the test structure shown as an example, we can obtain the results in... Figure 2a The test structure when N is 1 actually measures the time consumption, and Figure 2b Based on the measured time of the test structure, and referring to formulas (1) and (2), the previous measured time can be multiplied by two, and the subsequent measured time subtracted to obtain the input time, which can be regarded as the second time of data transmission; similarly, the input time can be obtained. Figure 2a The test structure when N is 2 actually measures the time consumption, and Figure 2b Based on the measured time of the test structure, and referring to formulas (1) and (2), the output time can be obtained by subtracting the previous measured time from the subsequent measured time, which can be regarded as the second time of data transmission. Finally, according to formula (1) or formula (2), the input time and output time obtained above can be substituted to solve for the first time of data processing of the cumulative module, which is the object under test. Of course, if in this case, a different structure is constructed Figure 2a and Figure 2b The test structure shown can be used to calculate the first and second execution times, and examples will not be repeated here. It should be noted that the test structure can be modified during actual testing. Figure 2aand Figure 2b The number of channels c, width w, and height h were used to verify a convolutional structure that is more suitable for the target NPU.

[0053] In another implementation scenario, when selecting a computing module as the object under test, if it is desired to obtain the transmission time between the object under test and the reference object, a structure such as... Figure 3a and Figure 3b For the test structure shown, please refer to [link / reference] for details. Figure 3a and Figure 3b And related descriptions, will not be repeated here. Figure 3a and Figure 3b Taking the test structure shown as an example, we can obtain the results in... Figure 3a The test structure when N is 1 actually measures the time consumption, and Figure 3b Based on the measured time of the test structure, the previously obtained input time, the first data processing time of the 2x accumulator module, and the output time can be subtracted from the previous measured time to obtain a time difference. Simultaneously, the previously obtained input time, the first data processing time of the 3x accumulator module, and the 2x output time can be subtracted from the next measured time to obtain another time difference. Finally, subtracting the previous time difference from the next time difference yields the second data transmission time from the nonlinear module to the accumulator module. Alternatively, subtracting the second data transmission time from the previous time difference, or subtracting the second data transmission time of the 2x nonlinear module to the accumulator module from the next time difference, yields the sum of the first data processing time of the nonlinear module and the second data transmission time from the accumulator module to the nonlinear module. Further, it is possible to obtain... Figure 3a When N is 2, the actual measured time of the test structure is then subtracted from the previously obtained input time, the first time of data processing of the 2x accumulator module, the output time, and the second time of data transmission from the nonlinear module to the accumulator module. This yields the sum of the second time of data transmission from the accumulator module to the nonlinear module and the first time of data processing of the 2x nonlinear module. Subtracting the previous sum from this sum gives the first time of data processing by the nonlinear module. Then, by combining any of the aforementioned sums of time with the first time of data processing by the nonlinear module and the second time of data transmission from the nonlinear module to the accumulator module, the second time of data transmission from the accumulator module to the nonlinear module can be calculated. Of course, in this case, a different structure may be constructed. Figure 3a and Figure 3b The test structure shown can be used to calculate the first and second time consumption, which will not be listed in detail here.

[0054] In another implementation scenario, when selecting the target NPU as the object under test, a structure such as... Figure 4a and Figure 4b For the test structure shown, please refer to [link / reference] for details. Figure 4a and Figure 4b And related descriptions, will not be repeated here. Figure 4a and Figure 4b Taking the test structure shown as an example, we can obtain... Figure 4a The actual test time of the test structure with 1 CPU in the middle was obtained, and the data was collected. Figure 4b Based on the measured time of the test structure, the previously calculated input and output times can be subtracted from the previous measured time to obtain a time difference. Simultaneously, the previously calculated input time and twice the output time can be subtracted from the next measured time to obtain another time difference. This second time difference can then be subtracted from the first time difference to obtain yet another time difference. This final time difference actually represents the sum of the first time spent on data processing by the target NPU and the second time spent on data transfer from the CPU to the target NPU. Furthermore, it is possible to obtain... Figure 4a The measured time of the test structure with 2 CPUs was then subtracted from the measured time with 1 CPU to obtain the first CPU data processing time. Based on this, the following can be obtained: Figure 4a The actual time consumption of the test structure was measured when the number of CPUs was 3 and other values ​​were taken. Based on the aforementioned formulas (5) and (6), a system of equations was established to obtain the first time consumption for data processing by the target NPU, the second time consumption for data transmission from the target NPU to the CPU, and the second time consumption for data transmission from the CPU to the target NPU. Of course, in this case, a different structure may be constructed. Figure 3a and Figure 3b The test structure shown can be used to calculate the first and second time consumption, which will not be listed in detail here.

[0055] It should be noted that the above examples are merely possible implementations for calculating the first and second time based on the actual measured time consumption in the case of the test structure shown in the attached figure, and are not limited to other possible calculation methods when using test structures other than those shown in the attached figure.

[0056] Of course, in practical applications, in addition to the aforementioned refined efficiency metrics such as the first and second execution times to characterize the target NPU's operating efficiency at a fine-grained level, coarse efficiency metrics can also be obtained to characterize the target NPU's operating efficiency at a coarse-grained level. For example, the first and second neural networks can be obtained first. It should be noted that the main time consumption of the first neural network is the data computation time, and the main time consumption of the second neural network is the data transmission time. For example, the first neural network can include, but is not limited to, neural networks with large feature resolution or a large number of channels, such as Res-Net, Dense-Net, and VGG, and large kernels for operators such as convolution. The second neural network can include, but is not limited to, neural networks with smaller feature resolution or a smaller number of channels, such as Mobile-Net and Inception Net; no limitation is made here. Based on this, a first result characterizing the target NPU's computing power can be obtained based on the time consumption data of running the first neural network on the target NPU, and a second result characterizing the target NPU's memory bandwidth can be obtained based on the time consumption data of running the second neural network on the target NPU. For example, the average time taken by the target NPU to run each of the first neural networks can be obtained. The first result includes a first score indicating the target NPU's computing power, and this first score is negatively correlated with the average time taken. Similarly, the average time taken by the target NPU to run each of the second neural networks can be obtained. The second result includes a second score indicating the target NPU's memory bandwidth, and this second score is negatively correlated with the average time taken. This method obtains information on the first and second neural networks, where the main time taken by the first neural network is data computation time, and the main time taken by the second neural network is data transmission time. Based on the time taken by the target NPU to run the first neural network, a first result characterizing the target NPU's computing power is obtained, and based on the time taken by the target NPU to run the second neural network, a second result representing the target NPU's memory bandwidth is obtained. Therefore, the operating efficiency of the target NPU can be tested at a coarse-grained level to obtain a macro-level efficiency index for the target NPU.

[0057] The above scheme selects the object under test (AUT) from the target NPU and its various computing modules, and constructs multiple test structures based on the input layer, output layer, and AUT. Then, it obtains the measured execution time of each test structure running independently. Based on the real-time execution time of each test structure, it obtains the first time for data processing of the AUT and the second time for data transmission between the AUT and its neighboring objects within the test structures. On the one hand, by constructing multiple test structures related to the AUT and obtaining the measured execution time of each structure independently, and using these measured times to calculate the target NPU's operating efficiency, the accuracy of NPU testing can be improved. On the other hand, after obtaining the measured execution time of each test structure, the first time for data processing of the AUT itself is calculated, and simultaneously, the second time for data transmission between the AUT and its neighboring objects is also calculated. This allows for the comprehensive measurement of key factors affecting NPU operating efficiency, contributing to the detailed nature of NPU testing. Therefore, it can accurately and meticulously test the operating efficiency of the NPU.

[0058] Please see Figure 5 , Figure 5 This is a schematic diagram of the framework of an embodiment of the NPU testing device 50 of this application. The NPU testing device 50 includes: a selection module 51, a construction module 52, an acquisition module 53, and a solution module 54. The selection module 51 is used to select the object under test from the target NPU and several computing modules of the target NPU; the construction module 52 is used to construct multiple test structures based on the input layer, the output layer, and the object under test; the acquisition module 53 is used to acquire the measured time of each test structure running independently; the solution module 54 is used to obtain the first time of data processing of the object under test and the second time of data transmission between the object under test and the adjacent objects of the object under test in the test structure based on the measured time of each test structure.

[0059] The above scheme, on the one hand, improves the accuracy of NPU testing by constructing multiple test structures related to the object under test and obtaining the measured execution time of each test structure independently, and then using the measured execution time to calculate the operating efficiency of the target NPU. On the other hand, after obtaining the measured execution time of each test structure, it calculates the first time of data processing of the object under test itself, and at the same time, it also calculates the second time of data transmission between the object under test and its adjacent objects. This allows for the comprehensive measurement of key factors affecting NPU operating efficiency, contributing to the detailed nature of NPU testing. Therefore, it can accurately and meticulously test the operating efficiency of the NPU.

[0060] In some disclosed embodiments, the construction module 52 includes a first construction submodule, which is used to construct multiple test structures based on the input layer, the output layer, and the target NPU in response to the test object being the target NPU; the construction module 52 includes a second construction submodule, which is used to select a network layer that is compatible with the computing object of the computing module as the target layer in response to the test object being the computing module, and construct multiple test structures based on the input layer, the output layer, and the target layer.

[0061] In some disclosed embodiments, the second construction submodule includes a first response unit, configured to select a convolutional layer as the target layer in response to the computation module being an accumulation module; the second construction submodule includes a second response unit, configured to select any one of a normalized exponential layer, a dimension renormalization layer, and an addition layer as the target layer in response to the computation module being a tensor processing module; the second construction submodule includes a third response unit, configured to select an activation layer as the target layer in response to the computation module being a nonlinear module.

[0062] In some disclosed embodiments, the second construction submodule includes a first substructure acquisition unit for acquiring different first substructures containing the target layer; the second construction submodule includes a first insertion unit for inserting different first substructures between the input layer and the output layer to obtain multiple test structures when the selected calculation module is the object to be tested.

[0063] In some disclosed embodiments, the first substructure acquisition unit is specifically used to acquire at least a series structure and a parallel structure formed by several target layers as different first substructures in response to the expectation of acquiring the transmission time of at least one of the input layer and the output layer; wherein, in the parallel structure, each target layer is connected to a different output layer, and the input layer and the output layer are adjacent objects; in response to the expectation of acquiring the transmission time between the object under test and the reference object, a network layer adapted to the computing object of the reference object is selected as the reference layer, and different first substructures formed by the target layer being connected between different numbers of reference layers are acquired; wherein, the reference object is a computing module in the target NPU, the computing object of the reference object is different from the object under test, and the reference object is an adjacent object.

[0064] In some disclosed embodiments, the first construction submodule includes a second substructure acquisition unit for acquiring different second substructures containing the target NPU; the first construction submodule includes a second insertion unit for inserting different second substructures between the input layer and the output layer respectively to obtain multiple test structures when the target NPU is selected as the test object; wherein, each of the different second substructures includes a CPU connected between the multiple target NPUs, and the CPU is an adjacent object.

[0065] In some disclosed embodiments, the second substructure acquisition unit is specifically used to acquire a second substructure formed by the CPU connected between two target NPUs; and to acquire a second substructure formed by the CPU and more than two target NPUs in a star connection, wherein one target NPU is connected to the input layer and the other target NPUs are connected to the output layer.

[0066] In some disclosed embodiments, the solution module 54 includes an equation construction submodule, which is used to construct an objective equation with the processing time of the object under test and the transmission time between the object under test and adjacent objects as unknowns based on the measured time of the test structure; the solution module 54 includes an equation solving submodule, which is used to solve the objective equations constructed for each test structure to obtain the first time and the second time.

[0067] In some disclosed embodiments, the NPU testing device 50 further includes a network acquisition module for acquiring a first neural network and a second neural network; wherein the main time consumption of the first neural network is the data computation time, and the main time consumption of the second neural network is the data transmission time; the NPU testing device 50 further includes a result acquisition module for obtaining a first result characterizing the computing power of the target NPU based on the time consumption data of running the first neural network on the target NPU, and obtaining a second result of the memory bandwidth of the target NPU based on the time consumption data of running the second neural network on the target NPU.

[0068] Please see Figure 6 , Figure 6 This is a schematic diagram of a framework of an embodiment of the electronic device 60 of this application. The electronic device 60 includes a memory 61 and a processor 62. The memory 61 stores program instructions, and the processor 62 is used to execute the program instructions to implement the steps in any of the above-described NPU testing method embodiments. Specifically, the electronic device 60 may include, but is not limited to, mobile phones, tablet computers, laptops, learning machines, office laptops, servers, etc., and is not limited thereto.

[0069] Specifically, processor 62 controls itself and memory 61 to implement the steps in any of the NPU testing method embodiments described above. Processor 62 can also be referred to as a CPU (Central Processing Unit). Processor 62 may be an integrated circuit chip with signal processing capabilities. Processor 62 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 62 can be implemented using integrated circuit chips.

[0070] The above-described scheme, through the electronic device 60, on the one hand, constructs multiple test structures related to the object under test and obtains the measured time of each test structure operating independently, and uses the measured time to calculate the operating efficiency of the target NPU, thereby improving the accuracy of NPU testing. On the other hand, after obtaining the measured time of each test structure, it calculates the first time of data processing of the object under test itself, and simultaneously calculates the second time of data transmission between the object under test and its adjacent objects. This allows for the comprehensive calculation of key factors affecting NPU operating efficiency, contributing to improved detail in NPU testing. Therefore, it can accurately and meticulously test the operating efficiency of the NPU.

[0071] Please see Figure 7 , Figure 7 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium 70 of this application. The computer-readable storage medium 70 stores program instructions 71 that can be executed by a processor. The program instructions 71 are used to implement the steps in any of the above-described NPU testing method embodiments.

[0072] The above-described scheme, through the computer-readable storage medium 70, improves the accuracy of NPU testing by constructing multiple test structures related to the object under test and obtaining the measured execution time of each test structure independently, and then using the measured execution time to calculate the operating efficiency of the target NPU. Furthermore, after obtaining the measured execution time of each test structure, it calculates the first data processing time of the object under test itself, and simultaneously calculates the second data transmission time between the object under test and its adjacent objects. This allows for the comprehensive measurement of key factors affecting NPU operating efficiency, contributing to improved NPU testing detail. Therefore, it enables accurate and detailed testing of NPU operating efficiency.

[0073] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0074] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0075] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0076] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0077] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0078] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0079] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

Claims

1. An NPU testing method, characterized in that, include: Among the target NPU and several computing modules of the target NPU, select the object to be tested; Multiple test structures are constructed based on the input layer, the output layer, and the object under test. Obtain the measured execution time of each of the test structures running independently; Based on the measured time consumption of each of the test structures, the first time consumption for data processing of the object under test and the second time consumption for data transmission between the object under test and its adjacent objects in the test structure are obtained; wherein, based on the input layer, output layer, and the object under test, multiple test structures are constructed, including at least one of the following: In response to the fact that the object under test is the target NPU, a second substructure formed by the CPU connecting between two target NPUs is obtained, and a second substructure formed by the CPU and more than two target NPUs in a star connection is obtained. Different second substructures are inserted between the input layer and the output layer to obtain multiple test structures. In the star connection, one target NPU is connected to the input layer, and the other target NPUs are connected to the output layer. In response to the fact that the object under test is the computing module, a network layer adapted to the computing object of the computing module is selected as the target layer. If it is desired to obtain the transmission time of at least one of the input layer and the output layer, at least a series structure and a parallel structure formed by several target layers are obtained as different first substructures. In the parallel structure, each target layer is connected to a different output layer. If it is desired to obtain the transmission time between the object under test and the reference object, a network layer adapted to the computing object of the reference object is selected as the reference layer. Different first substructures formed by the target layer being connected between different numbers of reference layers are obtained. The reference object is a computing module in the target NPU whose computing object is different from that of the object under test. Different first substructures are inserted between the input layer and the output layer to obtain multiple test structures.

2. The method according to claim 1, characterized in that, The selection of a network layer that is compatible with the computational object of the computation module as the target layer includes at least one of the following: In response to the fact that the calculation module is an accumulation module, a convolutional layer is selected as the target layer; In response to the fact that the calculation module is a tensor processing module, any one of the normalized exponent layer, dimension renormalization layer, and summation layer is selected as the target layer; In response to the fact that the computation module is a non-linear module, the activation layer is selected as the target layer.

3. The method according to claim 1, characterized in that, When the series structure and the parallel structure are obtained as the different first substructures, the input layer and the output layer are the adjacent objects; In the case of obtaining different first substructures formed by the target layer connecting different numbers of reference layers, the reference object is used as the adjacent object.

4. The method according to claim 1, characterized in that, When the object under test is the target NPU, the CPU is the adjacent object.

5. The method according to claim 1, characterized in that, The measured time consumption based on each of the test structures is used to obtain the first time consumption for data processing of the object under test and the second time consumption for data transmission between the object under test and its adjacent objects in the test structure, including: Based on the measured time consumption of the test structure, an objective equation is constructed with the processing time of the object under test and the transmission time between the object under test and the adjacent object as unknowns. Solving the objective equations constructed based on each of the aforementioned test structures yields the first time consumption and the second time consumption.

6. The method according to claim 1, characterized in that, Obtain a first neural network and a second neural network; wherein the main time consumption of the first neural network is the data computation time, and the main time consumption of the second neural network is the data transmission time; Based on the time consumption data of the target NPU running the first neural network, a first result characterizing the computing power of the target NPU is obtained, and based on the time consumption data of the target NPU running the second neural network, a second result characterizing the memory bandwidth of the target NPU is obtained.

7. An NPU testing device, characterized in that, include: The selection module is used to select the object to be tested from the target NPU and several computing modules of the target NPU; A construction module is used to construct multiple test structures based on the input layer, the output layer, and the object under test; The acquisition module is used to acquire the measured time of each of the test structures running independently; The solution module is used to obtain, based on the measured time consumption of each of the test structures, the first time consumption of data processing of the object under test and the second time consumption of data transmission between the object under test and its adjacent objects in the test structure; wherein, the construction of multiple test structures based on the input layer, the output layer, and the object under test includes at least one of the following: In response to the fact that the object under test is the target NPU, a second substructure formed by the CPU connecting between two target NPUs is obtained, and a second substructure formed by the CPU and more than two target NPUs in a star connection is obtained. Different second substructures are inserted between the input layer and the output layer to obtain multiple test structures. In the star connection, one target NPU is connected to the input layer, and the other target NPUs are connected to the output layer. In response to the fact that the object under test is the computing module, a network layer adapted to the computing object of the computing module is selected as the target layer. If it is desired to obtain the transmission time of at least one of the input layer and the output layer, at least a series structure and a parallel structure formed by several target layers are obtained as different first substructures. In the parallel structure, each target layer is connected to a different output layer. If it is desired to obtain the transmission time between the object under test and the reference object, a network layer adapted to the computing object of the reference object is selected as the reference layer. Different first substructures formed by the target layer being connected between different numbers of reference layers are obtained. The reference object is a computing module in the target NPU whose computing object is different from that of the object under test. Different first substructures are inserted between the input layer and the output layer to obtain multiple test structures.

8. An electronic device, characterized in that, The device includes a memory and a processor that are coupled to each other, wherein the memory stores program instructions and the processor executes the program instructions to implement the NPU testing method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The device stores program instructions that can be executed by a processor, the program instructions being used to implement the NPU testing method according to any one of claims 1 to 6.

Images