Model precision comparison method and device, chip and computing equipment

Abstract

The invention discloses a model precision comparison method and device, a chip and computing equipment, and relates to the field of computers. The model precision comparison method is used for comparing the operation precision of a reference model and a target model, the reference model and the target model are the same model and run in different software environments or hardware computing power, or the network structures of the reference model and the target model are different. The method comprises the steps of running a reference model based on to-be-processed data, and obtaining and storing running data of network nodes marked in advance in the reference model; and running the target model based on the to-be-processed data, and obtaining and storing running data of the pre-marked network nodes in the target model. Comparing the operation data of the pre-marked network nodes with the mapping relation to obtain a comparison result; wherein the network nodes with the mapping relation comprise one network node in the reference model and one network node in the target model.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a model accuracy comparison method, device, chip and computing equipment. Background Art

[0002] The network model includes one or more computing units, which can also be called network nodes. The network nodes correspond to the computing processes in the network model. Exemplarily, the weight summation process in the convolution layer and the fully connected layer (FC layer) can be called the network nodes of the network model. When the network model is deployed to run in multiple different operating environments, there are differences in the calculation accuracy. It is necessary to compare the calculation accuracy of the network model in different operating environments to adjust the network model according to the calculation accuracy of the network model.

[0003] Usually, it is necessary to manually mark specific locations in the network model, obtain the operating data of the specific locations, and compare the operation accuracy of the network model in different operating environments based on the operating data of the specific locations. However, the speed of manually marking and obtaining operating data is slow, which leads to low efficiency in comparing the operation accuracy of the network model. Summary of the invention

[0004] The present application provides a model accuracy comparison method, device, chip and computing equipment, which solves the problem of low efficiency in the calculation accuracy comparison of network models in conventional technologies.

[0005] In a first aspect, a model accuracy comparison method is provided, which can be applied to a computer system, or to a host supporting a computer system to implement the model accuracy comparison method, for example, the host includes a chip. The model accuracy comparison method is used to compare the computational accuracy of a benchmark model and a target model, wherein the benchmark model and the target model are the same model running in different software environments or hardware computing power, or the network structures of the benchmark model and the target model are different. The model accuracy comparison method includes: running a benchmark model based on data to be processed, acquiring and saving the operation data of a pre-marked network node in the benchmark model. And, running a target model based on data to be processed, acquiring and saving the operation data of a pre-marked network node in the target model. Then, comparing the operation data of the pre-marked network nodes with a mapping relationship, and obtaining a comparison result. Among them, the pre-marked network nodes in the benchmark model are functional nodes in the benchmark model for implementing preset functions; the pre-marked network nodes in the target model are functional nodes in the target model for implementing preset functions; the network nodes with a mapping relationship include a network node in the benchmark model and a network node in the target model.

[0006] Compared to the method of manually marking and saving in the network model to obtain operation data, which leads to the problem of low efficiency in obtaining operation data, in the present application, the network nodes are marked to obtain the operation data of the marked network nodes, which improves the efficiency of obtaining operation data, thereby improving the comparison efficiency of the operation accuracy of the network model. And compared to the method of manually marking and saving in the network model, the operation data obtained is the process data or intermediate data in the network node. The network nodes pre-marked in the present application are functional nodes in the benchmark model for realizing preset functions. Therefore, the operation data obtained is the result data. In addition, the mapping relationship indicates the network nodes that need to be compared, avoiding the problem of only being able to compare the operation data at the dotted positions in the benchmark model and the dotted positions in the target model, resulting in a complicated comparison process, and then comparing the operation data of the network nodes indicated by the mapping relationship, which can further improve the comparison efficiency of the operation accuracy of the network model.

[0007] In a possible scenario, the pre-marked network nodes having a mapping relationship are pre-marked network nodes in a reference model and pre-marked network nodes in a target model having the same identifier.

[0008] Exemplarily, the identifier may be a name or number of a network node.

[0009] If the identifier is the name of a network node, the following is a possible example. The name of a network node can be the name or index combination of all its parent nodes. For example, name1.name2.name3.0.name4.1.

[0010] In this application, the names of the above network nodes are used as identifiers to ensure that the positions and functions of the two compared network nodes in the model are the same, thereby ensuring the accuracy of the match. In addition, directly comparing two network nodes with the same identifier can increase the speed of determining the compared network nodes, thereby improving the efficiency of the entire comparison process.

[0011] In yet another possible scenario, the comparison result is the similarity and / or error between the operation data of the network nodes having a mapping relationship.

[0012] For example, the similarity may be cosine similarity, Euclidean metric, Pearson correlation coefficient, etc.

[0013] The error can be absolute error, mean square error, root mean square error, etc.

[0014] In a possible implementation, the model accuracy comparison method further includes: registering the network nodes pre-marked in the reference model in a first callback function, the first callback function being used to obtain and automatically save the operation data of the network nodes pre-marked in the reference model during the forward calculation and reverse calculation of the reference model. And, registering the network nodes pre-marked in the target model in a second callback function, the second callback function being used to obtain and automatically save the operation data of the network nodes pre-marked in the target model during the forward calculation and reverse calculation of the target model.

[0015] In the present application, by registering the pre-marked network nodes in the first callback function, the manual marking and saving method for obtaining the operation data is avoided, which improves the speed of obtaining the operation data and further improves the efficiency of comparing the calculation accuracy of the network model.

[0016] Exemplarily, the above registration of the network node in the callback function means inserting the hook function into the forward calculation and reverse calculation code of the network model, and through the hook function, the network node in the network model can obtain the operation data of the network node when the forward calculation or reverse calculation code is executed.

[0017] In yet another possible scenario, the operation data of the network node includes input sub-data and / or output sub-data of the network node during forward calculation, and input sub-data and / or output sub-data of the network node during reverse calculation.

[0018] In a possible implementation, the model accuracy comparison method further includes: outputting a network topology diagram of the target model, wherein the nodes in the network topology diagram are used to indicate the identifiers of the network nodes included in the target model, and the comparison results corresponding to the network nodes included in the target model.

[0019] In the present application, the host outputs a network topology map of the target model. Based on the visualized network topology map, the user can quickly identify abnormal network nodes with abnormal comparison results according to the identification and comparison results of the network nodes in the network topology map, thereby improving the efficiency of locating abnormal network nodes.

[0020] In a possible implementation, the model accuracy comparison method further includes: synchronizing the running speeds of the benchmark model and the target model.

[0021] In this application, the running speed of the benchmark model and the target model are synchronized to achieve the comparison of the running data generated by this iteration after both the benchmark model and the target model complete one iteration. This avoids the need to store a large amount of running data and occupy more storage space after completing a round of training using the data to be processed. Thus, the storage space occupied is reduced.

[0022] For example, the next iteration is performed only after both the baseline model and the target model complete an iteration based on the data to be processed.

[0023] In a possible implementation, the model accuracy comparison method further includes: deleting the saved operation data of the network nodes pre-marked in the reference model and the operation data of the network nodes pre-marked in the target model.

[0024] In this application, after the host completes an iteration and compares each pair of the benchmark model and the target model, the operation data generated in this iteration is deleted. In this way, the problem of the host needing to store a large amount of operation data is avoided, the storage space occupied by the operation data is reduced, and no additional storage space is required to store a whole round of operation data, thereby improving the storage space utilization rate.

[0025] In a possible implementation, a placeholder node is provided in the pre-marked network nodes whose comparison results are less than or equal to a threshold, and operation data of the placeholder node is used to compare the computational accuracy of the benchmark model and the target model.

[0026] For example, a placeholder node is further provided in an abnormal network node in the reference model (a pre-marked network node whose comparison result is less than or equal to a threshold value), and a placeholder node is further provided in an abnormal network node in the target model, and the abnormal network node is a pre-marked network node whose comparison result is less than or equal to the threshold value. The placeholder node provided in the abnormal network node in the reference model is used to obtain the first operation data, and the placeholder node provided in the abnormal network node in the target model is used for the second operation data.

[0027] In the present application, a placeholder node is set in a pre-marked network node (abnormal network node) whose comparison result is less than or equal to a threshold value to obtain the operating data at the placeholder node. Since the placeholder node is located within the network node, the scope of abnormal determination can be narrowed, thereby improving the accuracy of the precision comparison of the network model.

[0028] In a second aspect, a model accuracy comparison device is provided. The model accuracy comparison device is applied to a host, and the model accuracy comparison device includes various modules for executing the model accuracy comparison method in the first aspect or any optional design of the first aspect. The model accuracy comparison device is used to compare the computational accuracy of a reference model and a target model, wherein the reference model and the target model are the same model running in different software environments or hardware computing power, or the network structures of the reference model and the target model are different. For example, the model accuracy comparison device includes: a first acquisition module, a second acquisition module and a comparison module. Among them, the first acquisition module is used to run the reference model based on the data to be processed, and obtain and save the operation data of the network nodes pre-marked in the reference model. Among them, the network nodes pre-marked in the reference model are functional nodes in the reference model for implementing preset functions. The second acquisition module is used to run the target model based on the data to be processed, and obtain and save the operation data of the network nodes pre-marked in the target model. Among them, the network nodes pre-marked in the target model are functional nodes in the target model for implementing preset functions. The comparison module is used to compare the operation data of the pre-marked network nodes with a mapping relationship to obtain a comparison result. The network nodes having a mapping relationship include a network node in the benchmark model and a network node in the target model.

[0029] In a possible scenario, the pre-marked network nodes having a mapping relationship are pre-marked network nodes in a reference model and pre-marked network nodes in a target model having the same identifier.

[0030] In yet another possible scenario, the comparison result is the similarity and / or error between the operation data of the network nodes having a mapping relationship.

[0031] In a possible implementation, the model accuracy comparison device further includes a registration module. The registration module is used to register the network nodes pre-marked in the reference model in a first callback function, and the first callback function is used to obtain and automatically save the operation data of the network nodes pre-marked in the reference model during the forward calculation and reverse calculation of the reference model. And, the network nodes pre-marked in the target model are registered in a second callback function, and the second callback function is used to obtain and automatically save the operation data of the network nodes pre-marked in the target model during the forward calculation and reverse calculation of the target model.

[0032] In a possible implementation, the model accuracy comparison device further includes: an output module. The output module is used to output a network topology diagram of the target model. The nodes in the network topology diagram are used to indicate the identifiers of the network nodes included in the target model and the comparison results corresponding to the network nodes included in the target model.

[0033] In a possible implementation, the model accuracy comparison device further includes: a synchronization module. The synchronization module is used to synchronize the running speeds of the benchmark model and the target model.

[0034] In a possible implementation, the model accuracy comparison device further includes: a deletion module. The deletion module is used to delete the saved operation data of the network nodes pre-marked in the reference model and the operation data of the network nodes pre-marked in the target model.

[0035] In a possible implementation, the operation data of the network node includes input sub-data and / or output sub-data of the network node during forward calculation, and input sub-data and / or output sub-data of the network node during reverse calculation.

[0036] In a possible implementation, a placeholder node is provided in the pre-marked network nodes whose comparison results are less than or equal to a threshold, and operation data of the placeholder node is used to compare the computational accuracy of the benchmark model and the target model.

[0037] In a third aspect, a chip is provided, comprising a processor and a power supply circuit, the power supply circuit being used to supply power to the processor; the processor being used to call and run computer instructions to execute the method in the above-mentioned first aspect and any possible implementation of the first aspect.

[0038] In a fourth aspect, a computing device is provided, comprising a processor and a memory; the memory stores instructions, and the processor calls the instructions stored in the memory to implement the method in the above-mentioned first aspect and any possible implementation method of the first aspect.

[0039] In a fifth aspect, a computer-readable storage medium is provided, in which a computer program or instruction is stored. When the computer program or instruction is executed by a processing device, the method in the above-mentioned first aspect and any possible implementation method of the first aspect is implemented.

[0040] In a sixth aspect, a computer program product is provided, which includes a computer program or instructions. When the computer program or instructions are run on a processing device, the processing device executes the computer program or instructions to implement the method in the above-mentioned first aspect and any possible implementation method of the first aspect.

[0041] The beneficial effects of the second to sixth aspects above can be referred to the description of the first aspect or any implementation method of the first aspect, and will not be elaborated here.

[0042] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flowchart of a precision alignment method;

[0044] Figure 2 An application scenario diagram of a computer system provided for this application;

[0045] Figure 3 A schematic diagram of a flow chart of a model accuracy comparison method provided in this application;

[0046] Figure 4 A benchmark model diagram provided for this application;

[0047] Figure 5 The network topology diagram provided for this application;

[0048] Figure 6 A flowchart of the model accuracy comparison method deployed in different frameworks provided in this application;

[0049] Figure 7 Schematic diagram of the structure of the model accuracy comparison device provided in this application Figure 1 ;

[0050] Figure 8 Schematic diagram of the structure of the model accuracy comparison device provided in this application Figure 2 ;

[0051] Fig. 9 A schematic diagram of the structure of a processing device provided in this application. DETAILED DESCRIPTION

[0052] First, some technical terms involved in this application are explained:

[0053] Artificial intelligence (AI) framework is a collection of software tools for developing and deploying network models. The AI ​​framework provides various functions and interfaces that enable developers to build, train, and apply network models. The following are some common AI frameworks: pytorch, tensorflow, mindspore, paddlepaddle, oneflow, jax, etc.

[0054] The central processing unit (CPU) is a hardware device used to execute computer program instructions. The CPU is used to execute various computer program instructions to implement network models, etc.

[0055] Graphics processing unit (GPU) is a processor specially used for graphics computing and parallel computing. Compared with traditional CPU, GPU has more computing cores and higher storage bandwidth, and can process large-scale data and complex computing tasks in parallel. It is often used to run network models, etc.

[0056] Embedded neural network processors (neural-network process units, NPU) are processors specifically designed to accelerate neural network computing. Unlike CPUs and GPUs, NPUs are designed specifically to perform computing tasks of network models, with higher parallelism and computing efficiency.

[0057] Forward computing refers to the process of transferring and processing input data through the network nodes of each layer of the network model to finally obtain the output result. In forward computing, starting from the input of the network model, calculation and conversion are performed layer by layer, and the output of the previous layer is used as the input of the next layer. Each layer performs specific operations, such as matrix multiplication, nonlinear activation function, etc., to convert the input data into a more abstract or advanced representation. Finally, it is processed by the output layer of the network model to produce the final result.

[0058] Reverse calculation refers to updating the model parameters by back-propagating the error according to the difference between the output of the network model and the true label to optimize the performance of the network model. In reverse calculation, the gradient of the loss function of the network model with respect to the output of the network model is first calculated, and then the gradient is propagated from the output layer to the input layer. The gradient propagation process uses the chain rule and reverse calculation to calculate the gradient of each layer with respect to the input of the previous layer. Finally, the parameters of the model are updated according to the gradient calculation rules so that the value of the loss function gradually decreases.

[0059] Placeholder nodes are usually used to represent placeholders for input data or output data. Especially during training, when the actual data is not yet ready or the dimensions and number of rows of the running data cannot be determined in advance, placeholder nodes can be used to represent these data. In this application, placeholder nodes are used to add comparison points in the network model, and then obtain the running data at the comparison points for comparison.

[0060] A callback function is a function that is called when a specific event occurs or a specific condition is met. For example, when the corresponding network node is executed in this application, the callback function is called to obtain the operation data of the network node. In this application, the callback function can be a hook function. For example, when the network node receives data, outputs data, or performs a specific calculation process, the hook function is triggered, and the operation data of the network node is obtained through the hook function.

[0061] AI is being popularized in various industries, such as computer vision, natural language processing, speech semantics, multimodality and other fields. With the current diversification of AI frameworks and backend hardware (CPU, GPU, NPU, etc.), network models are switching to different AI frameworks or backend hardware operations more and more frequently. Due to differences in the application programming interface (API), syntax support, and operation mode of each AI framework, and the AI ​​frameworks supported by each backend hardware are also different. Therefore, after switching the AI ​​framework running the network model or switching the backend hardware, there is often a problem of inconsistent calculation accuracy of the network model before and after the switch.

[0062] Taking AI framework switching as an example, in order to solve the problem of inconsistent calculation accuracy of the network model before and after the AI ​​framework switching, it is determined whether the running data obtained by the network model running in the AI ​​framework after the switching is different from the running data obtained by the network model running in the AI ​​framework before the switching. If there is a large difference in the running data of the network nodes in the network model before and after the switching, the network nodes with large running data differences (abnormal network nodes) are located in the network model, so that the user can adjust the network nodes to ensure that the calculation accuracy of the network model is consistent before and after the AI ​​framework switching.

[0063] With respect to the above-mentioned determination of whether the operating data obtained when the network model runs in the AI ​​framework after switching is different from the operating data obtained when the network model runs in the AI ​​framework before switching, a possible implementation solution is provided below.

[0064] First, in the forward calculation process of the network model, the operation data in the network model can be obtained by manually marking and saving at specific locations in the network model. In the process of running the network model for multiple rounds, the binary method is used to continuously manually mark and save in the network model to narrow the scope of determining abnormal network nodes.

[0065] Secondly, because the AI ​​framework generates a reverse calculation graph based on the forward calculation code of the network model, it is impossible to use the binary method to locate abnormal network nodes in the network model during the reverse calculation phase. The common method of locating abnormal network nodes in the network model during the reverse calculation phase is to determine the abnormal network nodes based on the gradient of the parameters of each network node included in the network model.

[0066] In a possible example, the AI ​​framework switching scenario of running a network model is used as an example to illustrate the operation data obtained by running the network model in the AI ​​framework before the switching and the operation data obtained by running the network model in the AI ​​framework after the switching. Figure 1 As shown, Figure 1The figure is a flowchart of a precision alignment method.

[0067] ①: Configure network model a1 in framework a and configure network model a2 in framework b.

[0068] Exemplarily, the network model a1 is consistent with the network model a2, and the network model a1 can be used as a baseline model, and the network model a2 can be used as a model to be determined. Framework a and framework b are different AI frameworks, such as framework a is pytorch and framework b is mindSpore.

[0069] In a possible example, an optimizer may be configured for the network model a1 and the network model a2, so that the network model a1 and the network model a2 can gradually converge to a better state.

[0070] Furthermore, the network model a1 and the network model a2 are manually marked and saved to obtain the operation data of the network model a1 and the network model a2 at the marked positions.

[0071] ②: Initialize network model a1 and network model a2 to make the parameters of each network node in network model a1 and network model a2 consistent.

[0072] ③: Input data a into network model a1 and network model a2 respectively. After completing a round of training, obtain the operating data at each dot position in network model a1 and network model a2. Compare the operating data of network model a1 with the operating data of network model a2 at the corresponding dot position to obtain the comparison result of the network model at the aforementioned position.

[0073] For example, since the position of the dotted position in the network model is not fixed, but in the network node, the acquired operation data is the intermediate data of a calculation process in the network node. During the training process, the operation data at all dotted positions will be dumped to the disk. After a round of training is completed, the operation data is obtained from the disk for comparison.

[0074] ④: Based on the comparison results of the network models, the binary method is used to manually mark the network models a1 and a2 to obtain the marked network models a1 and a2.

[0075] Among them, in the present application, manually marking in the network model by using the binary method may include: if the comparison result at a marking position in the network model is abnormal, the marking position (or the abnormal marking position) is located in the network model, and the next marking position adjacent to the marking position is marked at a position between the positions in the network model.

[0076] It is worth noting that manual marking is performed at the same position in the network model a1 and the network model a2.

[0077] ⑤: Input the data b into the dotted network model a1 and the dotted network model a2 respectively. After completing a round of training, the operating data at each dotted position in the network model a1 and the network model a2 are obtained, and the operating data of the network model a1 is compared with the operating data of the network model a2 at the corresponding position.

[0078] Repeat the above steps ④ and ⑤ until the abnormal network node or abnormal computing means in the network model a2 is located. The abnormal computing means is a processing process of non-neural network processing in the network model, such as the processing process of selecting a data from multiple data in the network model.

[0079] It is worth noting that the above steps ① to ⑤ are for locating abnormal network nodes or abnormal computing methods during the forward calculation of the network model.

[0080] ⑥: Use the comparison results of the obtained network models to locate abnormal network nodes. If the similarity between the operation data of a network node in network model a1 and the operation data of the corresponding network node in network model a2 is less than the threshold (80%), the network node in network model a2 is determined to be an abnormal network node.

[0081] ⑦: Adjust the abnormal network node, such as adjusting the code corresponding to the abnormal network node or abnormal computing method.

[0082] Then, the network model a2 and the network model a1 after adjusting the abnormal network nodes are trained again to repeat the above step ③ and obtain the comparison result again. If the abnormal network nodes are still located, repeat steps ⑦ and ③. If there are still abnormal network nodes after several iterations, execute step ⑧.

[0083] In order to ensure the accuracy alignment of network models running in different frameworks, it is also necessary to determine the abnormal network nodes or computing methods of the network model during reverse calculation. For example, in the forward calculation stage, after the abnormal network nodes are located through the above steps ① to ⑦, in the reverse calculation stage, the above accuracy alignment method also includes the following steps ⑧ and ⑨.

[0084] ⑧: Obtain the gradient information 1 of the abnormal network node in the network model a1, and the gradient information 2 of the abnormal network node in the network model a2. The user compares the gradient 1 of parameter a in the gradient information 1 with the gradient 2 of parameter a in the gradient information 2, so as to determine the abnormal network nodes whose gradient difference is greater than the threshold. The gradient information includes the gradients of all parameters in the network model.

[0085] ⑨: The user adjusts the abnormal network node or abnormal computing means, such as adjusting the code corresponding to the abnormal network node or abnormal computing means.

[0086] The network model a2 and the network model a1 after the abnormal network nodes or abnormal computing means are adjusted are trained again to determine whether there are still abnormalities in the network nodes or computing means in the network model a2 and the network model a1 in the reverse calculation stage. If there are still abnormal network nodes or abnormal computing means in the network model a2, the abnormal network nodes or abnormal computing means are adjusted again, and the above ⑧ and ⑨ are repeated until the running data and gradients of the network nodes or computing means in the network model a2 and the network model a1 are consistent or the similarity is greater than a threshold (such as 80%), so that the alignment is completed.

[0087] In one possible scenario, after the adjustment, the above steps ① to ⑧ may be repeated until the computational precision of the network model a1 is aligned with that of the network model a2.

[0088] In the above-mentioned determination of abnormal network nodes in the network model in the forward calculation stage, manual dotting and saving are required to obtain the operation data of the network model at the dotting position. When the structure of the network model is complex, in order to obtain sufficient operation data for calculation accuracy comparison, a large number of mechanical dotting operations need to be performed manually in the network model, and the dotting position depends on experience, resulting in a slow speed of obtaining operation data, resulting in low efficiency of the calculation accuracy comparison of the network model. Due to the slow speed of obtaining operation data, the consumption of computing power resources and human resources are huge. In addition, in the reverse calculation stage of the network model, it is impossible to obtain operation data by manual dotting and saving, and positioning can only be performed by manually observing the gradient, further reducing the efficiency of comparing the calculation accuracy of the network model. The user manually dotted in the network model, and the dot position is often in the network node when the manual dot is saved, resulting in the operation data obtained at the dot position being the process data or intermediate data of the network node.

[0089] To solve the problem of low efficiency in the comparison of the computing accuracy of the above-mentioned network model. The present application provides a model accuracy comparison method, which is used to compare the computing accuracy of a benchmark model and a target model. The benchmark model and the target model are the same model running in different software environments or hardware computing power, or the network structures of the benchmark model and the target model are different. The model accuracy comparison method includes: running the benchmark model based on the data to be processed, acquiring and saving the operating data of the pre-marked network nodes in the benchmark model. And, running the target model based on the data to be processed, acquiring and saving the operating data of the pre-marked network nodes in the target model. Then, the operating data of the pre-marked network nodes with a mapping relationship are compared to obtain a comparison result. Among them, the pre-marked network nodes in the benchmark model are functional nodes in the benchmark model for realizing preset functions; the pre-marked network nodes in the target model are functional nodes in the benchmark model for realizing preset functions; the network nodes with a mapping relationship include a network node in the benchmark model and a network node in the target model.

[0090] Compared to the method of manually marking and saving in the network model to obtain operation data, which leads to the problem of low efficiency in obtaining operation data, in the present application, the network nodes are marked to obtain the operation data of the marked network nodes, which improves the efficiency of obtaining operation data, thereby improving the comparison efficiency of the operation accuracy of the network model. And compared to the method of manually marking and saving in the network model, the operation data obtained is the process data or intermediate data in the network node. The network nodes pre-marked in the present application are functional nodes in the benchmark model for realizing preset functions. Therefore, the operation data obtained is the result data. In addition, the mapping relationship indicates the network nodes that need to be compared, avoiding the problem of only being able to compare the operation data at the dotted positions in the benchmark model and the dotted positions in the target model, resulting in a complicated comparison process, and then comparing the operation data of the network nodes indicated by the mapping relationship, which can further improve the comparison efficiency of the operation accuracy of the network model.

[0091] Next, the model accuracy comparison method provided in this application is described in detail with reference to the accompanying drawings.

[0092] like Figure 2 As shown, Figure 2 This is an application scenario diagram of a computer system provided in the present application. The computer system may include a computing device 210 , a computing device 220 , and a computing device 230 , and the computing devices are connected via a channel 240 .

[0093] The computing device 210, the computing device 220 or the computing device 230 may include one or more processing units, for example, the computing device may include a CPU, an application processor (AP), a modem processor, a GPU, an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a discrete gate, a transistor logic device, a discrete hardware component, a baseband processor, and / or an NPU, etc. Or it may be a processing device including any one or more of the above processing units, such as a terminal, a server, etc.

[0094] For example, the computing device 210 may include an NPU, the computing device 220 may include a GPU, and the computing device 230 may include a CPU.

[0095] The channel 240 is used to transmit information between the computing devices. In addition to the data bus, the channel 240 may also include a power bus, a control bus, and a status signal bus. However, for the sake of clarity, various buses are labeled as channels 240 in the figure.

[0096] In one possible scenario, channel 240 may be a Peripheral Component Interconnect Express (PCIe) high-speed bus, or an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL), a Cache Coherent Interconnect for Accelerators (CCIX) and other buses. For example, computing device 210 may access computing device 220 via a PCIe bus. Computing device 210 may be connected to computing device 220 via a double data rate (DDR) bus. Here, different computing devices 220 may use different data buses to communicate with computing device 210, so the DDR bus may also be replaced with other types of data buses, and the bus type is not limited in the embodiment of the present application. It should be understood that multiple computing devices can be mounted on a host or server and communicate using a hardware communication channel, and multiple computing devices transmit data through memory or temporary files, which reduces the transmission delay of data or instructions and shortens the accuracy comparison time of the model.

[0097] In another possible scenario, Figure 2 The channel 240 shown may be an RPC channel, and multiple computing devices are connected through the RPC channel. Exemplarily, the computing device 210 may call objects (such as data, etc.) on the remote computing device 220 or computing device 230 based on the RPC channel without knowing the specific call details. For example, the computing device 210 instructs the computing device 220 or computing device 230 to run the model according to the data to be processed based on the RPC channel, and reads the running data of the operator on the computing device 220 or computing device 230 to the computing device 210. The bandwidth occupied when forwarding the running data and the storage resources occupied when storing the running data in the conventional technology are avoided, thereby improving the efficiency of data transmission between multiple computing devices, thereby shortening the operation accuracy comparison time of the network model.

[0098] For example, computing device 210 remotely calls the RPC interface of computing device 220 through a remote communication tool (such as Run-time Library) to establish an RPC channel connection between computing device 210 and computing device 220.

[0099] The computing device 210 remotely calls the RPC interface of the computing device 230 through the Run-time Library, so that the computing device 210 establishes an RPC channel connection with the computing device 230 .

[0100] It should be noted that Figure 2 This is just an example provided in this application. The computer system may also include other computing devices, or storage devices, power supplies, hard disks, optical drives, chassis, cooling systems, and other input / output controllers and interfaces that support the operation of computing devices. Figure 2 The embodiments of the present application do not limit the form and quantity of the other computing devices or the hardware supporting the operation of the computing devices.

[0101] Exemplarily, the computing device 210, the computing device 220, and the computing device 230 are deployed with the same or different software environments, which include AI frameworks, acceleration libraries, third-party libraries, etc. For example, pytorch is deployed on the computing device 210, mindspore is deployed on the computing device 220, and tensorflow is deployed on the computing device 230; or, pytorch is deployed on the computing device 210, the computing device 220, and the computing device 230. The AI ​​frameworks deployed on the above computing devices are only examples, and this application does not limit the AI ​​frameworks deployed on each computing device.

[0102] On the one hand, the present application provides a model accuracy comparison method, which can be applied to Figure 2 The computer system shown in FIG. Figure 3 As shown, Figure 3 A flow chart of a model accuracy comparison method provided in the present application, the model accuracy comparison method can be executed by a host, the host may include Figure 2 One or more computing devices in the illustrated computer system. Figure 3 The method shown is used to compare the computational accuracy of the benchmark model and the target model. Figure 3 As shown, the model accuracy comparison method may include the following steps S310 to S330.

[0103] S310: The host runs a benchmark model based on the data to be processed, and obtains and saves the running data of the network nodes pre-marked in the benchmark model.

[0104] In the present application, the network nodes pre-marked in the benchmark model are functional nodes in the benchmark model for implementing preset functions. Such as a convolutional layer for implementing convolution, a pooling layer for implementing feature dimension reduction or overfitting, etc. For ease of description, the network nodes pre-marked in the benchmark model may be referred to as first network nodes, and the operating data of the network nodes pre-marked in the benchmark model may be referred to as first operating data. The first network node represents the network node declared in init of the benchmark model, and the first network node has been registered in the network node of the callback function. For the content of the first network node identifier of the network node registered in the callback function, the content of the first operating data of the first network node in the benchmark model can be obtained by referring to the host described below, which will not be repeated here.

[0105] It is worth noting that all network nodes declared in the init of the benchmark model can be marked, or some network nodes declared in init can be marked, and this application is not limited. For example, the benchmark model includes convolution layer 1, convolution layer 2, and pooling layer 1, and convolution layer 1, convolution layer 2, and pooling layer 1 can be marked, or convolution layer 1 and pooling layer 1 can be marked. The running data of the marked network nodes can be used to compare the calculation accuracy of the benchmark model and the target model.

[0106] In this application, the network structure of the benchmark model is constructed by the user to solve problems in the fields of computer vision, natural language processing, speech semantics, multimodality, etc. This application does not limit the network structure of the benchmark model. For example, it can be a transformer model in the field of natural language processing, or a generative adversarial network (GAN) in the field of computer vision. The above-mentioned data to be processed varies according to the type of the benchmark model (computer vision, natural language processing, speech semantics, multimodality, etc.). Exemplarily, if the benchmark model is of computer vision type, the data to be processed is image data and labels corresponding to the image data, which are used to indicate the type, bounding box, key points, etc. of the image data. If the benchmark model is of natural language processing type, the data to be processed is text data and labels corresponding to the text data, which are used to indicate the type, features, etc. of the text data.

[0107] The host trains, tests or processes the benchmark model based on the data to be processed to obtain the operation data of the first network node in the benchmark model.

[0108] Exemplarily, when the benchmark model is trained or tested, the process is not limited to continuously inputting the data to be processed into the benchmark model in multiple times, so that the benchmark model continuously iterates and runs. The host can also only perform one round of training or testing on the benchmark model, that is, input a set of data into the benchmark model for processing to obtain the first running data of the benchmark model. The first running data is the data generated by the first network node in the benchmark model during the running of the benchmark model. Exemplarily, the first running data can be the input data and / or output data of the first network node in the benchmark model.

[0109] For example, the first operating data includes input sub-data and / or output sub-data obtained by the first network node during a forward calculation process of the network model, and input sub-data and / or output sub-data obtained by the first network node during a reverse calculation process of the network model.

[0110] like Figure 3 As shown, the benchmark model includes network node 111, network node 112, ..., network node 1mn, and the host obtains the operation data of network node 111, network node 112, ..., network node 1mn respectively. Wherein, m and n are constants.

[0111] Exemplarily, the labels in the network node 111 can represent that the first 1 represents a network node belonging to the baseline model, and the second and third 1s combined represent the first network node in the first layer in the baseline model.

[0112] If the network node is numbered 1112, the first 1 represents a network node belonging to the reference model, and the following 112 combination represents the first network node in the first sublayer under the first layer.

[0113] It is worth noting that the benchmark model may include more layers. For the labels of the benchmark models with more layers, such as name1.name2.name3.0.name4.1, please refer to the above examples and will not be elaborated here.

[0114] In a possible scenario, the first operating data includes first input sub-data and / or first output sub-data obtained by the first network node in a forward calculation process of the network model.

[0115] The first input sub-data and the first output sub-data of the first network node can be represented by vectors. Figure 4 As shown, Figure 4 A benchmark model diagram provided for this application. Figure 4The output sub-data of the pooling layer 1 is obtained by post-processing the output sub-data of the convolutional layer 1 as the input data. The input sub-data of the pooling layer 1 is (1, 1, 1, 3, 4, 6), and the output sub-data is (2, 3, 7, 3, 5, 6). The first input sub-data and the first output sub-data are only examples and do not constitute a limitation of the present application. The input sub-data and the output sub-data can be vectors of dozens or hundreds of dimensions.

[0116] The first operation data also includes: second input sub-data and / or second output sub-data obtained by the first network node during the reverse calculation process of the network model.

[0117] The host inputs the data to be processed Figure 4 After the network model shown is constructed, the output result is obtained, and the output result is compared with the label corresponding to the data to be processed. Then, the host uses the result of the difference comparison to perform reverse calculation (backward propagation) (activation function layer 1→full connection layer 1→pooling layer 1→convolution layer 1) to calculate the second input sub-data and second output sub-data of each network node.

[0118] The host may obtain the content of the first operation data of the first network node in the benchmark model by means of any one or more of the following embodiments.

[0119] In a first possible implementation example, the host obtains first operating data of the first network node through a callback function.

[0120] In one possible implementation, the host obtaining the first operating data of the first network node through a callback function may include: the host registers the first network node included in the benchmark model in the first callback function of forward calculation and reverse calculation, and the host uses the callback function to obtain and automatically save the first operating data of the first network node.

[0121] Exemplarily, the above-mentioned first callback function for registering the first network node included in the benchmark model in the forward calculation and the reverse calculation includes: the host inserts a hook function into the code of the benchmark model in the forward calculation and the code of the reverse calculation, and uses the hook function in the code of the network model during the forward calculation and the hook function in the code during the reverse calculation to trigger the hook function when the network node receives data or outputs data, thereby obtaining the first running data of the first network node.

[0122] Thus, in the first possible embodiment, the host uses a callback function to obtain the first operating data of the first network node, avoiding the need to manually perform manual dotting and saving in the benchmark model to obtain the operating data at the dotting position, which results in the problem of slow acquisition of the operating data, thereby improving the acquisition speed of the operating data and thus improving the efficiency of model comparison based on the operating data. In addition, it avoids the manual mechanical dotting and saving in the network model to obtain the operating data, improves the acquisition speed of the operating data, and reduces the labor cost and computing resource cost caused by manual dotting and saving.

[0123] Since the first network node is the network node declared in init, the host registers the first network node in the first callback function, and when executing the first network node, obtains the operation data of the first network node, which is the operation data of the entire first network node, that is, input data and output data (result data). Compared with manually dotting and saving in the network model, the dotted position is often inserted into the network node, and the operation data obtained at the dotted position is intermediate data or process data.

[0124] In a second possible embodiment, the host obtains the first operating data by deploying a placeholder node in a benchmark model.

[0125] For example: the host can set the network node in the benchmark model as a placeholder node, and then obtain the first operating data through the placeholder node. Different from the manual dot-marking and saving method, the placeholder node in the second possible embodiment is set at the granularity of the network node to obtain the full amount of operating data, rather than setting it at the granularity of a certain computing node.

[0126] Exemplarily, the placeholder node does not process the acquired data and directly passes it to the next node. In other words, the placeholder node receives the output data of the previous node and passes the output data to the next node. The placeholder node is an identity node.

[0127] Further optionally, after the host obtains the first operating data, the host stores the first operating data in a memory of the host.

[0128] Exemplarily, the host stores the first operating data in a base temporary directory in the memory, and the first operating data carries the identifier of the first network node. For example, a subdirectory named the identifier of the first network node can be created under the base temporary directory, and the first operating data is stored in the subdirectory named the identifier of the first network node.

[0129] S320: The host runs the target model based on the data to be processed, and obtains and saves the operation data of the network nodes pre-marked in the target model.

[0130] In the present application, the network node pre-marked in the target model is a functional node in the target model for implementing a preset function. For ease of description, the operating data of the network node pre-marked in the target model may be referred to as the second network node. The operating data of the network node pre-marked in the target model may be referred to as the second operating data. The second network node represents the network node declared in the init of the target model, and the second network node has been registered in the network node of the callback function. For the content of the second network node identifier of the network node registered in the callback function, the content of the second operating data of the second network node in the target model can be obtained by referring to the following host, which will not be repeated here.

[0131] It should be understood that all network nodes declared in the init of the target model can be marked, or some network nodes declared in init can be marked, without limitation. For example, if the target model includes convolution layer 1, convolution layer 2 and pooling layer 1, convolution layer 1, convolution layer 2 and pooling layer 1 can be marked, or convolution layer 1 and pooling layer 1 can be marked. The running data of the marked network nodes can be used to compare the calculation accuracy of the target model and the target model.

[0132] In a possible scenario, the benchmark model and the target model are the same model running in different running environments.

[0133] Example 1: The benchmark model and the target model run in different software environments, which may include an AI framework, an acceleration library, or a third-party library.

[0134] For example, the benchmark model runs on the pytorch framework, and the target model runs on the mindspore framework. The foregoing content is only an example and should not be understood as a limitation of the present application. In other examples of the present application, the benchmark model may also run on the tensorflow architecture, and the target model may run on the pytorch framework.

[0135] Example 2: The baseline model and target model run on different hardware computing power.

[0136] For example, the benchmark model runs on a GPU, and the target model runs on an NPU. The foregoing content is only an example and should not be understood as a limitation of the present application. In other examples of the present application, the benchmark model may also run on a CPU, and the target model may run on a GPU, etc.

[0137] In another possible scenario, the network structures of the baseline model and the target model are different.

[0138] For example, the code of the target model is changed compared to the code of the benchmark model. Then the host compares the accuracy of the target model and the benchmark model to determine whether the code change optimizes the model. The benchmark model and the target model will run in the same software environment and hardware computing power to ensure a single variable.

[0139] Regarding the host running the target model based on the data to be processed to obtain the content of the second operating data of the second network node in the target model, reference can be made to the above S310 in which the host runs the benchmark model based on the data to be processed to obtain the first operating data of the first network node in the benchmark model, which will not be elaborated here.

[0140] like Figure 3 As shown, the target model includes network node 211, network node 212, ..., network node 2mn, and the host obtains operation data corresponding to network node 211, network node 212, ..., network node 2mn respectively.

[0141] Exemplarily, the numbers in the network node 211 can represent respectively that the first 2 represents a network node belonging to the target model, and the second and third 1s combined represent the first network node in the first layer in the target model.

[0142] For more information about the meaning of the labels, please refer to the description of the network node labels in S310 above, which will not be repeated here.

[0143] In a possible implementation, the host registers the second network node included in the target model in the second callback function of the forward calculation and the reverse calculation, and the host uses the second callback function to obtain and save the second operation data of the second network node.

[0144] In another possible implementation, the host obtains the second operation data by deploying a placeholder node in the target model. For example, the host can set the network node in the target model as a placeholder node, and then obtain the first operation data through the placeholder node. Different from the manual dot-marking and saving method, the placeholder node in the second possible embodiment is set at the granularity of the network node to obtain the full amount of operation data, rather than at the granularity of a certain computing node.

[0145] The content of obtaining the second operation data may refer to the description of obtaining the first operation data in S310 , and will not be described in detail here.

[0146] In the present application, the host uses a callback function to obtain the operating data of the network nodes included in the target model, avoiding the need to manually deploy placeholder nodes in the target model to obtain the operating data at the placeholder nodes, resulting in slow acquisition and reading of the operating data, thereby improving the efficiency of model comparison based on the operating data.

[0147] In a possible implementation, the second operating data includes third input sub-data and / or third output sub-data of the target model during forward calculation, and fourth input sub-data and / or fourth output sub-data of the target model during reverse calculation.

[0148] For the content of the third input sub-data and the third output sub-data, reference may be made to the description of the first input sub-data and the first output sub-data in S310, which will not be described in detail here. For the content of the fourth input sub-data and the fourth output sub-data, reference may be made to the description of the second input sub-data and the second output sub-data in S310, which will not be described in detail here.

[0149] In a possible implementation, when the host completes an iteration of the target model using the data to be processed, the acquired running data is stored in the memory of the host.

[0150] Exemplarily, the host stores the second operating data in a target temporary directory in the memory, and the second operating data carries the identifier of the first network node.

[0151] In a possible example, a subdirectory named as the identifier of the second network node is created in the target temporary directory, and the second operating data is stored in the subdirectory named as the identifier of the second network node.

[0152] S330: The host compares the operation data of the pre-marked network nodes with mapping relationships to obtain a comparison result.

[0153] The network nodes with a mapping relationship include a network node in the reference model and a network node in the target model.

[0154] In a possible example, the mapping relationship is used to indicate that the identifiers of the network nodes included in the reference model correspond to the identifiers of the network nodes included in the target model.

[0155] Exemplarily, the host compares the first operation data and the second operation data respectively corresponding to the first network node and the second network node having a mapping relationship to obtain a comparison result.

[0156] As an example, a mapping relationship as shown in Table 1 below indicates that the identifiers of the network nodes included in the benchmark model correspond to the identifiers of the network nodes included in the target model.

[0157] Table 1

[0158] Benchmark Model Target Model Network Node 111 Network Node 211 Network node 112 Network Node 212

[0159] The above Table 1 is only an example and should not be construed as limiting the present application. In a possible example, the Table 1 may also include more correspondences between network nodes, such as network node 1mn included in the baseline model corresponds to network node 2mn included in the target node.

[0160] like Figure 3 As shown, the host compares the operating data of the network node 111 with the operating data of the network node 211, and compares the operating data of the network node 112 with the operating data of the network node 212, and obtains the comparison results respectively.

[0161] In a possible scenario, the shapes of the operation data of the network node 111 involved in the comparison are consistent with those of the operation data of the network node 211, that is, the dimensions and sizes are consistent. If the shapes of the two operation data are inconsistent, the comparison cannot be performed.

[0162] In the present application, the mapping relationship indicates two network nodes that need to be compared (such as the first network node and the second network node with the same identifier), avoiding the problem of complicated comparison process caused by only comparing the operating data at the dot positions in the benchmark model and the corresponding dot positions in the target model. By comparing the operating data of the network nodes indicated by the mapping relationship, the calculation accuracy and comparison efficiency of the network model can be further improved.

[0163] In a possible scenario, the first network node and the second network node having a mapping relationship are the first network node and the second network node having the same identifier.

[0164] The identifier may be the name or number of each network node in the benchmark model and the target model.

[0165] In a possible example, the name of a network node may be a combination of the names or indexes of all its parent nodes, for example, name1.name2.name3.0.name4.1.

[0166] In this application, the names of the above network nodes are used as identifiers to ensure that the positions and functions of the two compared network nodes in the model are the same, thereby ensuring the accuracy of the match. In addition, directly comparing two network nodes with the same identifier can increase the speed of determining the compared network nodes, thereby improving the efficiency of the entire comparison process.

[0167] After the benchmark model and the target model complete an iteration using the data to be processed, the host obtains the first operation data and the second operation data from the memory, and compares the first operation data and the second operation data corresponding to the first network node and the second network node having a mapping relationship to obtain a comparison result.

[0168] Exemplarily, the host determines a first network node and a second network node having a mapping relationship, obtains first operation data and second operation data corresponding to the first network node and the second network node having a mapping relationship from a memory, respectively, and compares them to obtain a comparison result. Alternatively, after obtaining the first operation data of the first network node from the memory, the host determines a second network node having a mapping relationship with the first network node, and then obtains second operation data corresponding to the second network node having a mapping relationship with the first network node from the memory, and compares them to obtain a comparison result.

[0169] In a possible implementation manner, the comparison result in the present application includes: the similarity between the first operation data and the second operation data respectively corresponding to the first network node and the second network node having a mapping relationship.

[0170] The similarity may be cosine similarity, Euclidean distance, Pearson correlation coefficient, etc. If the similarity is greater than or equal to a threshold, the aforementioned comparison result is abnormal.

[0171] In yet another possible implementation, the comparison result described in the present application includes an error between first operation data and second operation data respectively corresponding to the first network node and the second network node having a mapping relationship.

[0172] The error can be absolute error, mean square error, root mean square error, etc.

[0173] To solve the problem that in the prior art, when comparing operation data, after completing a round of training, the operation data stored in the disk is compared, and the operation data obtained from the training round occupies a large amount of storage space, reducing the utilization rate of storage resources. A possible implementation method is provided below.

[0174] After obtaining the comparison result, the host deletes the operation data stored in the memory, such as the operation data corresponding to the network nodes included in the reference model and the operation data corresponding to the network nodes included in the target model.

[0175] Exemplarily, the host deletes the operating data stored in the base temporary directory and the target temporary directory in the memory, or only deletes the operating data stored in the subdirectories under the base temporary directory and the target temporary directory.

[0176] In this application, after the host completes an iteration and compares each pair of the benchmark model and the target model, the operation data generated in this iteration is deleted. In this way, the problem of the host needing to store a large amount of operation data is avoided, the storage space occupied by the operation data is reduced, and no additional storage space is required to store a whole round of operation data, thereby improving the storage space utilization rate.

[0177] Furthermore, after the host obtains the comparison result, the abnormal network node with abnormal comparison result in the target model is determined, and the identifier of the abnormal network node is output so that the user can locate the abnormal network node according to the identifier of the abnormal network node and adjust the abnormal network node, such as adjusting the code of the network node during forward calculation and / or reverse calculation.

[0178] It is worth noting that the contents of S310-S330 above are only the contents of one iteration of the network model in multiple iterations. After the comparison results are obtained, the host receives the user's modification of the target model (i.e., the code for adjusting the network nodes with abnormal comparison results in the target model during forward calculation and / or reverse calculation). Then, the contents of S310-S330 above are executed again until the calculation accuracy of the target model is aligned with that of the benchmark model, that is, the comparison results of each network node in the target model and the benchmark model are greater than the threshold.

[0179] In one possible implementation, when the host performs multiple iterations on the target model and the benchmark model according to the data to be processed, it is necessary to synchronize the running speeds of the benchmark model and the target model, that is, to ensure that the target model and the benchmark model complete one iteration before simultaneously performing the next iteration.

[0180] Exemplarily, after receiving instructions that both the baseline model and the target model have completed one iteration, the host sends iteration instructions to the AI ​​framework running the baseline model and the AI ​​framework running the target model to instruct the AI ​​framework running the baseline model to start the next iteration of the baseline model and the AI ​​framework running the target model to start the next iteration of the target model.

[0181] In one possible scenario, if one of the baseline model or the target model is iterated first, before the AI ​​framework running the baseline model or the target model receives an iteration instruction, the network model that has iterated first will wait until it receives an iteration instruction.

[0182] In order to improve the efficiency of users in locating abnormal network nodes with abnormal calculation accuracy, in this application, the host outputs the identification of abnormal network nodes in the following manner: the host performs syntax tree parsing on the code corresponding to the target model to obtain a tree diagram. Each node in the tree diagram is used to indicate the identification of the network node included in the target model, and the association between the nodes in the tree diagram represents the processing order of the network nodes in the target model. The host matches the comparison results of the network nodes included in the target model with the corresponding nodes in the tree diagram, obtains and outputs the network topology diagram corresponding to the target model, so that the user can locate the abnormal network node according to the network topology diagram.

[0183] In this application, the host outputs the network topology of the target model. Based on the visualized network topology, the user can quickly identify the abnormal network nodes with abnormal calculation accuracy according to the identification and comparison results of the network nodes in the network topology, thereby improving the efficiency of locating abnormal network nodes. And the user can determine the impact of the abnormal network node on the upstream or downstream nodes of the abnormal network node according to the association between the nodes in the network topology.

[0184] like Figure 5 As shown, Figure 5 The network topology diagram provided by this application. The objective function includes convolution layer 1, pooling layer 1, fully connected layer 1 and activation function layer 1. The corresponding comparison results of convolution layer 1 are 0.999, pooling layer 1 is 0.990, fully connected layer 1 is 0.958, and activation function layer 1 is 0.950. If the threshold is 0.96, the user determines that fully connected layer 1 and activation function layer 1 are abnormal network nodes.

[0185] In a possible example, the host may output the network topology diagram to a front end connected to the host for display, or directly display it on a display interface of the host.

[0186] In a possible implementation, a placeholder node is set in a pre-marked network node whose comparison result is less than or equal to a threshold, and operation data of the placeholder node is used to compare the computational accuracy of the benchmark model and the target model.

[0187] Exemplarily, based on the above comparison results, the user inserts a placeholder node at the same position in the reference model and the target model (the same position in the network node with the same identifier), the placeholder node in the reference model is inside the first network node, and the placeholder node in the target model is inside the second network node. Compared with the calculation means included between two network nodes (comparison points), the placeholder node is inside the network node, so the calculation means included between the placeholder node and the comparison point are reduced, which reduces the scope of the user's investigation of abnormal calculation means, thereby improving the positioning accuracy of abnormal calculation means. Furthermore, after improving the positioning accuracy of abnormal calculation means, the user can quickly locate and adjust the abnormal calculation means, and finally achieve comparison efficiency with improved calculation accuracy.

[0188] Exemplarily, the position of the placeholder node in the first network node is consistent with the position of the placeholder node in the second network, and the first network node and the second network node are network nodes with a mapping relationship. Furthermore, the first network node and the second network node have the same identifier. For ease of description, the placeholder node located in the network node can also be called a comparison point.

[0189] Three possible embodiments are provided below to illustrate the above-mentioned model accuracy comparison method.

[0190] In a first possible embodiment, the accuracy comparison method provided in this embodiment can be applied to Figure 2 The computer system shown in FIG. Figure 6 As shown, Figure 6 This is a flow chart of the model accuracy comparison method deployed in different frameworks provided in this application. The model accuracy comparison method can be executed by a host, which may include Figure 2 In the illustrated computer system, one or more computing devices are deployed on the host with different AI frameworks, such as the pytorch framework and the mindspore framework. The pytorch framework and the mindspore framework are deployed on the same computing device, and the host is used to train the network model on the pytorch framework and the mindspore framework as an example. Figure 6 As shown, the model accuracy comparison method may include the following contents:

[0191] ①: The host configures the same network model on the pytorch framework and the mindspore framework.

[0192] The host first configures the network model (baseline model) on the pytorch framework, and according to the network model in the pytorch framework, uses the API interface provided by mindspore to implement the mindspore framework configuration network model (target model).

[0193] In one possible scenario, an optimizer may be configured for the above-mentioned baseline model and target model so that the baseline model and the target model can gradually converge to a better state.

[0194] ②: The host initializes the benchmark model and the target model to ensure that the parameters of the benchmark model of the pytorch framework and the target model on the mindspore framework are consistent.

[0195] Exemplarily, the host initializes the parameters of the benchmark model to obtain the initial parameters. Then, the host saves the initial parameters of the benchmark model as a pth file and converts the pth file into a ckpt file supported by the mindspore framework. After the mindspore framework reads the ckpt file, it uses the initialization parameters saved in the ckpt file as the initialization parameters of the target model. The host configures the target model according to the initialization parameters of the target model to achieve consistency between the parameters of the target model and the parameters of the benchmark model.

[0196] The above parameters include the weights and biases of each network node in the network model.

[0197] ③: The host registers the network nodes included in the benchmark model to the callback function of the forward calculation, or the callback function of the reverse calculation, and registers the network nodes included in the target model to the callback function of the forward calculation, or the callback function of the reverse calculation.

[0198] The callback function is used to obtain the operation data of the network node when the network node is executed. The operation data includes the input sub-data and output sub-data of the network node in the forward calculation process, and the input sub-data and output sub-data of the network node in the reverse calculation process.

[0199] After registration is completed, when the host executes to the corresponding network node, it obtains the operation data of all network nodes in the forward calculation or reverse calculation process of the network model through the callback function and saves it to the memory.

[0200] Exemplarily, the network nodes in the above-mentioned benchmark model belong to the network nodes of the nn.module class, and the network nodes in the target model belong to the network nodes of the nn.cell class.

[0201] The nn.module class is a base class in the pytorch framework, and the construction of all network nodes in the benchmark model depends on the nn.module class. The nn.cell class is a base class in the mindspore framework, and the construction of all network nodes in the target model depends on the nn.cell class.

[0202] ④: The host fixes the same random seed for the baseline model and the target model to ensure that the randomness of the baseline model and the target model is consistent during the training process.

[0203] A random seed is a starting value that is used to initialize a pseudo-random number generator (PRNG). A pseudo-random number generator is a mathematical function based on a deterministic algorithm that is used to generate a seemingly random sequence of numbers. A deterministic algorithm is one that always produces the same output given the same input conditions.

[0204] The above determination that the randomness of the baseline model and the target model is consistent during the training process means that the host initializes the pseudo-random number generator with a random seed to obtain a random number. The random number will be used for data enhancement and sample sampling (i.e., randomly selecting a batch of samples from the entire data to be processed for this iteration) during the network model training process.

[0205] ⑤: The host performs multiple iterations of training on the benchmark model and the target model according to the data to be processed, and each iteration obtains the first operating data corresponding to the first network node and the second operating data corresponding to the second network node.

[0206] The operation data is the input sub-data and / or output sub-data of the function node that realizes the preset function during forward calculation, and the input sub-data and / or output sub-data during reverse calculation.

[0207] In each iteration, the host randomly selects a batch of samples from the data to be processed for training.

[0208] For example, there are 100,000 pieces of data to be processed, and 1,000 pieces of data are selected from the 100,000 pieces of data for training in each iteration.

[0209] ⑥: Each time the host iterates the baseline model and the target model, it synchronizes the training speed of the baseline model and the target model to ensure that the baseline model and the target model start the next iteration at the same time.

[0210] ⑦: The host compares the first operation data and the second operation data corresponding to the first network node and the second network node having a mapping relationship respectively to obtain a comparison result.

[0211] After the comparison is completed, the host deletes the operating data of the network nodes included in the baseline model and the operating data of the network nodes included in the target model.

[0212] ⑧: The host obtains and outputs a network topology diagram based on the comparison results and the tree diagram obtained by parsing the syntax tree of the target model. The nodes in the network topology diagram are used to indicate the identifiers of the network nodes included in the target model.

[0213] In a possible scenario, the above method may further include the following contents:

[0214] ⑨: For abnormal network nodes whose host output comparison results are less than or equal to the threshold, the user adjusts the abnormal network node. Alternatively, the user determines the abnormal network node from the network topology diagram and adjusts the abnormal network node.

[0215] The host will retrain the target model and the benchmark model including the adjusted abnormal network nodes to repeat the above steps ⑤-⑧ until the comparison result is greater than or equal to the threshold (such as 0.97), or the comparison result obtained in one iteration is less than the threshold, then stop training the target model and the benchmark model.

[0216] If the comparison result is still less than the threshold, then execute step ⑨ again to obtain the adjusted abnormal network node, and execute steps ⑤-⑧ again until the comparison result between the baseline model and the target model is greater than or equal to the threshold.

[0217] In a possible example, the user can locate the abnormal network node with abnormal calculation accuracy through the comparison results of the above network nodes or the network topology map. If the abnormal network node covers a large range, the user can insert a placeholder node at the same position of the benchmark model and the target model, and the position is located in the network node, so as to narrow the scope of the abnormal area, thereby improving the user's efficiency in locating and adjusting the problem code in the abnormal area.

[0218] The host repeats the above steps ⑤-⑨ with the target model including the adjusted network nodes and the benchmark model until the comparison results of all network nodes in the target model with the corresponding network nodes in the benchmark model are greater than the threshold.

[0219] In a second possible embodiment, this embodiment is a model accuracy comparison method for network models deployed on different computing devices. The accuracy comparison method provided in this embodiment can be applied to Figure 2 The computer system shown. The model accuracy comparison method can be executed by a host, which may include Figure 2 In the illustrated computer system, there are multiple computing devices, such as a host including a CPU and a GPU, on which the pytorch framework is deployed. Training a network model in the pytorch framework deployed on the CPU and the GPU is used as an example for explanation.

[0220] The content of the model accuracy comparison method in this embodiment is Figure 6 The displayed content differs only in that:

[0221] Difference 1: The baseline model and target model of this embodiment run in the same software environment, such as the same AI framework, such as the pytorch framework, but the AI ​​framework is deployed on different computing devices. For example, the pytorch framework running the baseline model is deployed on the GPU, and the pytorch framework running the target model is deployed on the NPU.

[0222] The host configures a network model (baseline model) on the pytorch framework deployed on the GPU, and modifies the compute unified device architecture (cuda) interface of the network model configured on the GPU to the npu interface, thereby obtaining the network model (target model) configured on the NPU.

[0223] Difference 2: In the above step ③, when the host registers the network nodes included in the baseline model and the target model to the callback functions of the forward calculation and the reverse calculation, the network nodes all belong to the network nodes of the nn.module class.

[0224] In a third possible embodiment, this embodiment is a model accuracy comparison method for different network models deployed on the same computing device. The accuracy comparison method provided in this embodiment can be applied to Figure 2 The computer system shown. The model accuracy comparison method can be executed by a host, which may include Figure 2 In the illustrated computer system, one or more computing devices, different network models are deployed in the same AI framework. Different network models are run in the pytorch framework, and the pytorch is deployed on the GPU as an example.

[0225] The content of the model accuracy comparison method is Figure 6 The displayed content differs only in that:

[0226] ①: The host configures the benchmark model and target model on the pytorch framework.

[0227] For example, the target model may be obtained by adjusting the benchmark model by the user, such as modifying the code of the benchmark model for the purpose of performance tuning, or adjusting the network structure of the benchmark model.

[0228] ②: The host initializes the baseline model and the target model to ensure that the parameters of the baseline model and the target model are consistent.

[0229] And in step ③, when the host registers the network nodes included in the baseline model and the target model to the callback functions of the forward calculation and the reverse calculation, the network nodes all belong to the network nodes of the nn.module class.

[0230] The above-mentioned GPU may be a centralized GPU cluster or a distributed GPU cluster.

[0231] Centralized GPU cluster: A centralized GPU cluster is one where all GPU nodes are concentrated in one physical location or data center. They share the same network, storage, and other infrastructure resources. The management and maintenance of a centralized cluster is relatively simple because all nodes are in the same location and share a consistent environment.

[0232] Distributed GPU cluster: A distributed GPU cluster is one where GPU nodes are distributed in different geographical locations or data centers. Each node has independent computing resources and storage devices. Distributed clusters can provide higher computing performance and scalability because tasks can be distributed to multiple nodes for parallel processing.

[0233] It is understandable that in order to implement the functions in the above embodiments, the host includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and method steps of each example described in the embodiments disclosed in this application, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0234] Combined with the above Figures 2 to 6 , describes in detail the model accuracy comparison method provided by this embodiment, and will be combined with Figure 7 , describing the model accuracy comparison device provided according to this embodiment.

[0235] Figure 7 Schematic diagram of the structure of the model accuracy comparison device provided in this application Figure 1 The model accuracy comparison device can be used to implement the functions of the host in the above method embodiment, and thus can also achieve the beneficial effects possessed by the above method embodiment. In this embodiment, the model accuracy comparison device can be a module (such as a chip) applied to the host.

[0236] like Figure 7 As shown, the model accuracy comparison device 700 is used to compare the computational accuracy of the benchmark model and the target model. The benchmark model and the target model are the same model running in different software environments or hardware computing power, or the benchmark model and the target model have different network structures. The model accuracy comparison device 700 includes a first acquisition module 710, a second acquisition module 720 and a comparison module 730. The model accuracy comparison device 700 is used to implement the above Figures 2 to 6 The functions in the method embodiments shown in .

[0237] The first acquisition module 710 is used to run the benchmark model based on the data to be processed, and acquire and save the operation data of the network nodes pre-marked in the benchmark model; wherein the network nodes pre-marked in the benchmark model are functional nodes in the benchmark model for implementing preset functions.

[0238] The second acquisition module 720 runs the target model based on the data to be processed, and acquires and saves the operation data of the network nodes pre-marked in the target model; wherein the network nodes pre-marked in the target model are function nodes in the target model for realizing preset functions.

[0239] The comparison module 730 is used to compare the operation data of the pre-marked network nodes with mapping relationships to obtain comparison results. The network nodes with mapping relationships include a network node in the reference model and a network node in the target model.

[0240] To further achieve the above Figures 2 to 6 The present application also provides a model accuracy comparison device, such as Figure 8 As shown, Figure 8 Schematic diagram of the structure of the model accuracy comparison device provided in this application Figure 2 The model accuracy comparison device 700 also includes a registration module 740, an output module 750, a synchronization module 760 and a deletion module 770.

[0241] The registration module 740 is used to register the network nodes pre-marked in the reference model in the first callback function, and the first callback function is used to obtain and automatically save the operation data of the network nodes pre-marked in the reference model during the forward calculation and reverse calculation of the reference model. The network nodes pre-marked in the target model are registered in the second callback function, and the second callback function is used to obtain and automatically save the operation data of the network nodes pre-marked in the target model during the forward calculation and reverse calculation of the target model.

[0242] The output module 750 is used to output the network topology diagram of the target model. The nodes in the network topology diagram are used to indicate the identifiers of the network nodes included in the target model and the comparison results corresponding to the network nodes included in the target model.

[0243] The synchronization module 760 is used to synchronize the running speeds of the benchmark model and the target model.

[0244] So that the baseline model and the target model can proceed to the next iteration only after completing one iteration based on the data to be processed.

[0245] The deletion module 770 is used to delete the saved operation data of the network nodes pre-marked in the reference model and the operation data of the network nodes pre-marked in the target model.

[0246] It should be understood that the host according to the embodiment of the present application may correspond to the model accuracy comparison device 700 in the embodiment of the application, and may correspond to the execution of the method according to the embodiment of the present application. Figures 2 to 6 The corresponding corresponding subjects, and the operations and / or functions of each module in the model accuracy comparison device 700 are respectively to achieve Figures 2 to 6 For the sake of brevity, the corresponding processes of each method in the corresponding embodiments are not repeated here.

[0247] The present application provides a processing device, such as Fig. 9 As shown, Fig. 9 A schematic diagram of the structure of a processing device provided in this application. The processing device can be applied to Figure 2In the computer system shown, the processing device may be any one or more of computing device 210 , computing device 220 , or computing device 230 .

[0248] The processing device 900 can specifically be a mobile phone, a tablet computer, a television (also known as a smart TV, a smart screen or a large-screen device), a laptop computer, an ultra-mobile personal computer (UMPC), a handheld computer, a netbook, a personal digital assistant (PDA), a wearable electronic device (for example: a smart watch, a smart bracelet, smart glasses), a vehicle-mounted device, a virtual reality device, and other processing devices with the function of running a model.

[0249] like Fig. 9 As shown, the processing device 900 may include a processor 910 , a memory 920 , a communication interface 930 , a bus 940 , etc. The processor 910 , the memory 920 , and the communication interface 930 are connected via the bus 940 .

[0250] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the processing device. In other embodiments of the present application, the processing device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0251] The processor 910 may include one or more processing units, for example, the processor 910 may include an AP, a modem processor, a CPU, a GPU, an ISP, a controller, a memory, a video codec, a DSP, a baseband processor, and / or an NPU, etc. Different processing units may be independent devices or integrated into one or more processors.

[0252] The processor 910 may also be provided with an internal memory for storing instructions and data. In some embodiments, the internal memory in the processor 910 is a cache memory. The internal memory may store instructions or data that the processor 910 has just used or cyclically used. If the processor 910 needs to use the instruction or data again, it may be directly called from the internal memory. This avoids repeated access, reduces the waiting time of the processor 910, and thus improves the efficiency of the system.

[0253] The memory 920 can be used to store computer executable program codes, which include instructions. The processor 910 executes various functional applications and data processing of the processing device 900 by running the instructions stored in the internal memory 920. The internal memory 920 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a running model function, a sending function, etc.), etc. The data storage area may store data (such as running data, etc.) created during the use of the processing device 900, etc. In addition, the internal memory 920 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0254] The communication interface 930 is used to implement communication between the processing device 900 and an external device or component. In this embodiment, the communication interface 930 is used to perform data exchange with other processing devices.

[0255] The bus 940 may include a path for transmitting information between the above components (such as the processor 910, the memory 920, and the communication interface 930). In addition to the data bus, the bus 940 may also include a power bus, a control bus, and a status signal bus. However, for the sake of clarity, various buses are labeled as bus 940 in the figure. The bus 940 may be a PCIe bus, an EISA bus, a Ubus or UB, CXL, CCIX, etc.

[0256] It is worth mentioning that Fig. 9 In the example, the processing device 900 includes one processor 910 and one memory 920. Here, the processor 910 and the memory 920 are respectively used to indicate a type of device or equipment. In a specific embodiment, the number of each type of device or equipment can be determined according to business requirements.

[0257] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in the embodiment of the present application is executed in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device or other programmable device. The computer program or instruction may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instruction may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, for example, a floppy disk, a hard disk, a tape; it may also be an optical medium, for example, a digital video disc (DVD); it may also be a semiconductor medium, for example, a solid state drive (SSD).

[0258] In each embodiment of the present application, if there is no special explanation and logical conflict, the terms and / or descriptions between different embodiments are consistent and can be referenced to each other, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationships. The various digital numbers involved in the embodiments of the present application are only used to distinguish for the convenience of description and are not used to limit the scope of the embodiments of the present application. The size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and inherent logic.

Claims

1. A model accuracy comparison method, characterized in that: For comparing the computational accuracy of a benchmark model and a target model, the benchmark model and the target model are the same model running in different software environments or hardware computing power, or the benchmark model and the target model have different network structures; the method comprises: Running a benchmark model based on the data to be processed, acquiring and saving the running data of the network nodes pre-marked in the benchmark model; wherein the network nodes pre-marked in the benchmark model are functional nodes in the benchmark model for realizing preset functions; Running the target model based on the data to be processed, acquiring and saving the running data of the network nodes pre-marked in the target model; wherein the network nodes pre-marked in the target model are function nodes in the target model for realizing preset functions; The operation data of the pre-marked network nodes with mapping relationships are compared to obtain a comparison result; wherein the network nodes with mapping relationships include a network node in the benchmark model and a network node in the target model.

2. The method according to claim 1, characterized in that Before running the benchmark model based on the data to be processed, the method further includes: Registering the network nodes pre-marked in the benchmark model in a first callback function, wherein the first callback function is used to obtain and automatically save the operation data of the network nodes pre-marked in the benchmark model during the forward calculation and reverse calculation of the benchmark model; The network nodes pre-marked in the target model are registered in a second callback function, and the second callback function is used to obtain and automatically save the operation data of the network nodes pre-marked in the target model in the forward calculation and reverse calculation of the target model.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: Output a network topology diagram of the target model; the nodes in the network topology diagram are used to indicate the identifiers of the network nodes included in the target model, and the comparison results corresponding to the network nodes included in the target model.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: The operating speeds of the benchmark model and the target model are synchronized.

5. The method according to claim 3 or 4, characterized in that: The method further comprises: The stored operation data of the network nodes pre-marked in the reference model and the stored operation data of the network nodes pre-marked in the target model are deleted.

6. The method according to any one of claims 1 to 5, characterized in that The pre-marked network nodes with a mapping relationship are the pre-marked network nodes in the reference model and the pre-marked network nodes in the target model with the same identifier.

7. The method according to any one of claims 1 to 6, characterized in that The operation data of the network node includes input sub-data and / or output sub-data of the network node during forward calculation, and input sub-data and / or output sub-data of the network node during reverse calculation.

8. The method according to any one of claims 1 to 7, characterized in that The comparison result is the similarity and / or error between the operation data of the network nodes having the mapping relationship.

9. The method according to any one of claims 1 to 8, characterized in that A placeholder node is provided in the pre-marked network nodes whose comparison results are less than or equal to the threshold, and the operation data of the placeholder node is used to compare the calculation accuracy of the benchmark model and the target model.

10. A model accuracy comparison device, characterized in that: Used to compare the computational accuracy of a benchmark model and a target model, where the benchmark model and the target model are the same model running in different software environments or hardware computing power, or the benchmark model and the target model have different network structures; The device comprises: A first acquisition module is used to run a benchmark model based on the data to be processed, and acquire and save the operation data of the network nodes pre-marked in the benchmark model; wherein the network nodes pre-marked in the benchmark model are function nodes in the benchmark model for implementing preset functions; A second acquisition module is used to run the target model based on the data to be processed, and acquire and save the operation data of the network nodes pre-marked in the target model; wherein the network nodes pre-marked in the target model are function nodes in the target model for realizing preset functions; The comparison module is used to compare the operation data of the pre-marked network nodes with mapping relationships to obtain comparison results; wherein the network nodes with mapping relationships include a network node in the benchmark model and a network node in the target model.

11. The device according to claim 10, characterized in that The device also includes: A registration module is used to register the network nodes pre-marked in the baseline model in a first callback function, the first callback function is used to obtain and automatically save the operating data of the network nodes pre-marked in the baseline model during the forward calculation and reverse calculation of the baseline model; and register the network nodes pre-marked in the target model in a second callback function, the second callback function is used to obtain and automatically save the operating data of the network nodes pre-marked in the target model during the forward calculation and reverse calculation of the target model.

12. The device according to claim 10 or 11, characterized in that The device also includes: An output module is used to output a network topology diagram of the target model; the nodes in the network topology diagram are used to indicate the identifiers of the network nodes included in the target model, and the comparison results corresponding to the network nodes included in the target model.

13. The device according to any one of claims 10 to 12, characterized in that The device also includes: A synchronization module is used to synchronize the running speeds of the benchmark model and the target model.

14. The device according to claim 12 or 13, characterized in that The device also includes: The deleting module is used to delete the saved operation data of the network nodes pre-marked in the reference model and the operation data of the network nodes pre-marked in the target model.

15. The device according to any one of claims 10 to 14, characterized in that The pre-marked network nodes with a mapping relationship are the pre-marked network nodes in the reference model and the pre-marked network nodes in the target model with the same identifier.

16. The device according to any one of claims 10 to 15, characterized in that The operation data of the network node includes input sub-data and / or output sub-data of the network node during forward calculation, and input sub-data and / or output sub-data of the network node during reverse calculation.

17. The device according to any one of claims 10 to 16, characterized in that The comparison result is the similarity and / or error between the operation data of the network nodes having the mapping relationship.

18. The device according to any one of claims 10 to 17, characterized in that A placeholder node is provided in the pre-marked network nodes whose comparison results are less than or equal to the threshold, and the operation data of the placeholder node is used to compare the calculation accuracy of the benchmark model and the target model.

19. A chip, characterized in that: include: A processor and a power supply circuit, wherein the power supply circuit is used to supply power to the processor, and the processor is used to call and run computer instructions to execute the method according to any one of claims 1 to 9.

20. A computing device, characterized in that include: Processor and memory; The memory stores instructions, and the processor calls the instructions to implement the method according to any one of claims 1 to 9.

21. A computer-readable storage medium, characterized in that: The storage medium stores a computer program or instruction, and when the computer program or instruction is executed by a processing device, the method according to any one of claims 1 to 9 is implemented.

22. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed on a processing device, the method according to any one of claims 1 to 9 is implemented.

Images